Skip to content

1.0.8 json report "undefined" skipped tests #328

Closed
poum opened this Issue May 25, 2012 · 8 comments

2 participants

@poum
poum commented May 25, 2012

Hi and many thanks for cucumber-jvm.

Trying to use jenkins-cucumber-jvm-reports-plugin-java, I've noticed that skipped test (using @ignore and PendingException) are correctly reported in xml surefire-report but as "undefined" in json file.

I'm using cucumber-jvm 1.0.8, maven 3.0.4 & java openjdk 1.6.

damianszczepanik/cucumber-reporting-jenkins#7

@poum
poum commented May 29, 2012

Hi again. I've further investigate and according to my comprehension, cucumber-jvm has the following behaviour:

  • unimplemented step => 'pending': OK
  • @ignore step => 'skipped': OK
  • step throwing PendingException => 'failed': ??

I was expecting that step throwing a PendingException would be marked as 'pending' ... Am I wrong ?

@aslakhellesoy
Cucumber member

Cucumber steps can end up in one of the following native states:

  • passed (console green)
  • failed (console red)
  • pending (console yellow)
  • undefined (console yellow)
  • skipped (console cyan)

When using the JUnit runner we have to map those states onto the states that JUnit understands:

  • passed
  • failed
  • ignored

All of the states pending, undefined and skipped are mapped to the ignored state for JUnit. The JSON report uses Cucumber's native states.

Does this answer your question? If not, can you explain with some more concrete examples? (code snippets and output snippets). Please use triple backticks if you paste code. See GFM

@poum
poum commented May 31, 2012

Many thanks for this explanation. I mean: could cucumber-jvm catch PendingExceptions to avoid the step to fail and marked it as 'pending' in JSON (and 'ignored' in JUnit). To me, a pending step should not cause test fail ... but also not pass. Another way to express my question could be: what is PendingException purpose ?

@aslakhellesoy
Cucumber member

I don't understand your question. When PendingException is thrown, it doesn't cause a step to fail. Cucumber catches it internally and causes the step to be marked as pending.

That is, unless you're using the --strict option, which causes PendingException to be treated as a regular error (fail).

The purpose of marking a step as pending is to give yourself a reminder to implement this step definition.

I'm going to close this issue unless you provide me with some code to describe what you mean.

@aslakhellesoy
Cucumber member

Never mind my previous comment. It looks like there has been a recent regression here. I'm working on a fix.

@poum
poum commented Jun 8, 2012

Ouf ! I started thinking I forgot how throwing an exception ;-)

Many thanks for time spend on my request ...

@poum
poum commented Jun 8, 2012

Ok, it works perfectly for me. Many thanks !

@aslakhellesoy
Cucumber member

Great! Thanks for the feedback and sorry about the confusion/regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.