Skip to content

JUnit report layout #171

Closed
trnl opened this Issue Jan 24, 2012 · 26 comments

4 participants

@trnl
trnl commented Jan 24, 2012

When using one class to run multiply features, it should be possible to configure cucumber to create separate testsuite for each .feature file.

<testsuite errors="0" failures="0" name="{class.name}" skipped="0" tests="3" time="1.881000">
    <!-- ... -->
    <testcase classname="Scenario: {scenario.name}" name="{step.name}" time="0.047000"/>
    <!-- ... -->
</testsuite>

Also I believe that testcase should be created for each scenario, not for each step. Just like it was in cuke4duke.

<testsuite errors="0" failures="1" name="{feature.name}" skipped="0" tests="3" time="1.881000">
    <!-- ... -->
    <testcase classname="{feature.name}.{scenario.name}" name="{scenario.name}" time="0.047000">
         <failure message="failed {step.name}" type="failed">
             <![CDATA[Scenario: {step.name}

                Given ...
                And ...
                And ...
                When ...
                And ...

                Failed: When .... {failed step}

                Message: java.lang.NoSuchMethodError: null (NativeException)
                ...                    
             ]]>
         </failure>
    </testcase>
    <!-- ... -->
</testsuite>
@aslakhellesoy
Cucumber member

The JUnit report format was originally popularised by Ant's JUnitReport task.

Maven's Surefire plugin attempts to output XML in an identical format to the JUnitReport Ant task.

In other words, it's not Cucumber-JUnit that generates the XML, it's Maven. Cucumber-JUnit creates a hierarchical test suite from the features and fires events and results accordingly. (In Cuke4Duke however, features were running through Ruby cucumber without involving JUnit at all, and would output JUnitReport-style XML on its own).

So it looks like Surefire doesn't understand how to represent these hierarchical test results properly. I don't think that's something that can be fixed in Cucumber-JUnit.

@trnl
trnl commented Jan 24, 2012

I can see that in cucumber.junit.FeatureRunner ExecutionUnitRunner is being created for each scenario, and for each step we have EachTestNotifier created.

I believe that EachTestNotifier should be created for each scenario or example, and not for each step. And when reporting failure - you can gather all steps in message.

@trnl
trnl commented Jan 24, 2012

I mean that if

@Feature(value="com.test.cucumber")

And com.test.cucumber contains 2 features, we should create 2 test suites for each.
Now - one test suite created where all scenarios are messed.

@aslakhellesoy
Cucumber member

@trnl would you be interested in helping out with this? A pull request to improve this would be awesome.

@trnl
trnl commented Jan 24, 2012

@aslakhellesoy sure, let me get familiar with junit api.

At this moment for us it's a blocker. We've got around 200 features in folder tree structure, and creating separate public class for each sounds terrible.

@aslakhellesoy
Cucumber member

Who says you need to create a public class for each feature?

@trnl
trnl commented Jan 24, 2012

I mean you need a separate public class for each feature if you want separate testsuite for each class. Let's see by example:

test
|- java
  |- test_a.java
  |- test_b.java
  |- test_both.java
|- resources
  |- com.foo
     |- a.feature
     |- b.feature

If you want to run both features, there's 2 possible ways:
1. create separate class for each feature with @Feature(value="com.foo.a.feature") or @Feature(value="com.foo.b.feature")
2. create one class where @Feature(value="com.foo").

For 1st case you'll get TEST-com.foo.test_a.xml and TEST-com.foo.test_b.xml reports. Each report will have following layout:

<testsuite failures="0" time="2.571" errors="0" skipped="0" tests="4" name="com.foo.test_a">
    <testcase classname="scenario1(from a)" name="step1"/>
    <testcase classname="scenario1(from a)" name="step2"/>
    <testcase classname="scenario2(from a)" name="step1"/>
    <testcase classname="scenario2(from a)" name="step2"/>
</testsuite>

For 2nd - you'll get TEST-com.foo.test_both.xml with all scenarios from 2 features:

<testsuite failures="0" time="2.571" errors="0" skipped="0" tests="8" name="com.foo.test_both">
    <testcase classname="scenario1(from a)" name="step1"/>
    <testcase classname="scenario1(from a)" name="step2"/>
    <testcase classname="scenario2(from a)" name="step1"/>
    <testcase classname="scenario2(from a)" name="step2"/>
    <testcase classname="scenario1(from b)" name="step1"/>
    <testcase classname="scenario1(from b)" name="step2"/>
    <testcase classname="scenario2(from b)" name="step1"/>
    <testcase classname="scenario2(from b)" name="step2"/>
</testsuite>

So in our case executing all features with one class with @Feature annotation results in one xml file where there are about 4,5k testcase elements, and it's impossible to find out which feature they belong to.

@aslakhellesoy
Cucumber member

I see what you mean. It will be interesting to see if a EachTestNotifier per scenario/example will fix this problem.

@aslakhellesoy
Cucumber member

I'm not sure the number of created EachTestNotifier objects has an impact on the output format though - it's just a facade in front of a RunNotifier with an attached Description.

I think perhaps the solution lies in making sure the Description objects we create are constructed with the proper children - see Description.getChildren() and Description.addChild()

@aslakhellesoy
Cucumber member

I just verified - and it looks like the Description objects are created properly with children. I'm not sure what needs to be done to get a <testsuite> per scenario, but I'm looking forward to seeing what you find out.

@aslakhellesoy
Cucumber member

This is how IDEA shows the output

IntelliJ IDEA

In other words- the hierarchical structure is reflected in the output. Shouldn't it be considered a Surefire bug if it can't reflect the same structure in XML?

@aslakhellesoy
Cucumber member

I tried to create a regular JUnit suite and see how the output was for that:

package cucumber.runtime.jruby.test;

import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Suite.class)
@Suite.SuiteClasses({Sub1.class,Sub2.class})
public class SuiteTest {
}

And the other 2 tests:

package cucumber.runtime.jruby.test;

import org.junit.Test;

import static org.junit.Assert.fail;

public class Sub1 {
    @Test
    public void test_that_fails_sub1() {
        fail("I'm Sub 1");
    }

    @Test
    public void test_that_passes_sub1() {
    }
}
package cucumber.runtime.jruby.test;

import org.junit.Test;

import static org.junit.Assert.fail;

public class Sub2 {
    @Test
    public void test_that_passes_sub2() {
    }

    @Test
    public void test_that_fails_sub2() {
        fail("I'm Sub 1");
    }
}

This only creates a single XML file in target/surefire-reports/TEST-cucumber.runtime.jruby.test.SuiteTest.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite failures="2" time="0.037" errors="0" skipped="0" tests="4" name="cucumber.runtime.jruby.test.SuiteTest">
  <properties>
    <property name="java.runtime.name" value="Java(TM) SE Runtime Environment"/>
    <property name="sun.boot.library.path" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64"/>
    <property name="java.vm.version" value="20.1-b02"/>
    <property name="java.vm.vendor" value="Sun Microsystems Inc."/>
    <property name="java.vendor.url" value="http://java.sun.com/"/>
    <property name="path.separator" value=":"/>
    <property name="java.vm.name" value="Java HotSpot(TM) 64-Bit Server VM"/>
    <property name="file.encoding.pkg" value="sun.io"/>
    <property name="user.country" value="US"/>
    <property name="sun.java.launcher" value="SUN_STANDARD"/>
    <property name="sun.os.patch.level" value="unknown"/>
    <property name="java.vm.specification.name" value="Java Virtual Machine Specification"/>
    <property name="user.dir" value="/home/dfukdev/src/github/cucumber-jvm/jruby"/>
    <property name="java.runtime.version" value="1.6.0_26-b03"/>
    <property name="java.awt.graphicsenv" value="sun.awt.X11GraphicsEnvironment"/>
    <property name="basedir" value="/home/dfukdev/src/github/cucumber-jvm/jruby"/>
    <property name="java.endorsed.dirs" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/endorsed"/>
    <property name="os.arch" value="amd64"/>
    <property name="surefire.real.class.path" value="/home/dfukdev/src/github/cucumber-jvm/jruby/target/surefire/surefirebooter2577422097098355850.jar"/>
    <property name="java.io.tmpdir" value="/tmp"/>
    <property name="line.separator" value="
"/>
    <property name="java.vm.specification.vendor" value="Sun Microsystems Inc."/>
    <property name="os.name" value="Linux"/>
    <property name="sun.jnu.encoding" value="UTF-8"/>
    <property name="java.library.path" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64/server:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/amd64:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib"/>
    <property name="surefire.test.class.path" value="/home/dfukdev/src/github/cucumber-jvm/jruby/target/test-classes:/home/dfukdev/src/github/cucumber-jvm/jruby/target/classes:/home/dfukdev/.m2/repository/info/cukes/cucumber-core/1.0.0.RC13-SNAPSHOT/cucumber-core-1.0.0.RC13-SNAPSHOT.jar:/home/dfukdev/.m2/repository/info/cukes/gherkin/2.7.4/gherkin-2.7.4.jar:/home/dfukdev/.m2/repository/com/thoughtworks/xstream/xstream/1.4.2/xstream-1.4.2.jar:/home/dfukdev/.m2/repository/com/googlecode/java-diff-utils/diffutils/1.2.1/diffutils-1.2.1.jar:/home/dfukdev/.m2/repository/com/google/code/gson/gson/2.1/gson-2.1.jar:/home/dfukdev/.m2/repository/info/cukes/cucumber-html/0.2.1/cucumber-html-0.2.1.jar:/home/dfukdev/.m2/repository/org/jruby/jruby-complete/1.6.5.1/jruby-complete-1.6.5.1.jar:/home/dfukdev/.m2/repository/info/cukes/cucumber-junit/1.0.0.RC13-SNAPSHOT/cucumber-junit-1.0.0.RC13-SNAPSHOT.jar:/home/dfukdev/.m2/repository/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar:/home/dfukdev/.m2/repository/junit/junit/4.10/junit-4.10.jar:/home/dfukdev/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar:"/>
    <property name="java.specification.name" value="Java Platform API Specification"/>
    <property name="java.class.version" value="50.0"/>
    <property name="sun.management.compiler" value="HotSpot 64-Bit Tiered Compilers"/>
    <property name="os.version" value="2.6.32-37-generic"/>
    <property name="user.home" value="/home/dfukdev"/>
    <property name="user.timezone" value=""/>
    <property name="java.awt.printerjob" value="sun.print.PSPrinterJob"/>
    <property name="file.encoding" value="UTF-8"/>
    <property name="java.specification.version" value="1.6"/>
    <property name="user.name" value="dfukdev"/>
    <property name="java.class.path" value="/home/dfukdev/src/github/cucumber-jvm/jruby/target/test-classes:/home/dfukdev/src/github/cucumber-jvm/jruby/target/classes:/home/dfukdev/.m2/repository/info/cukes/cucumber-core/1.0.0.RC13-SNAPSHOT/cucumber-core-1.0.0.RC13-SNAPSHOT.jar:/home/dfukdev/.m2/repository/info/cukes/gherkin/2.7.4/gherkin-2.7.4.jar:/home/dfukdev/.m2/repository/com/thoughtworks/xstream/xstream/1.4.2/xstream-1.4.2.jar:/home/dfukdev/.m2/repository/com/googlecode/java-diff-utils/diffutils/1.2.1/diffutils-1.2.1.jar:/home/dfukdev/.m2/repository/com/google/code/gson/gson/2.1/gson-2.1.jar:/home/dfukdev/.m2/repository/info/cukes/cucumber-html/0.2.1/cucumber-html-0.2.1.jar:/home/dfukdev/.m2/repository/org/jruby/jruby-complete/1.6.5.1/jruby-complete-1.6.5.1.jar:/home/dfukdev/.m2/repository/info/cukes/cucumber-junit/1.0.0.RC13-SNAPSHOT/cucumber-junit-1.0.0.RC13-SNAPSHOT.jar:/home/dfukdev/.m2/repository/org/mockito/mockito-all/1.9.0/mockito-all-1.9.0.jar:/home/dfukdev/.m2/repository/junit/junit/4.10/junit-4.10.jar:/home/dfukdev/.m2/repository/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar:"/>
    <property name="java.vm.specification.version" value="1.0"/>
    <property name="sun.arch.data.model" value="64"/>
    <property name="java.home" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre"/>
    <property name="sun.java.command" value="/home/dfukdev/src/github/cucumber-jvm/jruby/target/surefire/surefirebooter2577422097098355850.jar /home/dfukdev/src/github/cucumber-jvm/jruby/target/surefire/surefire6280478816897999201tmp /home/dfukdev/src/github/cucumber-jvm/jruby/target/surefire/surefire6518901775569516441tmp"/>
    <property name="java.specification.vendor" value="Sun Microsystems Inc."/>
    <property name="user.language" value="en"/>
    <property name="java.vm.info" value="mixed mode"/>
    <property name="java.version" value="1.6.0_26"/>
    <property name="java.ext.dirs" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ext:/usr/java/packages/lib/ext"/>
    <property name="sun.boot.class.path" value="/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/charsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes"/>
    <property name="java.vendor" value="Sun Microsystems Inc."/>
    <property name="localRepository" value="/home/dfukdev/.m2/repository"/>
    <property name="file.separator" value="/"/>
    <property name="java.vendor.url.bug" value="http://java.sun.com/cgi-bin/bugreport.cgi"/>
    <property name="sun.cpu.endian" value="little"/>
    <property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
    <property name="sun.desktop" value="gnome"/>
    <property name="sun.cpu.isalist" value=""/>
  </properties>
  <testcase time="0.004" classname="cucumber.runtime.jruby.test.Sub1" name="test_that_fails_sub1">
    <failure message="I&apos;m Sub 1" type="java.lang.AssertionError">java.lang.AssertionError: I&apos;m Sub 1
    at org.junit.Assert.fail(Assert.java:93)
    at cucumber.runtime.jruby.test.Sub1.test_that_fails_sub1(Sub1.java:10)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:24)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:35)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:115)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
    at $Proxy0.invoke(Unknown Source)
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
</failure>
    <system-out>@HL
@HL-------------------------------------------------------
@HL T E S T S
@HL-------------------------------------------------------
@SLRunning cucumber.runtime.jruby.test.SuiteTest
</system-out>
  </testcase>
  <testcase time="0" classname="cucumber.runtime.jruby.test.Sub1" name="test_that_passes_sub1"/>
  <testcase time="0" classname="cucumber.runtime.jruby.test.Sub2" name="test_that_passes_sub2"/>
  <testcase time="0.001" classname="cucumber.runtime.jruby.test.Sub2" name="test_that_fails_sub2">
    <failure message="I&apos;m Sub 1" type="java.lang.AssertionError">java.lang.AssertionError: I&apos;m Sub 1
    at org.junit.Assert.fail(Assert.java:93)
    at cucumber.runtime.jruby.test.Sub2.test_that_fails_sub2(Sub2.java:14)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:24)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:35)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:115)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.maven.surefire.booter.ProviderFactory$ClassLoaderProxy.invoke(ProviderFactory.java:103)
    at $Proxy0.invoke(Unknown Source)
    at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:150)
    at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcess(SurefireStarter.java:91)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:69)
</failure>
  </testcase>
</testsuite>

So it turns out that one XML file is created per test class, regardless of whether it's a suite, a cucumber test or a regular test.

Let's back up a little. Can you explain why you need a different output format? Is it because of the way you share results with the rest of your team?

Have you considered using the HTML formatter to do this?

@trnl
trnl commented Jan 24, 2012

@aslakhellesoy yeah, if you are combining tests with @Suite it will produce a single xml, because in this case you are assuming that this tests can be grouped. But if you run multiply classes with methods annotated with @Test - xml per each file will be created.

Let's back to cucumber-jvm. When you using @Feature(value="somepackage") - that means not grouping all tests form somepackage. This means what features to run (similar when you running multiply classes with @Test), so I believe xml per feature should be created.

This is for jenkins, at this moment all scenarios are flatten.

@trnl
trnl commented Jan 24, 2012

Also, please take a look how result looks for me: IDEA 11, OSX Lion. 1.0.0.RC12

screenshot

@aslakhellesoy
Cucumber member

Regarding your last image - please open a separate ticket for that, and explain what you expected to see. It's not related to this issue.

@aslakhellesoy
Cucumber member

You didn't answer my question about whether you can use the HTML formatter to achieve what you want. (This would require #160 to be implemented first).

If you want one JUnit class to output more than one XML file I think you need to modify Surefire, not Cucumber. -But I'd be happy if you prove me wrong with a pull request.

@trnl
trnl commented Jan 27, 2012

@aslakhellesoy we can't use html reports, because junit reports is necessary for jenkins or any other CI tool.

I've tried to dig into the code but haven't found any possible solution yet.

As for last image - Scenario node duplicated because of Background section in feature file. I'will open the new issue for that.

@aslakhellesoy
Cucumber member

Why are junit reports necessary for jenkins? Who reads them, what information do they provide and what do you do with that information?

Just trying to understand the underlying need here.

@trnl
trnl commented Jan 27, 2012

We've got a project, where we need to test system in various languages. We're writing tests in cucumber and then implement them via various languages: php(behat), java(cuke4duke), python(lettuce) and later ruby, javascript, etc.

So jenkins for us is a system where we see the overall results of testing. Some screenshots for cuke4duke (sorry, some fields are blurred):

1
2
3

@aslakhellesoy
Cucumber member

Am I right in assuming that it is important for you to see the trend over time?

In what way does the current JUnit output not give you similar charts?

Finally - do you agree with me that the inability to create a file-per-child of a test suite's children is a Maven Surefire shortcoming and not a Cucumber-JVM shortcoming?

@trnl
trnl commented Jan 27, 2012

Yes, not only the trend, but the entire results too - what is failed and why.

First problem is that features are not mentioned in junit report for cucumber-jvm. We can see only scenarios<->steps, but not features. For in jenkins report I can't even find out to what feature failed scenario belongs to.

As for file-per-child, yes, looks like you are right, but I'm still hoping :)

As a solution - we can add a parameter to @Feature (@Opts) annotation where we can set the depth of the output level.
For ex:

  • 3 - classname=Scenario, name=Step
  • 2 - classname=Feature, name=Scenario

-vova

@trnl
trnl commented Jan 27, 2012

Also may be we can produce a report where the root element will be <testsuites>? From my point of view this will be the most correct variant.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
   <testsuite name="Datacenters">
      <testcase classname="Datacenters"/>
      <testcase classname="Datacenters"/>
   </testsuite>
   <testsuite name="Network">
      <testcase classname="Network"/>
   </testsuite>
</testsuites> 

-vova

@AnubhavQA

Hey guys @trnl , @aslakhellesoy
I am quite new to cucumber jvm, and is facing same issue of tracking exact no. of features ran via Junit on Jenkins, but getting count of no. of steps performed (Given, When, Then). Do you guys came up with any solution of getting count of scenarios performed on Jenkins.

@aslakhellesoy
Cucumber member

I'm closing this ticket since support for this must be implemented in Maven's surefire, not cucumber-jvm.

@robertoandrade

Just bumped into this while trying to modify cucumber-jvm to report the hierarchy of its test in a way most jUnit consumers would expect.

My two cents based on my previous jUnit experience and Testing Theory: http://en.wikipedia.org/wiki/Classification_Tree_Method

  • jUnit expects the hierarchy of tests to be represented as:
    • Test Suites
      • Test Cases
        • Name: describing the test case itself
          • Traditionally this would be the name of a method in a test case class in Java
          • This is free form and should describe the whole test - the details such as every pre-condition, assertions, etc. are hidden as far as jUnit is concerned.
        • Classname: describing the organizational hierarchy (classifications) of the test case container (class)
          • Things like "packages" within the suite - subfolders or other grouping mechanism used to organize the different test cases logically together - is what's used to describe the classification of the test cases.

with that said, tracing parallels between standard unit testing Suite, Case, Naming and Classification and the integration testing correspondents provided by cucumber-jvm:

  • Traditional Java Unit Test (assuming jUnit 4 syntax Java tests):

    • Test Suite: defined in MyGroupOfTestCasesSuite.java (extends jUnit's Suite class and defines which test classes/methods to run)
      • Test Case: defined in logical/place/that/groups/similar/test/cases/MyFeatureTest.java
        • Name: myGreatTestCaseThatMakesSureSomethingWorks defined in @Test myGreatTestCaseThatMakesSureSomethingWorks()
        • ClassName: logical.place.that.groups.similar.test.cases.MyFeatureTest

jUnit Renderers such as the one in Eclipse, IDEA, Surefire and others, will translate, Suite, Classification and Naming into the hierarchy you see.

for cucumber-junit (which is the module in question) to output the feature files' execution state into a junit compatible hierarchy I would suggest the best mapping to be:

  • Moderately organized cucumber files and scenarios:

    • Test Suite: defined in MyGroupOfTestCasesSuite.java (defines @RunWith( Cucumber.class ) and @Cucumber.Options() to determine which features/scenarios to run)
      • Test Case: defined in each logical/place/that/groups/similar/tests/cases/my-feature-test.feature
        • Name: Scenario/Scenario Outline+Example: My Great Test Case That Makes Sure Something Works
      • ClassName: logical/place/that/groups/similar/tests/cases/my-feature-test (or if you somehow organize your .feature files' headers, describing each feature in a prefixed/organized manner, i.e.: Feature: Logical Place That Groups Similar Test Cases. My Feature Test - for a friendlier output, but regardless, this defines which feature contains the test case)

In short, traditional jUnit result reporting, assumes each test case to be a scenario and each class (alongside its classification) to be the unit that groups all cases together, using a dot-notation name to organize its hierarchy under the className portion of each result being posted.

I'm modifying my forked version of cucumber-jvm (more specifically cucumber-junit) to post to jUnit a hierarchy that matches what it would normally expect from any other "unit" testing runtime (key word here being "unit" - which I translate to be the "feature" in cucumber land, then "test cases" for that "unit" are the actual scenarios, and the "class" of that "unit" is the hierarchical location of that "feature" in the suite).

The "steps" as defined in many testing methods, normally do not surface other than in stack traces and failure reports to highlight in what "part" of the execution of such "test case"/"scenario" did it fail. It might want to log which step it is on, but it should not be part of the hierarchy of the Test Tree cucumber is forming for jUnit to render.

i guess that was way more than 2 cents, but I've been dwelling with this for months now and thought since the ticket was closed, it was a proper location to voice what I believe is not only my view but from many others in the community, and yes, I will go ahead and make the change ou my fork and if it interests anyone or the project itself, I'm happy to PR it back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.