Skip to content

Parallel running with surefire 2.12 #333

Closed
mattharr opened this Issue May 31, 2012 · 12 comments

9 participants

@mattharr

Apologies, I'm writing quite a long post, because I'm not sure what the answer to the issue might be, if I have some ideas maybe I can condense this down

I've been having a look at the parallel running from surefire 2.12, which seems a bit different to the 2.10 version.The 2.10 stuff does give 0 results as mentioned here. However the 2.12 runner gives a NullPointerException:

at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
Caused by: org.apache.maven.surefire.testset.TestSetFailedException: java.lang.NullPointerException; nested exception is java.lang.NullPointerException: null
Results :
        at org.apache.maven.surefire.common.junit4.JUnit4RunListener.rethrowAnyTestMechanismFailures(JUnit4RunListener.java:166)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:63)
        at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:139)
        ... 9 more
Caused by: java.lang.NullPointerException
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
        at org.apache.maven.surefire.junitcore.ConcurrentReporterManager.testStarting(ConcurrentReporterManager.java:106)
        at org.apache.maven.surefire.common.junit4.JUnit4RunListener.testStarted(JUnit4RunListener.java:87)
        at org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:87)
        at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:41)
        at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:84)
        at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:45)
        at cucumber.junit.JUnitReporter.startExecutionUnit(JUnitReporter.java:47)
        at cucumber.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:74)
        at cucumber.junit.FeatureRunner.runChild(FeatureRunner.java:65)
        at cucumber.junit.FeatureRunner.runChild(FeatureRunner.java:20)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)

Having a bit of a dig around in the surefire runner stuff (this is the first time I've dug around in here - so I could be mistaken), it seems that it very much expects the test descriptions to be set up in a tree like this:

(null description)
     |
     |-(test class name)
          |
          |- (test method)

In that it sets up a Map of TestSets to hold the results of the test runs, which are keyed by the name of the class.

When the test descriptions come in from cucumber-jvm it has this sort of tree:

(null description)
     |
     |- (Feature class)
          |
          |- (Feature name)
               |
               |- (Scenario name)
                    |
                    |- (Given step)
                    |- (When step)
                    |- (Then step)

Which ends up giving a Map keyed by the Given, When, Then step names.

When the tests are run, things like the testStarted notifications are passed a 'Scenario name' Description

When it then tries to find the item in the Map to record against it doesn't find it (currently surefire looks for the Description passed in, and the name of the class that is running - neither of which are keys in the Map created for Cucumber-JVM results). Which gives the NullPointer (this was the last bit posted here - #86).

I'll start having a think about how to make this work, but just thought I would post this in for now, just for consideration, given there seem to be a few bits floating around about the JUnit/Cucumber-JVM integration.

Hope this is of use, and that I've been clear explaining the issue, and question/thoughts let me know.

@mattharr
mattharr commented Jun 1, 2012

I had a little hack trying to add class names to the Description (https://github.com/mattharr/cucumber-jvm/tree/master/examples/java-parallel). But its still a not right - its looking for method names now. Will look to have a further play with this later, unless someone has a better thought on how to get this working?

@mattharr
mattharr commented Jun 5, 2012

I've now got this working here: https://github.com/mattharr/cucumber-jvm/tree/master/examples/java-parallel

(The tests should run, with 1 (intentional) failure in the simple standard junit test)

So this now works for the Surefire and Failsafe runner.

It does mean a few changes, and raises a few questions:

a) Doing this has meant some changes to Description - I know there's some work going on in #322,, not sure if it jibes with this. In that this change is really trying to get class names into the Description?

b) One of the tweaks is to stop the scenarios reporting successful completion - is this needed? The Surefire reporter bails when this occurs.

c) In relation to (b), what should the test counts be counting? Is it all the Given, When, Then steps? Or should it include the Scenarios too? At the moment it seems to be the latter is this intentional?

Thanks,

@joshughes

Any plan on merging this in?

@joshughes

It would be very useful to be able to run tests in parrallel... esp dealing with selenium

@aslakhellesoy
Cucumber member

@joshhughes @mattharr there is no pull request for me to merge in yet.

@paulkrause88

For those just looking for an acceptable workaround, the following works:

            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.12</version>
                <executions>
                    <execution>
                        <configuration>
                            <parallel>classes</parallel>
                            <threadCount>100</threadCount>
                            <excludes>
                                <!--
                                    Oddly, this doesn't actually exclude RunCukesTest.
                                    However, it does work around issue #333 
                                 -->
                                <exclude>**/RunCukesTest.java</exclude>
                            </excludes>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

However, it only works for parallel classes.

@amoid123

Hi paulkrause88,

I am still not be able to run the tests in -Pparallel. Tests running in sequential fashion.

@joshughes

when I use excludes it does exclude my tests...

@srr84
srr84 commented Apr 14, 2013

For me it is the same, can`t run them in parallel. Tried also with Surefire 2.14 where parallel execution is done with

<forkCount>5</forkCount>
<reuseForks>false</reuseForks>

instead of

<parallel>classes</parallel>
<threadCount>100</threadCount>

Maybe it is an option to use TestNG instead of JUnit. Parallelizing there is absolutely easy and has more options than on JUnit. The only reason I don`t use cucumber is because of the missing parallelization. Need this for my selenium tests.

@sureshnature

Hi,

Is there any solution available Parallel running of scenario's in Cucumber-jvm ?

Regards,
Suresh

@bjoernmartin

There is a pull request that might fix this issue: #554

@ffbit
Cucumber member
ffbit commented Aug 24, 2013

Can anyone write a unit test or create a small project which would show parallel facilities based on #544 PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.