Apologies, I'm writing quite a long post, because I'm not sure what the answer to the issue might be, if I have some ideas maybe I can condense this down
I've been having a look at the parallel running from surefire 2.12, which seems a bit different to the 2.10 version.The 2.10 stuff does give 0 results as mentioned here. However the 2.12 runner gives a NullPointerException:
Caused by: org.apache.maven.surefire.testset.TestSetFailedException: java.lang.NullPointerException; nested exception is java.lang.NullPointerException: null
... 9 more
Caused by: java.lang.NullPointerException
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
Having a bit of a dig around in the surefire runner stuff (this is the first time I've dug around in here - so I could be mistaken), it seems that it very much expects the test descriptions to be set up in a tree like this:
|-(test class name)
|- (test method)
In that it sets up a Map of TestSets to hold the results of the test runs, which are keyed by the name of the class.
When the test descriptions come in from cucumber-jvm it has this sort of tree:
|- (Feature class)
|- (Feature name)
|- (Scenario name)
|- (Given step)
|- (When step)
|- (Then step)
Which ends up giving a Map keyed by the Given, When, Then step names.
When the tests are run, things like the testStarted notifications are passed a 'Scenario name' Description
When it then tries to find the item in the Map to record against it doesn't find it (currently surefire looks for the Description passed in, and the name of the class that is running - neither of which are keys in the Map created for Cucumber-JVM results). Which gives the NullPointer (this was the last bit posted here - #86).
I'll start having a think about how to make this work, but just thought I would post this in for now, just for consideration, given there seem to be a few bits floating around about the JUnit/Cucumber-JVM integration.
Hope this is of use, and that I've been clear explaining the issue, and question/thoughts let me know.
I had a little hack trying to add class names to the Description (https://github.com/mattharr/cucumber-jvm/tree/master/examples/java-parallel). But its still a not right - its looking for method names now. Will look to have a further play with this later, unless someone has a better thought on how to get this working?
I've now got this working here: https://github.com/mattharr/cucumber-jvm/tree/master/examples/java-parallel
(The tests should run, with 1 (intentional) failure in the simple standard junit test)
So this now works for the Surefire and Failsafe runner.
It does mean a few changes, and raises a few questions:
a) Doing this has meant some changes to Description - I know there's some work going on in #322,, not sure if it jibes with this. In that this change is really trying to get class names into the Description?
b) One of the tweaks is to stop the scenarios reporting successful completion - is this needed? The Surefire reporter bails when this occurs.
c) In relation to (b), what should the test counts be counting? Is it all the Given, When, Then steps? Or should it include the Scenarios too? At the moment it seems to be the latter is this intentional?
Any plan on merging this in?
It would be very useful to be able to run tests in parrallel... esp dealing with selenium
@joshhughes @mattharr there is no pull request for me to merge in yet.
For those just looking for an acceptable workaround, the following works:
Oddly, this doesn't actually exclude RunCukesTest.
However, it does work around issue #333
However, it only works for parallel classes.
I am still not be able to run the tests in -Pparallel. Tests running in sequential fashion.
when I use excludes it does exclude my tests...
For me it is the same, can`t run them in parallel. Tried also with Surefire 2.14 where parallel execution is done with
Maybe it is an option to use TestNG instead of JUnit. Parallelizing there is absolutely easy and has more options than on JUnit. The only reason I don`t use cucumber is because of the missing parallelization. Need this for my selenium tests.
Is there any solution available Parallel running of scenario's in Cucumber-jvm ?
There is a pull request that might fix this issue: #554
Can anyone write a unit test or create a small project which would show parallel facilities based on #544 PR?
Fixed concurrency issue. Closes #333,#554,#591,#661