Parallel Threaded Execution of Scenarios

[slaout]

Summary

Cucumber does not offer a parallelization option out of the box.

I'd like to get a Runtime option like "--threads 4".
The option will put all scenarios of all .features files in a queue.
As soon as a thread is idle, the thread will pick a scenario from the queue and execute it.

I'd also like to have a tagging mechanism to forbid some scenarios to run in parallel if they are prone to interfere with each other.

Current Behavior

Maven plugins or other external components tend to offer hacky ways to implement some sort of parallelism.
They almost all span multiple Cucumber processes, each one running a different .feature file, and then they run a process that merges all the JSON reports into one big report.json.

I'm not happy with this situation because I (anybody can) have .feature files that vary in number, length, and duration of scenarios (depending on varying things like network load required by each scenario).
This results in having one thread taking very long to execute a bigger .feature file while other threads have finished executing all other small .feature files.

Expected Behavior

Given the current behavior, I'd like the parallelism to be smarter, to be optimal about execution time, and a better thread occupation efficiency.

Possible Solution

For our project, I've made a quick fork of Cucumber with this feature, and we use it with success:
https://github.com/slaout/cucumber-jvm/tree/parallel-scenarios-execution-1.2.4

It has the following additions:

• --threads parameter to tell Cucumber how much executor threads must be spanned
• At the begin of the execution, all .feature files are scanned and all scenarios are put in a queue
• Each thread picks a scenario to execute from the queue, and as soon as it finished, it picks another scenario from the queue
• Some scenario needs to be synchronized:
  • For instance, say two scenarios begin with setting the online stock of a product before testing it: they must be executed sequentially for no interference
  • Say two other scenarios are modifying the in-store stock of a product before testing it: the two of them must also be executed sequentially BUT they can perfectly be executed in parallel with the first two scenarios that modify the online stock
  • To get that behavior, all four scenarios are tagged with a tag starting with "@synchronized-". In this case, we will tag the two first scenarios with @synchronized-online-stock and the two others are tagged with @synchronized-store-stock
  • Cucumber will group all scenarios with the same @synchronized- tag name in one "execution group" and will place them first in the queue, because they contain several not-parallelizable scenarios, so executing them first is more efficient if one group is quite large (imagine this large group being at the bottom of the queue: we would end up with the poor parallelization efficiency discussed in "Current Behavior" with other threads having finished their job and being idle)
• Not implemented, but we could imagine the order of scenarios in the queue could be randomized, so if there are potential concurrency problems, they get spotted quicker, and developers can either fix the scenarios (and ensure scenarios are independent from each other, not just working by luck of current schedule) or add "@synchronized-" tags to concurrent scenarios.
• Not implemented too, but some heuristics could put scenarios with a lot of steps at the start of the queue, as they could be longer to execute, which could lead to poorer threading efficiency (but as the gain could be marginal, it could as well not be implemented, it's just an idea)

The fork is launching multiple theads: each scenario has its own Formatter and Reporter: executing a scenario will "print" to those formater and reporter: hey will just record method calls in memory.
At the end, I list all scenarios in the order they are declarer and I tell the formater and reporter of these scenarios to replay what they memorized into the "real" formatter and reporter to construct the report.json exactly the same as it would have been created without parallelism.
To not change the code too much, I heavily added ThreadLocal variables and a few synchronized blocks where needed.
A proper refactor of all the code base will be needed for it to be cleaner, but I read somewhere you were already planning to do so for the next major version of Cucumber.

It's developed mainly for the Java backend, but I think it would need basic adjustments to expose the new threads option to other languages.
It's done for the CLI runner.
It should work for the JUnit runner too, but because all reports are stored in memory and merged at the very end, the JUnit view of the IDE will not move in real time, and it will change all at once at the end.

Note: this fork also contains the @ContinueNextStepsFor({Throwable.class}) evolution seen on issue #79.
Because our project needs both parallel execution and continuing execution on Then step failures.

You can see the changes introduced by the fork with this comparison:
slaout/cucumber-jvm@continue-next-steps-for-exceptions-1.2.4...slaout:parallel-scenarios-execution-1.2.4
Mainly:

• ThreadLocals
• synchronized blocks
• added PlaybackFormatter (implements Reporter, Formatter): records all events of the current thread and is able to play them back to another instance that will merge the reports of all scenario executions in the expected order
• added ProxyFormatter (implements Reporter, Formatter): just for debug purpose: will be removed
• ScenarioExecutionRunnable.java
• ThreadedRuntime.java

[danielwegener]

Since you mentioned cucumber-jvm, here is a longish issue describing what changes are nesessary to make parallel execution of cucumber scenarios work: cucumber/cucumber-jvm#630
A possible problem with your suggestion is that the glue code instances (provided by the di framework of your choice) as well as "world" instances (as in groovy) won't be scenario scoped and therefore be subject to racing issues (whether it's a good idea or not to keep state in the glue) (at least thats what I understood from a quick look through your changes).
Unfortunately the issue mentioned above has been closed (not even with a "wont fix because: ..." but rather a "just write faster tests").

[slaout]

Yes, we don't store data in our glue code: we have a special ScenarioScopedData class managing the multi-threading issues of data stored per scenario / thread.

OK for closing the issue, but a lot of your users are using Cucumber with Selenium, and Web tests are not fast by design, it cannot be changed.

So we will continue to use our own fork of Cucumber.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs.

[danielwegener]

Still an issue. Still annoying. ping.

[aslakhellesoy]

The core team doesn't currently have bandwidth to work on this. You can help by submitting a PR, or by offering financial support: https://cucumber.io/blog/2017/10/03/invest-in-cucumber

[danielwegener]

I think the topic is too big for single PRs but requires a good vision and coordination (as discussed in cucumber/cucumber-jvm#630). It is okay if it is not a priority right now but I just wanted to prevent the issue from being closed due to inactivity. Because it still exist and because there are still users out there who would love to have it fixed :)

[aslakhellesoy]

This is the sort of feature that isn't done in a jiffy. It would probably involve a considerable amount of changes to the codebase (whether it's ruby, java or javascript), and could easily take weeks of work, and involve several people.

This is the kind of work that is only likely to happen if it's funded through the opencollective.

In order to incentivise more organisations and people to provide financial support for big efforts like this, I plan to set up a voting scheme similar to what webpack has done.

Sponsors gets more votes, increasing the likelihood of people picking up what they voted/paid for.

[enkessler]

Some thoughts:

• An internal tool (e.g. CLI parallel runner options) would have to be made for each version of Cucumber because they are different internal languages whereas external tools (e.g. feature file splitters and report aggregators) can work with all versions of Cucumber because they all use the same Gherkin language and standard (mostly) formatters. That is a large multiplication of effort and maintenance.

• Testing at such scale that parallelization becomes critical is also the kind of situation where people tend to be rolling their own CI pipelines anyway, so the odds of there being more than one Cucumber process anyway on lots of machines (as opposed to one process with loads of threads on one machine) goes up. Smaller, composable tools become a lot more valuable than one tool tool that tries to do everything.

• If you are not satisfied with existing external tools, make better ones. That's what most of my open source projects are. :)

[boaty82]

There is a parallel runner in this repo, but as it states only java backend has been tested. But I've used this on projects with java & groovy and all was OK.

https://github.com/djb61/parallel-cucumber-jvm

[seanleblanc]

What version of cucumber-groovy/cucumber-java did you use @boaty82 ?

[Susirya]

@seanleblanc, this tool uses cucumber-jvm 1.2.2(gherkin-2.12.2) 
see new PR over here (if you haven't found it yet) cucumber/cucumber-jvm#1357

[boaty82]

@seanleblanc not sure if you mean my forked repo (and recent PR) or the earlier quoted library

my fork of cucumber-jvm is working off of the latest code. I hope it will be reviewed soon and I either get feedback or it is considered to be merged in

parallel-cucumber-jvm uses cucumber-jvm 1.2.5 (I've used the groovy plugin along side this) - I did look into upgrading this to support the newer libraries, but it was messy at best and didn't account for the comments around performance e.g. not performing expensive operations multiple times

[aslakhellesoy]

This is being worked on for Cucumber-JVM over at cucumber/cucumber-jvm#1389 so I am closing this.

[mdrasul]

Hello Quick Question
Trying to add the threads = 3 inside Junit runner not working check the screen shot
let me know what i am making mistake

@RunWith(Cucumber.class)
@CucumberOptions(
features = { "src/test/java/samplefeature/sampletest.feature" },
glue = { "samplesteps" },
tags = { "@regression"},
plugin = {"pretty"}, threads = 3
)

[slaout]

Hello @mdrasul,

With the parallel fork, scenarios are executed out of order. The produced report.json is aggregated as if all scenarios were ran in sequential order. As a result, this aggregation can only take place at the very end of the process, when all scenarios ran. For this reason, parallel run is not supported for IDE JUnit panels. I should probably remove the "threads" parameter in @CucumberOptions. Or add some complicated logic to report scenarios in order as soon as a direct chain of scenarios is done, but with @synchronized-* scenarios are run first (to optimize parallelism), this complicated logic could prove useless in several cases.

Parallel is only supported with the CLI runner.

Also, I released a final release (non-SNAPSHOT) a few days ago: be sure to use 1.2.4 instead of the old 1.2.4-SNAPSHOT.

[slaout]

@danielwegener Regarding your comment of 10 Dec 2016, the new version I released a few days ago fixes this for Spring: the Spring DI layer is now thread-safe: each thread has its own Glue scope instance.

Also, from what I see in:

• Support concurrent executions of scenarios cucumber-jvm#630
• Parallel support cucumber-jvm#1357
• [Core] Parallel cukes cucumber-jvm#1389

... Cucumber now supports parallelization. Not on a scenario or pickle (or even rule, I've discovered) level yet, but at a feature level.

And not with @synchronized-* annotations (but this one, I understand you would not want to support this requirement).

That's a very good news! Congrats for all the work done these last months.
And for the big refactoring that will come next: https://docs.google.com/document/d/12Y3Ut-SVSGdw9L_aW6NGQY9JaCcn3z0alY4d7kzOndU .

[mpkorstanje]

... Cucumber now supports parallelization. Not on a scenario or pickle (or even rule, I've discovered) level yet, but at a feature level.

This depends on the runner you use. The CLI and TestNG support parallel execution of pickles, JUnit per only supports parallel execution of features. I would have liked to support the parallel execution of pickles by JUnit too but JUnit 4s architecture simply does not allow it.

[slaout]

Oh, thanks for the precision, that's wonderful (and feature-level is already very helpful in JUnit, if used only during development).
Do you know if it would be possible in JUnit 5?

[mpkorstanje]

Being a test runner Cucumber would use the JUnit Platform by implementing the JUnit Engine rather then integrate with JUnit Jupiter.

I am currently not aware of anything in the JUnit Engine that would prevent it.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.