Configure parallelism to run at most N methods per class in parallel, while running all classes in parallel

[DJ-Glock]

Discussed in #3513

^{Originally posted by DJ-Glock October 23, 2023}

I use cucumber with JUnit.

I have several feature files. Each feature file is designed to test one application instance: A.feature, B.feature, C.feature. Each feature file contains one scenario template(outline) with N examples (usually more than 10).

Application instances are independent from each other. And each application instance can process input data in M threads, usually 2. So it does not make sense to run scenarios for particular application instance in more than M (2) threads, because it will lead to flaky tests and unstable execution time.

In the same time running scenarios for one instance (with execution-mode.feature=same_thread) in one thread can take much time.

What do I want to achieve:

• Run feature files in parallel.
• Run scenarios from particular feature file in parallel, but limited to 2 simultaneously running scenarios.
• Overall threads limit should be equal to CPUs count or fixed pool size, let's say 6 (parallel.config.fixed.max-pool-size=6)

Example:

Being executed simultaneously:

A.feature
- scenario 1
- scenario 2

B.feature
- scenario 1
- scenario 2

C.feature
- scenario 1
- scenario 2

Waiting in the queue:

A.feature
- scenario 3
- scenario 4
...
- scenario 10

B.feature
- scenario 3
- scenario 4
...
- scenario 10

C.feature
- scenario 3
- scenario 4
...
- scenario 10

Graphical visualization for better understanding:

I suppose that it may be achieved using CustomStrategy:
https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#parallel-execution

To control properties such as the desired parallelism and maximum parallelism, Cucumber supports JUnit 5's ParallelExecutionConfigurationStrategy. Cucumber provides two implementations: dynamic and fixed that can be set through cucumber.execution.parallel.config.strategy. You may also implement a custom strategy.

But it could be tricky especially with the fact that I do not understand for now, at what point of time these properties are being set. And could they be changed dynamically (another words based on feature files configuration, not only pom.xml).

My idea was to try using tags on scenario level, something like @feature=A, @feature=B etc to limit parallelism for pool of scenarios. But I'm on the beginning of my journey, so need an advice if it's possible or not.

Question on SO: https://stackoverflow.com/questions/77388756/configure-parallelism-on-features-and-scenarios-level-in-the-same-time-cucumbe

After discussion on SO with @mpkorstanje, he advised that with current implementation of JUnit 5 it's not possible to do it, quote:

JUnit 5 handles parallel execution for Cucumber. What you are looking to do is currently not possible. While JUnit 5 does support exclusive resources, it does not support resource pools. You would have to ask the JUnit 5 team if they would support this and possibly provide a pull request.

But you could implement this in your own code with something like https://commons.apache.org/proper/commons-pool/

Then for each system you're testing against, the pool should contain 2 tokens.

@Resource1
Feature: Example 1

Scenario Outline: Examples

@resource2
Feature: Example 2

Scenario Outline: Examples

Then before starting the scenario you block and acquire the resources from the pool.

static ObjectPool<?> pool = ...

@Before
public void before(Scenario scenario) {
    var tags = scenario.getTags(); // Will contain either @Resource1 or @Resource2.

    // block and acquire a resource from the pool
}

@After
public void after(Scenario scenario) {
    var tags = scenario.getTags();

    // return a resource to the the pool
}

You would have quite a few threads waiting to acquire a resource from the pool, and definitely have more threads then CPUs, which is not pretty, but I do reckon it is manageable.

I also tried another approach: tried to set up several Suites/TestRunners (one test runner per feature(s) I want to run per app instance) with configured parallelism for each runner, for example:

@Suite
@IncludeEngines("cucumber")
@ConfigurationParameter(
    key = "cucumber.execution.parallel.enabled",
    value = "true"
)
@ConfigurationParameter(
    key = "cucumber.execution.parallel.config.strategy",
    value = "fixed"
)
@ConfigurationParameter(
    key = "cucumber.execution.parallel.config.fixed.max-pool-size",
    value = "3"
)
@SelectClasspathResource("cucumber/folder/with/some/features")
@ConfigurationParameter(
    key = "cucumber.plugin",
    value = "pretty"
)
@ConfigurationParameter(
    key = "cucumber.execution.parallel.config.fixed.parallelism",
    value = "2"
)
class RunCucumberTest

But it does not met my requirements because test runners are run sequentially.

I'm not very experienced in java programming, so if someone from the community is willing to try implementing this mechanism, it would be great.

[marcphilipp]

For Jupiter, the closest thing you could do would be to run test classes in parallel but methods in each class sequentially as documented in the User Guide via:

junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent

I took a quick look at the Cucumber test engine and it doesn't seem to provide equivalent options. Could you please open an issue in the cucumber-jvm repo instead?

[DJ-Glock]

@marcphilipp well, it works this way with same_thread value.

With cucumber.execution.execution-mode.feature=same_thread - it works as you described, test classes run in parallel, methods in test class run sequetially.
With cucumber.execution.execution-mode.feature= concurrent - all methods will run in parallel.

But I would like to have a combined behaviour.

[DJ-Glock]

For Jupiter, the closest thing you could do would be to run test classes in parallel but methods in each class sequentially as documented in the User Guide via:

junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent

I took a quick look at the Cucumber test engine and it doesn't seem to provide equivalent options. Could you please open an issue in the cucumber-jvm repo instead?

Hi @marcphilipp
Can you please check my last previous comment? I believe you closed it wrongly.

[marcphilipp]

As pointed out by @mpkorstanje, limited concurrent execution within one test class/feature file is currently not possible. Would it help if you implemented that behind the scenes when accessing the services? For example, you could add a decorator around your service that would block calls when the limit has been reached.

[DJ-Glock]

Ah, unfortunately adding decorators like this is not what I am looking for. Application is safe, it won't process more than it could. The issue is in tests - I want them to be stable, efficient and with predictable execution time. If I go this way, I'll need to increase timeouts a lot and it will be hard to predict execution time for each particular test. It will be a mess.

[github-actions]

If you would like us to be able to process this issue, please provide the requested information. If the information is not provided within the next 3 weeks, we will be unable to proceed and this issue will be closed.

[github-actions]

Closing due to lack of requested feedback. If you would like to proceed with your contribution, please provide the requested information and we will re-open this issue.