Can no longer limit the number of parallel sessions from JUnit 5

[titusfortner]

💥 Regression Report

Can no longer limit the number of parallel sessions from JUnit 5 (v3.0.0-M5)
Likely related to this ticket: junit-team/junit5#2273

Last working Selenium version

Worked up to version: 4.0.0-alpha-3

Stopped working in version: 4.0.0-alpha-4

To Reproduce

This is with Maven Surefire plugin, but should be the same behavior for Gradle I think.
Run mvn test with a test suite that has this plugin config in the pom.xml file:

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
                <configuration>
                    <properties>
                        <configurationParameters>
                            junit.jupiter.execution.parallel.enabled = true
                            junit.jupiter.execution.parallel.mode.default = concurrent
                            junit.jupiter.execution.parallel.config.strategy = fixed
                            junit.jupiter.execution.parallel.config.fixed.parallelism = 3
                        </configurationParameters>
                    </properties>
                </configuration>
            </plugin>

Expected behavior

Only 3 driver sessions should be running at a time, instead, all tests in the suite are executed.

[pujagani]

Thank you for providing the details and the link. I was able to use https://github.com/rbok78/framework-template to recreate the issue.
After switching to later versions of Selenium, I observed that tests run as per the configured parallelism value. All the drivers are not started together as it happens with Selenium Alpha 4 and 5. Though, I do see UnreachableBrowserException exception till Selenium Alpha 7.
However, I was unable to reproduce the issue from Selenium Beta onwards. All 3 beta versions were running drivers as per the configured parallelism value.

[titusfortner]

@pujagani thanks for looking into this. I created a reproducible project for you to look at with the latest beta 3 code:
https://github.com/titusfortner/bug9359

should be able to justgit clone / mvn run test

[pujagani]

Thank you for sharing the sample repo and the update. I was able to reproduce the error. After running the tests a considerable number of times, I see the error intermittently where the number of tests exceeds the parallelism value.

Upon digging deeper, I suspect the change in behaviour is due to the HTTP client used. Selenium Alpha-4 made a switch from OkHttp to AsyncHttpClient. This switch might have lead to a change in behaviour since both have very different threading models in the background.

JUnit leverages the fork-join pool to run the tests in parallel. Thread pools in Java have a core thread pool size and max thread pool size. The parallelism value is used for configuring the core thread pool size. Whereas the max thread pool size to kept to the recommended 256 + the parallelism value. https://github.com/junit-team/junit5/blob/43638eb6a870e0d6c49224053dfeb39dcf0ef33f/junit-platform-engine/src/main/java/org/junit/platform/engine/support/hierarchical/DefaultParallelExecutionConfigurationStrategy.java#L44
The fork-join pool can spawn threads up to the max thread pool size if some threads are blocking. (https://docs.oracle.com/javase/9/docs/api/java/util/concurrent/ForkJoinPool.html) Hence the parallelism is not guaranteed as mentioned in https://junit.org/junit5/docs/snapshot/user-guide/index.html#writing-tests-parallel-execution section.

A workaround for this is to have a custom strategy to configure the thread pool size.

Providing an example below:

package com.titusfortner.bug9359;

import org.junit.platform.engine.ConfigurationParameters;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfiguration;
import org.junit.platform.engine.support.hierarchical.ParallelExecutionConfigurationStrategy;

public class CustomStrategy implements ParallelExecutionConfiguration, ParallelExecutionConfigurationStrategy {

    @Override
    public int getParallelism() {
        return 3;
    }

    @Override
    public int getMinimumRunnable() {
        return 3;
    }

    @Override
    public int getMaxPoolSize() {
        return 3;
    }

    @Override
    public int getCorePoolSize() {
        return 3;
    }

    @Override
    public int getKeepAliveSeconds() {
        return 30;
    }

    @Override
    public ParallelExecutionConfiguration createConfiguration(final ConfigurationParameters configurationParameters) {
        return this;
    }
  }

Update the pom.xml as follows:

 junit.jupiter.execution.parallel.enabled = true
  junit.jupiter.execution.parallel.mode.default = concurrent
  junit.jupiter.execution.parallel.mode.classes.default = concurrent
  junit.jupiter.execution.parallel.config.strategy = fixed
  junit.jupiter.execution.parallel.config.strategy = custom
  junit.jupiter.execution.parallel.config.custom.class = com.titusfortner.bug9359.CustomStrategy

I suggest please try the above changes and let me know if this is a workable solution. Appreciate the feedback. Thank you!

[titusfortner]

Ah, I didn't realize alpha 4 is when we switched to the new AsyncHttpClient that completely makes sense, then.

So, yes, this now works for "3"
What if I want to do "12?"
The number that runs at any given time is kind of all over the place, plus I'm getting

org.openqa.selenium.WebDriverException: Timed out waiting for driver server to stop.

Which indicates that the parallel is kind of all over the place.

I've updated the repo with the code you suggested. I'm using values based on system properties I'm setting in the pom.

I'm not familiar with the differences between the values getting set, so perhaps those shouldn't all be the same?

[pujagani]

Thank you for the quick feedback and for trying out the sample configuration.

org.openqa.selenium.WebDriverException: Timed out waiting for driver server to stop. This is something I have seen as well and I know we set a timeout for 20 seconds

selenium/java/client/src/org/openqa/selenium/remote/service/DriverService.java

Line 214 in 64447d4

.get(getTimeout().toMillis() * 2, TimeUnit.MILLISECONDS);

but not sure why that is not sufficient when running parallelly. I will look into this next.

Regarding providing the configuration, ideally, the values should be using the available processors (using Runtime.availableProcessors()) on the machine to work well.
Parallelism and core pool size are usually the same value. I suggest setting it to Runtime.availableProcessors()
Maxpoolsize and core pool size are the same because we want to have a strict upper bound of threads that can be created.
Otherwise, more threads than parallelism value can be created up to max pool size. Hence, the example configuration has all the same values.

[diemol]

Linked PR with a fix was merged and it will be available in beta-4.

@pujagani, could you please share the steps how to test this before the beta-4 release?

[pujagani]

Thanks, Diego!

Step 1
Pull in the latest changes by referring to https://github.com/SeleniumHQ/selenium/blob/trunk/CONTRIBUTING.md
Ensure you are on the trunk branch.

Step 2
Go to the selenium project folder and build the jar using
bazel build grid

The jar will be created in the following path :

<path-to-the-project>/selenium/bazel-bin/java/server/src/org/openqa/selenium/grid/selenium_server_deploy.jar

Step 3
Go the test maven project folder and run :
mvn install:install-file -Dfile=<path-to-the-project>/selenium/bazel-bin/java/server/src/org/openqa/selenium/grid/selenium_server_deploy.jar -DgroupId=org.seleniumhq.selenium -DartifactId=selenium-java -Dversion=4.0.0-beta-3 -Dpackaging=jar

Step 4
Build the test maven project:
mvn clean install -DskipTests

Now everything is in place with the new changes.

Step 5
Run the tests!

[titusfortner]

Yup, this was fixed in beta4. I can now tell people to use JUnit 5 again!

[dsmoons]

I have a similar problem. Please, tell me, what am I doing wrong?
Project to reproduce https://github.com/dsmoons/webdriver-bug

[pujagani]

I have a similar problem. Please, tell me, what am I doing wrong?
Project to reproduce https://github.com/dsmoons/webdriver-bug

Thank you for providing the test project. You might want to add a Custom Strategy and add the details to pom as directed in #9359 (comment). Please try it out and provide feedback if that helps.

[titusfortner]

@dsmoons you can see my repo reproduction here: https://github.com/titusfortner/bug9359/blob/main/pom.xml#L70

When the se version is changed to 4 beta this code works as expected

[dsmoons]

@pujagani Maybe this can be fixed without using a Custom Strategy? It seems to be Selenium's bug, not JUnit.
@titusfortner I am already using 4.0.0-beta-4.

[titusfortner]

The JUnit fix for running in parallel when a library is using ForkJoinPool.managedBlock is to use a Custom Strategy.
The Selenium fix for running properly with a Custom Strategy is in beta 4.

I linked to my repo so you can see the Custom Strategy example with working code.

[dsmoons]

@titusfortner I understand, thank you. I will use the Custom Strategy with Beta4.
But I mean, will I be able to use parallelization without Custom Strategy with the next versions?
Custom Strategy is still a crutch, not a solution (sorry).

[titusfortner]

No, it's a limitation of how JUnit 5 has implemented parallelization. The library that Java Selenium is using to manage asynchronous http calls (to support DevTools functionality), uses commands that JUnit assumes that no one will use. Custom Strategy is their recommended solution to avoid this.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

[titusfortner]

As pointed out in #10113

the solution only works for Java 11.
Java 8 does not respect the limit at all
the latest version of Java 8 actually completely chokes on it
Java 17 seems to run one set and then gives up.

Of interest, Selenium 4.0.0-alpha-3 does work with JUnit 5 in all Java versions, so it is related to what AsyncHttpClient is doing in combination with JUnit 5.

Perhaps this fixes things? junit-team/junit5#1858

[titusfortner]

Continuing discussion in #10113