Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce support for parallel test execution in declarative test suites #964

Closed
casid opened this issue Jul 18, 2017 · 28 comments
Closed

Comments

@casid
Copy link

casid commented Jul 18, 2017

I'm currently migrating a project from JUnit4 to JUnit5 (Branch: https://github.com/casid/mazebert-ladder/tree/junit5).

A feature of the JUnit4 tests is that I can run all unit tests in parallel. This is an essential feature since it a) chops down the test execution time significantly and b) prevents developers from adding shared state to unit tests.

In JUnit4 with the help of junit-toolbox:

@RunWith(ParallelSuite.class)
@SuiteClasses("**/*Test.class")
public class UnitTestSuite {
}

I wished JUnit5 to support something like this:

@Suite(parallel = true)
@SuiteClasses("**/*Test")
public class UnitTestSuite {
}

But I couldn't find anything that looked even slightly like it.

I think suites could greatly improve configuration options for test executions. In fact, it would remove the need for external tools to implement the same features (e.g. parallel execution / test discovery). You could simply point maven-surefire, gradle or IntelliJ to the TestSuite(s) you want to execute.

Integration tests could benefit from this too:

@Suite(parallel = true)
@SuiteClasses("**/*IT")
@IncludeTags("parallel")
public class ParallelIntegrationTestSuite {
    @BeforeClass
    public static void setUp() throws Exception {
        TestDataSourceProvider.instance.prepare();
    }

    @AfterClass
    public static void tearDown() throws Exception {
        TestDataSourceProvider.instance.dispose();
    }
}

I would love to have something like this in JUnit5! Currently I have some mixed feelings how to proceed with adopting JUnit5 (great new features like nested tests) vs JUnit4 (great test execution control).

Related Issues

@sormuras
Copy link
Member

sormuras commented Jul 18, 2017

Related to #60

@sormuras sormuras marked this as a duplicate of #60 Jul 18, 2017
@casid
Copy link
Author

casid commented Jul 18, 2017

I don't think that this duplicates #60. There's not much information on how to achieve parallel execution, except a reference to #90 - which I think suggests to delegate this to external tools.

Instead, the idea here is to let JUnit handle it. For instance with suites.

@sormuras
Copy link
Member

Understood. #60 still serves as a good umbrella issue for parallel, other thread, multi-thread feature requests.

@sbrannen
Copy link
Member

This is closely related to #744.

@sbrannen sbrannen added this to the 5.1 Backlog milestone Jul 19, 2017
@sbrannen sbrannen changed the title Feature Request: Suite support to configure test execution in Java Introduce support for parallel test execution in declarative test suites Jul 19, 2017
@sbrannen
Copy link
Member

I have reworded this issue's title accordingly.

@sbrannen
Copy link
Member

Added Related Issues section.

@casid
Copy link
Author

casid commented Jul 20, 2017

Thanks for looking into this. What's the status of #744? If things turn out to become more concrete, I would happily support the development.

@sbrannen
Copy link
Member

@casid, #744 is currently in the 5.1 backlog.

So the team won't begin working on it until after 5.0 GA has been released.

@sbrannen
Copy link
Member

If things turn out to become more concrete, I would happily support the development.

Well, you could of course start experimenting in your own branch, develop a proof of concept, etc.

@koretzki
Copy link

koretzki commented Oct 26, 2017

not sure if is part of the same issue,

I am trying to migrating a maven project from JUnit4 to JUnit5.
I am using maven-surefire-plugin and maven-failsafe-plugin to run tests.

When I added the needed dependencies in pom.xml file for the plugins,
the tests do run but they do not execute in parallel like they used to do without the dependencies (the plugins have the thread use in their configuration).
Without the dependencies maven won't pick up and run Junit jupiter tests but do run Junit Vintage tests in parallel.

Is there any support for parallel test executions in maven ?
if not, is it part of this issue or should there be a new issue ?

@marcphilipp
Copy link
Member

Regarding Maven: parallel is not supported at the moment, but forkCount should work.

@koretzki Can you please try that?

@koretzki
Copy link

koretzki commented Oct 26, 2017

Yes, it does work.
But I require it to run as threads, runing in forks might cause issue with our environment.
Do you know when it would be supported in Maven or Gradle ?

@marcphilipp
Copy link
Member

The plan is to handover the plugin to the Maven Surefire team. Please consult them regarding their roadmap. /cc @Tibor17

@JLLeitschuh
Copy link
Contributor

Is this currently supported in the gradle plugin? Is the team waiting for the Gradle team to take over the Junit5 gradle integration and embed it into the gradle tool by default.

@marcphilipp
Copy link
Member

Our Gradle plugin does currently not support parallel test execution. And, indeed, we're waiting for the Gradle team to start work on first-class support for JUnit 5. As a sidenote, there's also a prototype for parallel execution within the Jupiter engine, see #60.

@jamiewastin
Copy link

@marcphilipp - Now that there is gradle plugin support via the junit-platform-gradle-plugin does this pave the way for parallel test support?

My specific use-case is to be able to run each test in its own isolated JVM to ensure that static state is not carried between unit tests (legacy system). This seems closely related to parallel test support.

@JLLeitschuh
Copy link
Contributor

@jamiewastin
Consider using the new Junit 5 support that is now built into Gradle 4.6.
When configuring your test you can set the maxParallelForks on the Test task.
https://docs.gradle.org/current/javadoc/org/gradle/api/tasks/testing/Test.html#setMaxParallelForks-int-

@casid
Copy link
Author

casid commented Jun 14, 2019

The parallel test execution feature is exactly what I needed. This is an amazing and truly elegant solution!

For my use case this worked perfectly:

Create junit-platform.properties in src/test/resources:

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.mode.classes.default = concurrent

Add @Execution(ExecutionMode.SAME_THREAD) to classes where methods cannot be run in parallel.

The concurrent execution of test methods (not just classes) is amazing.

Thank you so much for adding this feature! I'm closing this issue now.

@casid casid closed this as completed Jun 14, 2019
@sbrannen
Copy link
Member

@casid, thanks for letting us know.

We are very glad to hear your excitement about the parallel test execution support.

@koretzki
Copy link

does it work with recent version of Maven Surefire/ Failsafe plugins ?
based on Maven docs here
https://maven.apache.org/surefire/maven-failsafe-plugin/examples/junit-platform.html
the presence of JUnit 5 tests triggers a test runner called "JUnit 5 Platform Engine"
which does not support running tests in parallel.
I tried to with the issue a number of times, even forcing a runner Junit47 to run.
could not make it to run parallel with Junit 5.
Junit 5.3-5.4 had the junit-platform-runner it did work. but alerts on depreciation on Maven, 2.22.0+ .
Any solutions to Parallel execution in coming Maven version / Junit 5.5 ?

@sormuras
Copy link
Member

@koretzki It does work with Maven Surefire/Failsafe.

Consult the User Guide how to a) enable parallel execution in Jupiter and b) how to configure Maven Surefire.

The Maven documentation states "Running Tests in Parallel -- From JUnit Platform does not support running tests in parallel." due to the fact that the <parallel>+*</parallel> has no effect. The JUnit Platform doesn't provide such a feature.

@Tibor17
Copy link

Tibor17 commented Jun 15, 2019

@koretzki
If you want to run the JUnit5 in Maven Surefire you have to add engine in dependencies, see https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html
In the next coming version 3.0.0-M4 adding the jupiter-api would be enough.

You can use the latest version 3.0.0-M3 of Surefire with parallel execution and the tests will be executed but the statistics in test result will be very messy as well as the log. We would like to fix it or rework some Surefire code in order to fix it in 3.0.0-M4.

@marcphilipp marcphilipp removed this from the 5.x Backlog milestone Jul 5, 2019
@koretzki
Copy link

Thanks @Tibor17 and @sormuras this helped my with surefire plugin to work.
I tried the same with the Maven Failsafe plug-in but it didn't work,
It ran but not in parallel.
Should it work in Failsafe, or it not supported yet ?

@Tibor17
Copy link

Tibor17 commented Sep 15, 2019

@koretzki The real execution should be parallel. The only problem is that sometime the reports are lost or incomplete. Is this happening?

@koretzki
Copy link

koretzki commented Jan 7, 2020

Hi @Tibor17, reports are ok, but Failsafe does not run in parallel. I took my parallel working surefire maven profile and replaced just it with the Failsafe plugin. Any printout that was with thread name changed from ForkJoinPool (names) to Main in all test printouts (same check with thread ids).
tried failsafe versions 2.22.2 to 3.0.0-M4.
Junit 5.5.2 and 5.6.0-RC1.
pom file configuration params :

<configurationParameters>
    junit.jupiter.execution.parallel.enabled = true
    junit.jupiter.execution.parallel.config.dynamic.factor = 1 
    junit.jupiter.execution.parallel.mode.default = concurrent
    junit.jupiter.execution.parallel.mode.classes.default =concurrent
</configurationParameters> 

@marcphilipp
Copy link
Member

@koretzki Does it work if you put those properties into src/test/resources/junit-platform.properties?

@koretzki
Copy link

koretzki commented Jan 7, 2020

@marcphilippI I did try it, but it did not work.

@GethDeeo
Copy link

GethDeeo commented Aug 18, 2022

@marcphilipp My bad to chime in, but this issue more or less describes my problem, and I saw that the last mention of you contains a typo :)

So I'm trying to reproduce a working parallel suites scenario from JUnit 4 to JUnit 5.
Ok, so I know that Surefire <parallel> is off the table.
In itself, parallel execution is working fine in JUnit 5.
In itself, suites are working fine in JUnit 5.
I see that junit-platform-suite implements TestEngine instead of HierarchicalTestEngine, and only the latter contains code for parallelism.

Are there any plans to change this? I have a large testbase so I desperately need parallelism, but within a "unit" (suite) a single threaded, fixed-order execution of test classes is a must. These are NOT unit tests, but functional tests with webdriver - the test classes/methods can be executed independently, but without shared states (same thread) and predetermined order, execution takes a very, very long time.
(I've somewhat succeeded with Spock by dynamically adding resource locks with package names (so package == suite), but Spock does not guarantee class order at all - and for optimal execution time, also the "suites" should be ordered.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants