Test suites not working with JUnit 5

[PE-test]

I was moving tests from JUnit 4 to JUnit 5. We are using test suites in our project, and there I got some weird behavior. I'm able to launch tests with JUnit 5 RC versions, but on release 5.1 tests does not start.

Project and POM structure same as here.

Here is a working POM:

    <junit.jupiter.version>5.0.0-M2</junit.jupiter.version>
    <junit.vintage.version>4.12.0-M2</junit.vintage.version>
    <junit.platform.version>1.0.0-M2</junit.platform.version>

When I update to latest release versions:

    <junit.jupiter.version>5.1.0</junit.jupiter.version>
    <junit.vintage.version>5.1.0</junit.vintage.version>
    <junit.platform.version>1.1.0</junit.platform.version>

Tests does not start, neither from Eclipse or Maven.

Could not find any mentions, that suites are not supported in latest release. So it seems like JUnit issue.

[marcphilipp]

What do you use for your test suites? @RunWith(Suite.class) @SuiteClasses({TestClass1.class, ...}) or @RunWith(JUnitPlatform.class)?

[PE-test]

We use:
@RunWith(JUnitPlatform.class)
@SelectClasses({Test1.class, ...})

Does it make sense which of them to use?

[marcphilipp]

From the Javadoc: The JUnitPlatform runner is a

JUnit 4 based Runner which runs tests on the JUnit Platform in a JUnit 4 environment.

It cannot be used when running on the JUnit Platform. You should be able to use the Suite runner, though.

[marcphilipp]

Moreover, there's #744 to introduce native suites for the JUnit Platform.

[jhovell]

@marcphilipp can you elaborate?

The current documentation says suites are supported via the syntax that @PE-test provides.

https://junit.org/junit5/docs/current/user-guide/#running-tests-junit-platform-runner-test-suite

Is the documentation incorrect? Is there a way to run the following command? If MySuiteOfTests is a test class, the tests run. If it is defined as a suite per the documentation, no tests are run.

mvn failsafe:integration-test  -Dit.test=MySuiteOfTests

Using Junit 5.2 / Junit platform runner/engine 1.2.

Seems like #744 is not yet released so not a viable option at this point

[jhovell]

Also using maven-failsafe-plugin:2.22.0

[marcphilipp]

Test suites that use @RunWith(JUnitPlatform.class) can only be executed using JUnit 4 directly. The JUnitPlatform runner is merely a way to run tests written using the Jupiter API in tools that only support JUnit 4. Thus, you cannot use it when Surefire uses the JUnit Platform provider. You should be able to force it to use its JUnit 4 provider instead.

#744 will provide a way to declare and run such suites directly using the JUnit Platform Launcher instead of going through the JUnit 4 infrastructure. For now, I'm afraid you'll have to resort to using a different way to organize and run your tests, e.g. a naming convention or a special package.

[sbrannen]

For now, I'm afraid you'll have to resort to using a different way to organize and run your tests, e.g. a naming convention or a special package.

The use of @Tag("xyz") is also an option for achieving something similar to suites in the interim.

Note that you can use quite elaborate "tag expressions" to select your suite.

[jhovell]

Thanks @marcphilipp. The part about forcing surefire/failsafe to use the JUnit 4 provider was what I was missing. Do you think it would help to add this to the Junit5 documentation under the section about suites? That was not obvious at all how to set it up. I assume you take documentation PRs.

[sbrannen]

Do you think it would help to add this to the Junit5 documentation under the section about suites?

I imagine that might be useful to some people as an interim solution.

I assume you take documentation PRs.

Yes, of course we do!

[interkelstar]

Test suites that use @RunWith(JUnitPlatform.class) can only be executed using JUnit 4 directly. The JUnitPlatform runner is merely a way to run tests written using the Jupiter API in tools that only support JUnit 4.

But nevertheless we can run both Junit4 and Junit5 tests via @RunWith(JUnitPlatform.class), despite the version of the runner we use. So i just don't see any difference, why not to run tests in junit4 env using suites and this annotation to run both junit4 and junit5 written tests?

[sbrannen]

But nevertheless we can run both Junit4 and Junit5 tests via @RunWith(JUnitPlatform.class), despite the version of the runner we use. So i just don't see any difference, why not to run tests in junit4 env using suites and this annotation to run both junit4 and junit5 written tests?

You can indeed do that, but only if you make sure that your build tool or IDE is executing the class annotated with @RunWith(JUnitPlatform.class) as a JUnit 4 test. Otherwise, it will not work.

Also, any class annotated with @RunWith(JUnitPlatform.class) cannot include other classes annotated with @RunWith(JUnitPlatform.class) in its suite.

Also, any time you use @RunWith(JUnitPlatform.class), the test tree displayed in your IDE will not work properly for JUnit Jupiter features such as custom display names, nested test classes, parameterized tests, dynamic tests, etc.

In summary, @RunWith(JUnitPlatform.class) has known limitations.

[OmkarModak]

When you create a test case, create it in JUNIT4 not in Jupiter. It works for the latest version 2019-06