Support classes selection by FQN in JUnit 5 test suites

[loicrouchon]

Context

• In order to highlight flaky tests in our codebase, we are using a randomized test order via the maven surefire plugin. This ensure tests depending on some other test/environment state are fixed to remove such a dependency.
• In order to run our tests faster, we are also using multi-fork execution in surefire.

When a test is failing in this mode but not when ran alone, it is very probable that an interaction with the previous test happened. We therefore need a mechanism to run the tests in the exact same order.

Surefire provides a mechanism to set the seed to initialize the randomized test order. But this is not useful in a multi-fork environment as the dispatch of tests among the forks will depend on the speed at which each is processing the tests.

Hence, we need another mechanism.

What we implemented so far is a TestExecutionListener that will generate for each fork a test suite we can then re-use to run the test in the exact same order of the fork.

From there, we proceed with dichotomy to remove the classes from the suite that are not causing the failure of the test identified as flaky.

The problem

In JUnit 4, this worked perfectly. However, with JUnit 5 we faced some issues.

Our Sonar rules are enforcing us to put test classes and methods as package private instead of public.
However, the JUnit 5 @SelectClasses annotation only provides a way to reference the Class objects.

As the classes are now package private as per our sonar compliance requirement, this leads to compilation errors.

As far as I checked, non of the other options for configuring a test suite allow for selecting individual test classes while guaranteeing the order of test executions of those classes.
I checked documentation of package org.junit.platform.suite.api and its content.

In practice, @SelectClasses does guarantee this order of execution, even thought I'm unsure if that is part of its specifications.

The workaround

So far, the workaround was to generate a programmatic test-suite in a main method, but this has the drawback that we loose the IDE integration during the execution.

Possible solution idea

A possible solution would be to enhance @SelectClasses with an additional attribute to provide the list of fully qualified names (FQN) as strings instead of Class objects which are not compile-time visible.
A dedicated annotation could also be created for this purpose.

It is important that the order of the fully qualified names specified in the annotation is reflected in the test execution order as it currently is (but maybe unspecified) for @SelectClasses.