Provide mechanism to order the sequence of tests

[elygre]

Overview

Although it is typically discouraged, having a way to order tests is useful to some people. The JUnit 4 @FixMethodOrder annotation specifically and deliberately makes it impossible to implement a simple ordering scheme, forcing users who want/need this to resort to method names like test245_ensureReasonableTestName().

Proposal

• The minimal "requirement" would be to allow users to write their own MethodSorter for @FixMethodOrder
• A reasonable implementation would be to allow @FixMethodOrder(ORDERED), and then @Order(integer) to specify individual tests. Tests could be run from smallest number to highest, with non-annotated methods having a default of 0. Methods with the same order could be run in parallel; different order tiers could not.

Related Issues

• Introduce first-class support for scenario tests #48
• Introduce mechanism for inter-test dependencies #607
• Running tests in certain order #884
• Introduce declarative mechanism for ordering extensions registered via @RegisterExtension #1707

Deliverables

• Introduce a mechanism for ordering testable methods within JUnit Jupiter.
• Validate collection of methods ordered by an extension; log warning or error if methods were added or removed.
• Introduce built-in support for alphanumeric ordering.
• Introduce built-in support for @Order based ordering.
• Introduce built-in support for random ordering.
• Introduce Configuration Parameter for setting the seed used in random ordering.
• Test ordering support in top-level classes.
• Test ordering of @Test, @TestFactory and @TestTemplate methods.
• Test ordering support in @Nested classes.
• Test ordering support in conjunction with parallel test execution.
• Extract junit.jupiter.execution.order.random.seed constant and document it.
• Write Javadoc for all new types.
• Update @API declarations for types with changed visibility, etc.
• Document new features in User Guide.
• Document new features in Release Notes.

Out of Scope

• ❌ Consider introducing a mechanism for ordering tests with the JUnit Platform -- potentially at a later date in conjunction with a separate dedicated issue.

[sbrannen]

Hi @elygre,

Test execution order is something we are certainly considering for the core feature set of JUnit 5.

In fact, introducing an annotation such as @Order is something we've already discussed.

We'll chime in here once we begin work on ordering.

Cheers,

Sam

[kcooney]

@elygre ine thing we considered was something like junit-team/junit4#1130

[ttddyy]

In addition to test method, I believe ordering should be applicable to @BeforeAll/@AfterAll/@BeforeEach/@AfterEach as well.

Also, it would be nice if behavior are clearly defined(or documented) when multiple @BeforeAll/@AfterAll/@BeforeEach/@AfterEach are presented. (i.e. within same class, via inheritance, or via interface).
Currently, it is HierarchyDown and HierarchyUp, but not defined for cases multiple annotations are presented within the same class.(I think it is JDK's returuning order for now.)

Thanks,

[dweiss]

I like the changes and the new API proposal, although (sigh) I'm not entirely convinced resignation from @RunWith is a good idea. My point is that with JUnit4 you had a set of contracts which you could replicate in your custom runner. With this new API tweaking certain things seems impossible if there's no support for them in the "default" engine... at least that's my impression from looking at the code/ specs so far.

Take method ordering. The way I implemented it for randomized testing is by adding an annotation that provides a class which receives a list of tests to be executed:

https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/annotations/TestCaseOrdering.java#L32

Would this kind of flexibility be possible with JUnit5?

[jlink]

Well, you can always build and register your own test engine. If it
subclasses from the junit5 engine, anything should be possible. But that's
a mere guess. We haven't really thought about one test engine replacing
another...

2015-11-23 22:39 GMT+01:00 Dawid Weiss notifications@github.com:

I like the changes and the new API proposal, although (sigh) I'm not
entirely convinced resignation from {{RunWith}} is a good idea. My point is
that with JUnit4 you had a set of contracts which you could replicate in
your custom runner. With this new API tweaking certain things seems
impossible if there's no support for them in the "default" engine... at
least that's my impression from looking at the code/ specs so far.

Take method ordering. The way I implemented it for randomized testing is
by adding an annotation that provides a class which receives a list of
tests to be executed:

https://github.com/randomizedtesting/randomizedtesting/blob/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/annotations/TestCaseOrdering.java#L32

Would this kind of flexibility be possible with JUnit5?

—
Reply to this email directly or view it on GitHub
#13 (comment)
.

[dweiss]

That's my point. What I liked about JUnit4 was the fact that contracts (ok, not clearly formalized, but still there) were somewhat separate from the "execution". I hoped for JUnit5 to follow this paradigm. Specify annotations, interfaces and the execution model, leaving actual implementation out (but providing a default reference).

I am not saying the route taken is wrong. I just express my expectations and desires. :)

[jlink]

I still don't see how extending an existing test engine is qualitatively
different from extending junit4's default runner - and this is what many of
the more powerful runners actually do. Maybe you can clarify?

JUnit5TestEngine is not designed with "subclassability" in mind but that
could be changed...

2015-11-23 23:17 GMT+01:00 Dawid Weiss notifications@github.com:

That's my point. What I liked about JUnit4 was the fact that contracts
(ok, not clearly formalized, but still there) were somewhat separate from
the "execution". I hoped for JUnit5 to follow this paradigm. Specify
annotations, interfaces and the execution model, leaving actual
implementation out (but providing a default reference).

I am not saying the route taken is wrong. I just express my expectations
and desires. :)

—
Reply to this email directly or view it on GitHub
#13 (comment)
.

[dweiss]

Let me explain the underlying need a bit. I am a committer to Lucene and Solr. We use a lot of pseudorandomness in tests. Interchangeable components can get swapped, parameters can get picked at random, strings get picked from a random space, etc. Simplifying a lot, every test is still reproducible because the source of randomness is controlled by the framework. There are also tons of other infrastructure elements: parameter factory methods (parameterized tests), more realistic test and suite timeouts that detect and warn about stray threads left behind, different seed test reiterations, test listeners (they can attach to the runner, so they receive suite started/ finished events), custom test method providers... Lots of it. If you're interested in more details and the philosophy behind it there are some lectures I presented in the past about the topic [1, 2].

The reasons we chose JUnit as the underlying framework were basically twofold:

• virtually everybody knows (more or less) how to use it and is more or less familiar with the default contracts (the test class is loaded once, test methods are invoked on a new instance of the class, before/after hooks wrap around the test or class, etc.),
• there is strong tooling support in every IDE, CI system, reporting tools, you name it.

But the two above don't really mean the test runner is the same as in regular JUnit. It is similar, but has multiple differences that go well beyond the default implementation. Most of these are so low-level that they couldn't be implemented by extending base JUnit4 runner because it encapsulates or hides certain test preprocessing points (I've tried). Or it combines functionality scattered across different runners (for example parameterized tests). Finally, having our own runner makes the implementation less tightly bound to JUnit's version -- the details of the base runner's implementation may change, but all we use from JUnit is basically annotations and the (unfortunate) Description object.

Perhaps an example would highlight it better. Take this class as a simple example:

@RunWith(RandomizedRunner.class)
public class TestFoo {
  final int a;

  public TestFoo(int a) {
    this.a = a;
  }

  @Test
  @Repeat(iterations = 3, useConstantSeed = false)
  public void checkMe() {
    Assert.assertTrue(Math.abs(a + randomIntBetween(0, 3)) >= 0);
  }

  @ParametersFactory(argumentFormatting = "a = 0x%08x")
  public static Iterable<Object[]> parameters() {
    return Arrays.asList(
        $(1),
        $(0x7fffffff - 2));
  }
}

The output of running this is Eclipse is 6 tests, with a pseudo-nested suite above.

Note the tests need to have unique Descriptions (otherwise Eclipse gets terribly confused) and this description is generated at the runner's level (the runner also expands the hierarchy so that test repetitions for different seeds and arguments are unique and placed under the method's node).

Finally and again -- I'm not saying anything in JUnit is badly implemented. We just need very specific stuff that goes beyond the defaults (and would occasionally require stuff that is hidden from subclasses for the right reasons). So if there is a way to extract a "meta test API" level and abstract it from the actual runner implementation it would be beneficial.

[1] https://berlinbuzzwords.de/sites/berlinbuzzwords.de/files/media/documents/dawidweiss-randomizedtesting-pub.pdf
[2] https://berlinbuzzwords.de/session/randomize-your-tests-and-it-will-blow-your-socks

[marcphilipp]

Closed in favor of #48.

[devrandom]

There is a specific need to have a pseudo-random test order, since that allows finding unintended test interactions, and allows repeating the test in that order using the logged seed.

#48 does not cover that use case.

[marcphilipp]

@devrandom Please open a new issue for the pseudo-random order use case.

[devrandom]

@marcphilipp - I see actually that junit4 has a deterministic test order (based on String.hashCode) by default. I couldn't quickly find the default ordering in junit5. Can you tell me where it is defined?

[sbrannen]

I couldn't quickly find the default ordering in junit5. Can you tell me where it is defined?

You couldn't find it... because it's simply not defined.

The ordering of execution of tests within a test class in JUnit Jupiter is based solely on the order in which the methods are returned from the JVM via reflection, which since Java 7 is undefined. In other words, there is no guaranteed order.

[sbrannen]

Reopening in order to address concerns raised in #884 and #48.

[sbrannen]

Introduced Deliverables section.

[sbrannen]

Drum roll........

Support for ordering the sequence of test methods has been merged into master in commit 5300e65. 😎

[aalmiray]

Yay! Thanks again

[sbrannen]

¡de nada! / You're welcome! 😉

[anatoliy-balakirev]

Hi. @sbrannen thanks for adding this to Junit!
We are currently using Spring Cloud Contract in conjunction with Junit 5 and would like to start using Wiremock's scenarios there. However, it looks like SCC team is waiting for this functionality to move out of EXPERIMENTAL status, so I wonder if you have an idea on when it might happen? Is there some way to track this? What is the process in general?

[sbrannen]

Hi. @sbrannen thanks for adding this to Junit!

You're very welcome.

We are currently using Spring Cloud Contract in conjunction with Junit 5 and would like to start using Wiremock's scenarios there. However, it looks like SCC team is waiting for this functionality to move out of EXPERIMENTAL status,

Do you have a link to an issue for that?

so I wonder if you have an idea on when it might happen? Is there some way to track this? What is the process in general?

In general, we flag new APIs as experimental for their initial release and then wait a few minor releases to see if the community raises any issues with the API or implementation.

This particular feature was released in JUnit Jupiter 5.4, and 5.5 is currently being worked on. There have been a few improvements to the implementation since 5.4.0 but no changes to the API. So if that continues to be the case, we might switch the API status from experimental to stable in 5.6 (even potentially for 5.5 GA -- to be discussed).

[anatoliy-balakirev]

Ok, thanks for info!

Do you have a link to an issue for that?

Here is the link: spring-cloud/spring-cloud-contract#887

[sbrannen]

Thanks for the link.

[linusjf]

How does this work with @Execution(CONCURRENT)? It doesn't.

[sbrannen]

@Fernal73, as stated in the Javadoc for MethodOrderer, the default execution mode for ordered methods is SAME_THREAD, but it...

Can be overridden via an explicit @Execution declaration on the test class or in concrete implementations of the MethodOrderer API.

If you are experiencing behavior contrary to that, please open a new ticket providing further details.

Thanks

[linusjf]

I've experienced out of order invocations when setting the Execution mode as concurrent.
Is this expected behaviour given that the feature is experimental and not fully mature?

How do you decide that the ordering does not interfere with concurrency? I'd expect ordered methods to be executed in a single thread wrapped in a synchronizer that orders the methods as per their values so that shared state in the test class is not adversely impacted causing failed tests.

If all my methods are ordered, shouldn't the methods be executed in a single thread irrespective of ExecutionMode?

I have since updated my tests to not have a shared state; @order no longer matters and has been removed.

[sbrannen]

I've experienced out of order invocations when setting the Execution mode as concurrent.

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Defaults to SAME_THREAD, since ordered methods are typically sorted in a fashion that would conflict with concurrent execution.

---

How do you decide that the ordering does not interfere with concurrency?

The MethodOrderer implementations provided by JUnit Jupiter do not, since none of them overrides getDefaultExecutionMode().

I'd expect ordered methods to be wrapped in a synchronizer should the execution mode be set to concurrent.

This issue was closed more than 1.5 years ago.

If you wish to make suggestions for improvements or changes to the current behavior or feature set, please open a new issue.

[linusjf]

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Is this to be expected even for @order annotated methods? Other test methods, of course.

This ought to be documented better. If @order methods go out of sync with ExecutionMode.CONCURRENT, then they ought to be permitted only with SINGLE_THREAD mode.

See previous response for edits.

[linusjf]

This issue was closed more than 1.5 years ago.

If you wish to make suggestions for improvements or changes to the current behavior or feature set, please open a new issue.

I'm inclined to go with the workaround that @order annotated methods not be mixed with methods to be executed concurrently viz, have a separate Test class for each type.

[sbrannen]

That is to be expected. The Javadoc for MethodOrderer#getDefaultExecutionMode() states the following which hints at that.

Is this to be expected even for @order annotated methods?

Yes

This ought to be documented better. If @order methods go out of sync with ExecutionMode.CONCURRENT, then they ought to be permitted only with SINGLE_THREAD mode.

Again, this closed issue is not the place to make recommendations.

If you feel strongly about those points, please open one or more new issues to address them.