@Order does not work if tests are run in parallel

[krzbro]

Hi guys!

I have a test repository with tests marked by @Order annotation. Values are meant to be used as an "importance factor": most important tests should be executed first, least important / longer tests should be executed at the end of the suite. It is supposed to help devs catch any issues at earlier stages instead of waiting for the whole suite to finish.
It works great when I execute tests in a single thread, however it seems there is an issue when I use multi-threading to speed things up. Tests are being picked up rather randomly when I do that.

Let's consider this example:

I'm using class level annotation, it is configured globally.

Ten tests (test_1 - test_10) in classes with @Order(10)
Ten tests (test_11 - test_20) in classes with @Order(20)
Ten tests (test_21 - test_30) in classes with @Order(30)

When I debug these tests and put a breakpoint at HierarchicalTestExecutor constructor, my ExecutionRequest request --> request.getRootTestDescriptor().getChildren() returns set which seems to be sorted correctly.

test_1, test_2, ..., test_10, test_11, ... test_20, test_21, ..., test_30)

When I update any order values in any classes - again, it is being sorted correctly and works in single thread execution.

At the same time, when I run the suite and ForkJoinPoolHierarchicalTestExecutorService starts its job, it seems that at first it works correctly, but then starts picking tests randomly. So it will start executing test with @Order(30) even though it didn't touch any tests from @Order(20), and even some tests in classes annotated by @Order(10) have not been executed yet.

test_1, test_2, test_3, test_4, test_5, test_6, test_7, test_8, test_13, test_21, test_25, test_29, test_30, test_9, ...

So basically, running tests in parallel makes @Order annotation pointless.

I am not sure if this is the expected behavior. I didn't find anything useful about possible workarounds and so on in docs. Thanks for looking into it if possible!

• Used versions (Jupiter/Vintage/Platform): 5.9.2
• Build Tool/IDE: IntellIJ/mvn

[sbrannen]

I am not sure if this is the expected behavior. I didn't find anything useful about possible workarounds and so on in docs. Thanks for looking into it if possible!

This is the expected behavior and is documented in the Parallel Execution section of the User Guide:

When parallel execution is enabled and a default ClassOrderer is registered (see Class Order for details), top-level test classes will initially be sorted accordingly and scheduled in that order. However, they are not guaranteed to be started in exactly that order since the threads they are executed on are not controlled directly by JUnit.

In light of that, I am closing this issue.