Specifying the order of tests in test reports

[luontola]

Motivation

With testing frameworks which consider test names as documentation/specification, the order of test results matters. IDEs and CIs should report all tests in the order they were written (not necessarily same as their execution order), especially if they are used for generating documentation.

For example see Specsy's Fibonacci example which has two tests ("The first two Fibonacci numbers are 0 and 1" and "Each remaining number is the sum of the previous two") whose names make sense only when read in the order they were written. This is also required by Cucumber and similar tools (assuming Cucumber features, scenarios and steps are represented as nested tests).

Current state

AFAIK, currently JUnit 5 does not maintain the order of tests, because methods like TestPlan.getChildren return a Set. I hope that the test reporting order will be made a first-class citizen.

Ideas for a solution

JUnit 4 maintained the test order by returning a List from Description.getChildren, but relying on a mutable class has its problems, and I'm not sure whether maintaining test order was a design goal or just an implementation detail in JUnit 4.

Jumi uses the fi.jumi.api.drivers.TestId class to specify both test order and test identity using a "path", which is a list of integers.

A solution which I think would work for JUnit 5 is to combine its string-based org.junit.gen5.launcher.TestId with information about the order of tests. If each part of the unique ID (see #106) would have both String name and int order, and the unique ID would be a list of those pairs, then they would be easy to sort predictably: order firstly by part.order, secondly by part.name, thirdly by the next part. The order may be the same for all parts at the same level, in which case the name will determine the order (for example packages should be sorted like this). It would need to be discussed whether the order should affect the ID's equality; I have arguments against and for it.

This issue relates to #48 (scenario tests), but not directly to #13 (runtime order). My proposed solution relates to #106 and #68 (unique ID).

[marcphilipp]

Since JDK 7 the order of methods as returned by the JDK's reflection API is not deterministic anymore. Therefore, to guarantee/specify a certain order additional work has to be done. JUnit 4 has solved this using @FixMethodOrder on the class level. I'm afraid there is no way to get the order "the tests were written in" (assuming that means their order in the source file) using reflection. For that we would have to parse the source file which might not even be available at runtime.

Please note that JUnit 5 does in fact allow engines to maintain an order of tests by using a LinkedHashSet, as AbstractTestDescriptor does. Unfortunately, the JDK does not have an OrderedSet interface. Otherwise we would have used that for the TestDescriptor interface.

Do you have a compelling reason why one would want the reporting order to differ from the execution order?

[luontola]

As you said, for JUnit the reporting order has no guarantees due to how reflection works, but I'm concerned about the needs of other test engines. (BTW, it's possible to find out the source order of methods by reading the line numbers from the bytecode.)

The problem with having ordering as a implementation detail of using a LinkedHashSet, as opposed to it being an explicit feature, is that the order may easily be lost when tools transfer the test results between JVMs.

The reporting order will differ from execution order when tests are executed in parallel (such as in Specsy).

[bechte]

In order to get the line number from the byte code we need to rely on the line number table (https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-4.html#jvms-4.7.12). This is provided only if the code was compiled accordingly (e.g. with debug information). I'm not happy with a dependency on such a deep detail. We plan to support scenario testing, allowing to explicitly specifying the order of tests and, therefore, also the order in the tree. I would rather go this path. If order is really required, a scenario test could be used. In general, I believe the order of tests should not be a requirement at all. Anyways, the JVM does not guarantee the order but it does a good job most of the time. I would state the same for JUnit and refer to scenarios if order is an issue.

[luontola]

This issue is not about the JUnit 5 engine. It is about the needs of other testing framework engines, and and whether the reporting order should be a explicit feature or an implementation detail.

[bechte]

As @marcphilipp already mentioned, any TestEngine is free to support a guaranteed order of the tests. Still, we cannot guarantee that in general and, therefore, it seems reasonable to me, that the interface represent a more general case (which also could be a Collection indeed).

The problem with having ordering as a implementation detail of using a LinkedHashSet, as opposed to it being an explicit feature, is that the order may easily be lost when tools transfer the test results between JVMs.

I agree that one could be misleaded by Set and disregard the fact that the actual implementation was ordered. Still, I think we are talking about a detail. Would it be helpful for you if a TestEngine could provide the information whether or not it guarantees the order, e.g. with #90 ?

[luontola]

Would it be helpful for you if a TestEngine could provide the information whether or not it guarantees the order, e.g. with #90 ?

I doubt that an "ordered capability" would help, because IDEs and reporting tools should not need to handle differently TestEngines which have the ordered capability and those which don't have it. As long as the reporting tools maintain the test order from the TestEngine, they will work regardless of whether the TestEngine produces a deterministic test order or not.

A thing that would help is a Technology Compatibility Kit (TCK). A TCK for JUnit could consist of example test classes and the expected results of running those test classes. Then IDE and build tool vendors can run the TCK test suite and check that their tool reports the tests as expected.

As an example, for testing that an IDE maintains the test order, the TCK would contain a test class which should produce the result tree shown below. If an IDE reports the tests in any other order (e.g. in random order or sorted alphabetically), then that IDE would fail the TCK.

• ReportingOrderTest
  • These tests must be reported from A to Z
    • A
    • B
    • C
      ...
    • X
    • Y
    • Z
  • These tests must be reported from Z to A
    • Z
    • Y
    • X
      ...
    • C
    • B
    • A

[luontola]

There's also the question that in which order dynamically registered tests should be shown in test results (relative to the initially registered tests and other dynamically registered tests).

[manueljordan]

Hello, what is the status of this concern/situation? Seems is not easy do this with the current status of JDK 7 and 8.

Just being curious if a new annotation could help this. Something like @OrderDeclaration(number=1), the downside is that the developer must include this annotation for each @Test, and JUnit should throw an exception if the order is repeated twice in the same Test class

I use Spock Framework, I don't if it by itself or through Groovy, but it respects the order of the declaration of each test method in the Test class or Specification.

Thanks

[smoyer64]

I'm working on a proposal for an AfterDiscoveryCallback that would run immediately after discovery filtering. It would contain a single method that looks something like this:

public Set<TestDescriptor> translate(Set<TestDescriptor> testDescriptors);

One of the use cases for this callback is to allow a ScenarioExtension that would include an annotation (that extends @test) similar to:

@Scenario({"myfirstTest", "myOtherTest"})

In this case, the value() function would return an array of String names of the test methods that must be executed before the current test (the test's predecessors). So long as the test dependency graph is acyclic, the are executed in the order needed to simulate a scenario (flow, use case). Most importantly, the ScenarioExtension would take in any type of Set but in this case would return one of the implementations that provides an ordered Set. This callback would also allow a (simpler) @OrderedTest(1), @OrderedTest(2), etc.

Comments? I was working on a PoC for this callback but my day job has been a bit hectic. Note that there are currently no callbacks that execute while the test plan is still mutable so there are a few classes that would have to be moved to the engine API (from the implementation). Test concurrency is also an issue since SureFire likes to run tests in parallel. I envision all steps using @Scenario to be included in a single test class, and that if concurrency is desired, processes are spun up for each test class. So far I haven't thought of a clean way for the engine to know that one class' test methods are safe for concurrency but another class' aren't.

[ctapobep]

I just stumbled upon the fact that the order of methods changes from run to run. Specifically if there are 2 methods where one name contains another:

void generatesStringWithCorrectSymbols_whenLengthUsed(){}
void generatesString_from1_to100_byDefault()

Do you have a compelling reason why one would want the reporting order to differ from the execution order?

For me this appeared to be critical since I'm writing an extension for randomization. If tests fail I'd like users to be able to explicitly specify the seed to get exactly the same data generated again. But because the order differs from run to run this functionality will not work in a lot of cases.

Would be great if JUnit5 was sorting the methods alphabetically or in any other deterministic way.

[marcphilipp]

I think we should for now at least ensure that methods are always run in the same order. JUnit 4 uses this default ordering: https://github.com/junit-team/junit4/blob/master/src/main/java/org/junit/internal/MethodSorter.java#L13-L22.

@junit-team/junit-lambda What do you think?

[sbrannen]

@marcphilipp, I've already reopened #13 and assigned it to M6 to address the ordering issue for tests.

[sbrannen]

As for whether #13 will adequately address the "order of tests in test reports" issue, that remains to be seen.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[stale]

This issue has been automatically closed due to inactivity. If you have a good use case for this feature, please feel free to reopen the issue.