Enhance support for dynamic tests

[sbrannen]

Status Quo

Since 5.0 M1, dynamic tests can be registered as lambda expressions, but there are a few limitations with the current feature set.

Topics

1. Lifecycle callbacks and extensions are not applied around the invocation of a dynamic test.
2. A dynamic test cannot directly benefit from parameter resolution.
   • A dynamic test cannot make use of a TestReporter.

Related Issues

• Introduce support for parameterized tests #14 "Introduce support for parameterized tests"
• Improve documentation of DynamicTest lifecycle #386 "Improve documentation of DynamicTest lifecycle"
• TestReporter does not capture the correct TestIdentifier when used with DynamicTests #393 "TestReporter does not capture the correct TestIdentifier when used with DynamicTests"
• Introduce mechanism for terminating Dynamic Tests early #431 "Introduce mechanism for terminating Dynamic Tests early"
• Let @BeforeEach and @AfterEach work around all the tests from a single TestFactory (or introduce a new similar @Before/@After pair) #694 (duplicates this issue, with example)

Deliverables

Address each topic.

[mkobit]

I think it could also be useful/interesting for extensions in this case to do be able to compute "test matrix" by using multiple different parameter resolver extensions and different declarative parameterized values.

For example:

@Test
@ForAValue({3, 4})
@ForBValue({5, 6})
void runMyTest(AClass aValue, BClass bValue) { ... }

I would like to have the ability to have an extension that can register additional tests by looking at the context of a method or class.

[smoyer64]

@mkobit Take a look at PR #577 and the associated proposal in issue #354 ... the idea is to be able to alter the test plan before it's made immutable.

[nipafx]

Regarding lifecycle integration for dynamic tests... What exactly should that look like? Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

Mmmh, now that I'm writing this I don't actually see a problem with that anymore. But I'm sure I saw one when creating #530... Damn brain! 😫

[sbrannen]

Would the whole extension shebang (test instance processing, test parameter injection, before/after, ...) be run for test factory methods and then again for each individual dynamic test?

That's certainly what I have in mind!

[ledoyen]

Here is a proposition for lifecycle hooks invocations around dynamic tests : #735

[mmichaelis]

We just stumbled across this issue because we need the test-lifecycle for dynamic tests. We also thought about using test templates instead, but we need the test instance variables (here wired via Spring) to actually retrieve the test parameters. So our factory method looks something like this:

@ExtendWith(SpringExtension.class)
class SpringTest {
  @Autowired
  private List<Bean> beans;

  @ExtendWith(MyExtension.class)
  @TestFactory
  Stream<DynamicTest> cmTeasableTests() {
    return beans.stream().map(b -> dynamicTest("display name", () -> doTestFor(b)));
  }
}

So unless we find a solution how to access instance level fields from test-templates the test factory is much more convenient to use - but misses the important test lifecycle.

[sormuras]

Why don't you implement doTestFor(b) in such a way that it calls custom before/after methods?

[mmichaelis]

This is what I do as workaround now. But it does not fit to the extension mechanism. So what I do now is to manually create for example after each and handleTestExecutionException by using try-catch-finally. But my impression is, that this is not the expected behavior. It took my a bunch of time to realize that my tests failed, because the cleanup (in after each) was only done when all dynamic tests are done.

[mibutec]

For me Dynamic Tests are handled far to stepmotherly. I'd love to:

• have Lifecycle support for them (as already discussed in this Thread)
• have some kind of usable Surefire feedback for them (see Surefire Plugin should evaluate @DisplayName #990)
• discover them early (see Discover dynamic tests early during discovery phase #1338)
• add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

[sormuras]

Dynamic tests are just grouped assertions that show up in the test plan. Think of them as assertAll(...) calls on steroids! Nothing more, nothing less. I like to call them "Testlets" on my mind. Dynamic tests are good as they are. They fill a small gap that grouped assertions have: you see all assertions and you can pick a single one an re-run only that ... testlet.

Most of your requested features are already provided by normal or parameterized tests. If those two don't fit your need maybe we can extend them? Perhaps you want and need to roll your own TestTemplate implementation?

[sbrannen]

@mmichaelis,

So unless we find a solution how to access instance level fields from test-templates

What's keeping you from doing that?

Are you just saying you want to access Spring beans from within a custom test template implementation?

If so, that's already possible.

[sbrannen]

add any annotation (@Tag("myTag"), @MyVeryOwnTestAnnotation(myVeryOwnParameter=42)) to them.

That's unfortunately not possible in Java with Java's standard reflection APIs, and any attempt to support such annotation lookups would rely on lambda expression implementation details that may change in future JDK releases.

[sbrannen]

However, if you're curious about some hacky way to achieve that, feel free to take a look at my serialized lambda PoC here: sbrannen/junit-lambda-playground@ef38dde

[sbrannen]

I am sad to say that the "serialized lambda" technique no longer works. See the following JDK issues for details.

• https://bugs.openjdk.java.net/browse/JDK-8138729
• https://bugs.openjdk.java.net/browse/JDK-8140279

It is therefore not possible to perform parameter resolution based on annotations.

Thanks to @nicolaiparlog for bringing this to my attention.

[sbrannen]

we need the test instance variables (here wired via Spring) to actually retrieve the test parameters.

@mmichaelis, this answer I posted on StackOverflow shows how to parameterize tests from Spring beans: https://stackoverflow.com/a/56769619/388980

If you go that route, you have proper test lifecycle method support for each parameterized test invocation.

[ST-DDT]

Would it be possible to just "redefine" DynamicTests / @TestFactory as:

// Virtual part start
@ParameterizedTest
@MethodSource
void testDynamicStuff(DynamicTest test) {
    test.getExecutable().execute();
}
// Virtual part end

// @TestFactory
Stream<DynamicTest> testDynamicStuff() {
      return ...;
}

or at least a new annotation that behaves like this?

[marcphilipp]

Not really since dynamic tests can be nested.

[arun-mano]

Hi Tibor,

I was working with JUnit 5, creating dynamic tests, when I run the build using maven,
The XML file under /surefire-reports has the information as expected.

But in the Console logs still lists with indexes, and the test case Displayname is not printed.

I have the below Plugin config:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${maven-surefire-plugin.version}</version>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>${junit-jupiter.version}</version>
        </dependency>
        <dependency>
            <groupId>org.junit.vintage</groupId>
            <artifactId>junit-vintage-engine</artifactId>
            <version>${junit-vintage.version}</version>
        </dependency>
    </dependencies>
    <configuration>
        <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedTestSuiteClassName>true</usePhrasedTestSuiteClassName>
            <usePhrasedTestCaseClassName>true</usePhrasedTestCaseClassName>
            <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
        </statelessTestsetReporter>
        <consoleOutputReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5ConsoleOutputReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
        </consoleOutputReporter>
        <statelessTestsetInfoReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5StatelessTestsetInfoReporter">
            <usePhrasedFileName>false</usePhrasedFileName>
            <usePhrasedClassNameInRunning>true</usePhrasedClassNameInRunning>
            <usePhrasedClassNameInTestCaseSummary>true</usePhrasedClassNameInTestCaseSummary>
        </statelessTestsetInfoReporter>
    </configuration>
</plugin>

Error trace:
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.203 s - in null
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] com..........TestCase.testGenericClientsFactory
[INFO] Run 1: PASS
[INFO] Run 2: PASS
[INFO] Run 3: PASS
[INFO] Run 4: PASS
[INFO] Run 5: PASS
[INFO] Run 6: PASS
[INFO] Run 7: PASS
[INFO] Run 8: PASS
[INFO] Run 9: PASS
[INFO] Run 10: PASS
[INFO] Run 11: PASS
[INFO] Run 12: PASS
[INFO] Run 13: PASS
[INFO] Run 14: PASS
[INFO] Run 15: PASS
[ERROR] Run 16: TestCase.lambda$null$3:81->testClientExecute:133->lambda$testClientExecute$7:162 » JSON
[INFO] Run 17: PASS
[INFO] Run 18: PASS
[INFO]
[INFO]
[ERROR] Tests run: 33, Failures: 0, Errors: 1, Skipped: 0

Please help me out on this scenario

[marcphilipp]

@arun-mano Please report this issue to the Maven Surefire project.

[t1]

I wrote an extension that supports @TestFactory tests that read tests from files. But I also need setup/cleanup code to run before/after those individual tests, not the whole factory, so I wrote my own @Before/AfterDynamicTest annotations. This is a very limited lifecycle that I think is consistent with the testlet idea described above.

Maybe it would be an idea to leave the current lifecycle as it is and add these annotations instead?

[sbrannen]

Maybe it would be an idea to leave the current lifecycle as it is and add these annotations instead?

That's another option; however, it would not allow individual dynamic tests to benefit from extensions that provide additional behavior via the standard lifecycle callbacks.

[t1]

it would not allow individual dynamic tests to benefit from extensions that provide additional behavior via the standard lifecycle callbacks.

Yes. So dynamic tests are either 'testlets' or have a fully fledged test lifecycles. I'm fine with both options.

[Druckles]

As dynamic tests are functional, would a functional approach be more suitable and more flexible instead of trying to hook into the existing lifecycle & annotations?

E.g.

return DynamicTest.dynamicTest("Test getting users", () -> {
  var result = db.getUsers():
  assertThat(...);
})
  .before(this::setUpDatabase)
  .after(this::tearDownDatabase);

This has the flexibility of being arbitrarily applicable, e.g. to a DynamicContainer:

return DynamicTest.dynamicContainer("Even numbers in database", DynamicTest.stream(...))
  .before(this::setUpForEvenNumbers)
  .after(this::tearDownForEvenNumbers);

This is especially useful if something needs to be performed before or after a group of tests, rather than all tests or each test. I've not found a way to do this other than counting down how many tests have been executed and executing some code when we reach 0.

It can also work to an arbitrary depth of nesting.

[ST-DDT]

I like that approach, but for the container it must be clear, whether the before will be beforeAll or beforeEach (same for after).

[Druckles]

Having thought about it, I would argue that one doesn't need the concept of all/each in this context. It is always before or after whatever container/node you added it to.

This is equivalent to a beforeAll, but the term doesn't make sense semantically on a single node. I.e.

DynamicTest.dynamicTest("Test my function", () -> {})
  .beforeAll(...)

is misleading (as there's only one test).

And to do the equivalent of a @BeforeEach, you can:

DynamicTest.stream(...)
  .map(test -> test.before(...))

[ST-DDT]

@Druckles Just to confirm, is your api call stored in a new field or is it used to mutate the test instance?

• New field => requires usage changes for the test runners.
• Mutates instance => its only syntactic sugar

If it's only syntactic sugar, I don't see a reason, why I should call before manually for all instances/map them myself.

[t1]

This is especially useful if something needs to be performed before or after a group of tests, rather than all tests or each test.

Just a hint: I found @Nested classes a useful solution for grouping test setup/teardown.