Introduce Extension API for registering dynamic tests

[vmassol]

Overview

I currently have a JUnit 4 based custom Suite which automatically registers new tests to the JUnit runner and which allows to write very simple tests:

@RunWith(RenderingTestSuite.class)
public class IntegrationTests {
}

The custom suite reads *.test files found in the CL's classpath and generates dynamic tests from them.

I'd like to move this to JUnit 5, but I'm a bit stuck ATM since it seems the best approach would be to use some test factory or parameterized test, but this would require each of my tests to declare a test method with something like:

class IntegrationTests {

   @TestFactory
   Stream<DynamicTest> renderingTests() {
           return RenderingTestSuite.generateStreamOfDynamicTests();
   }
}

I'd really like my tests to remain as simple as the following (and this without duplication):

@ExtendWith(RenderingExtension.class)
public class IntegrationTests {
}

PS: I've also asked the question on Gitter (which contains additional details).

Related Issues

• Introduce extension API for dynamic test generation #371

[sbrannen]

Tentatively assigned to the 5.3.x backlog for team discussion

[vmassol]

Thanks @sbrannen

[sbrannen]

Closely related to #371.

[kriegfrj]

Any movement on this one? I have a use case that I think I can almost implement using standard extensions points, but not quite.

You guys are no doubt familiar with soft assertions in AssertJ - it collates the test failures into a single MultipleFailuresError and throws that instead so you have perform several assertions per test and not have the first failure prevent the rest of the assertions running.

A problem we are having is dealing with the verbosity of the resulting error message, in particular if we add the stack traces to it for easy IDE navigation.

I had a crazy idea - instead of collating them into a single error and reporting them all against a single test node in the test hierarchy, each of the assertions would generate a DynamicTest node and store the success/failure against it. This is also inspired in part by Specsy which has a similar philosophy of generating multiple nodes, but it allows you to leverage the richness of AssertJ's fluent assertions to do so.

There are two main ways I could implement it. One of them is to come up with a soft proxy generator that produces a stream or collection of DynamicTest nodes as a return value. This could then be used inside of a @TestFactory method. The disadvantage of this is that it would require major refactoring of existing tests using soft proxies if the developer wishes to leverage it.

The alternative is to have the smarts in an extension's afterEach() callback to transform the collected assertions into DynamicTest nodes. This would make for an extremely simply migration path - the developer simply needs to change their @ExtendWith declaration to use the new extension, and it would work without needing to change any of the individual tests. But this is not possible with the current API.

My proposal is quite simple: add a registerTest(DynamicNode) method to ExtensionContext. This method will allow any extension to register a new test node as a child of the currently-executing node. It would solve my use case as I could call registerTest() from within my extensions afterEach() callback.

Thoughts?

[kriegfrj]

Overview

I currently have a JUnit 4 based custom Suite which automatically registers new tests to the JUnit runner and which allows to write very simple tests:

@RunWith(RenderingTestSuite.class)
public class IntegrationTests {
}

The custom suite reads *.test files found in the CL's classpath and generates dynamic tests from them.

I'd like to move this to JUnit 5, but I'm a bit stuck ATM since it seems the best approach would be to use some test factory or parameterized test, but this would require each of my tests to declare a test method with something like:

class IntegrationTests {

   @TestFactory
   Stream<DynamicTest> renderingTests() {
           return RenderingTestSuite.generateStreamOfDynamicTests();
   }
}

I'd really like my tests to remain as simple as the following (and this without duplication):

@ExtendWith(RenderingExtension.class)
public class IntegrationTests {
}

PS: I've also asked the question on Gitter (which contains additional details).

@vmassol, it also occurred to me that your problem could be solved with the existing API, as follows:

public interface RenderingTests {
    @TestFactory
    default Stream<DynamicTest> renderingTests() {
            return RenderingTestSuite.generateStreamOfDynamicTests();
    }
}

class IntegrationTests implements RenderingTests {
    ...
}

Hope that helps (although I see it was a long time ago... you've probably either solved it or moved on!)

I still would like to push for the API to allow extensions to register dynamic tests, as I don't think my use case has a workaround using the existing API.

[vmassol]

Hope that helps (although I see it was a long time ago... you've probably either solved it or moved on!)

@kriegfrj Yes that could be a workaround. I still consider that a workaround and I'd prefer to have a way to register dynamic tests in a JUnit Extension. That would be cleaner and nicer.

Thanks for the idea!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[kriegfrj]

I'm still interested in this one. Haven't given up on the long-term AssertJ plan.

Pinging @joel-costigliola, @scordio from the AssertJ team. I will raise an issue there too and link it to this one.

[vlsi]

I wonder if generating tests during test execution will be compatible with "restart test from IDE", "run specific test from command line", "filter tests by tag".

[stale]

This issue has been automatically closed due to inactivity. If you have a good use case for this feature, please feel free to reopen the issue.

[jbduncan]

JUnit 5 team, it looks like the stale bot forgot to remove the "stale" label for this issue and subsequently closed it too early. Would you mind re-opening it?