Add support for test lifecycle events in TestExecutionListener

[ehborisov]

Now there's no ability to track invocations of Before/After methods with TestExecutionListener, we need it there to be able to link test invocations with all corresponding fixtures and their timings in the reporting tool (Allure). Extensions are not entirely a solution for that, since we want to track events around fixture, one callback on the finish is not enough. And also registering such an extension via annotations everywhere would be quite clumsy.

Related Issues

• Introduce TestWatcher extension API for processing test execution results #542

[sbrannen]

Changed title to reflect terminology used in JUnit Jupiter.

[sbrannen]

Is the proposal here intended to replace #542?

[baev]

I think that is much better to have engine-independent listener for base test events such as TestGroupStarted/Finished TestFixtureStarted/Finished and TestCaseStarted/Finished

[sbrannen]

Please keep in mind that the JUnit Platform and JUnit Jupiter do not have groups, fixtures, or cases.

Thus, we will need to use terminology in line with containers, tests, and lifecycle events.

[ehborisov]

What @baev wrote about TestGroupStarted/Finished TestCaseStarted/Finished is basically a question of splitting general executionStarted/executionFinished events in TestExecutionListener into separate events for containers and for tests, at least as I got it, which is a slightly different discussion. Though it would simplify things a bit, I agree.

What is needed at the least is engine independent lifecycle events support (specifically, something to determine both start and finish of BeforeAll, BeforeEach, AfterAll, AfterEach for tests run with Jupiter engine as well as for Before, BeforeClass, After, AfterClass for tests run with Vintage engine).
This has little in common with what is proposed in #542

[ehborisov]

Any updates? @sbrannen

[marcphilipp]

@ehborisov @baev Since M4 you can register extensions via ServiceLoader (see http://junit.org/junit5/docs/current/user-guide/#extensions-registration-automatic). Can you please try and check if the lifecycle callbacks (see http://junit.org/junit5/docs/current/user-guide/#extensions-lifecycle-callbacks) are sufficient for your use case?

[baev]

Nope. We need to have around callbacks for all user code that executed during test execution, including configuration methods (@BeforeAll, @BeforeEach etc) and even test data generation methods for parameterized tests

[marcphilipp]

Have you looked at http://junit.org/junit5/docs/current/user-guide/#extensions-execution-order? There's a lifecycle callback around all test class methods except test data generation methods.

In case that's really not sufficient, we could add a generic method to TestExecutionListener, e.g.

void executionEvent(Event event)

However, I'm not sure what events we need and which information needs to be contained in such events. Can you propose a list of events/event types, possible as a sequence diagram that includes the existing TestExecutionListener methods?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. Thank you for your contribution.

[stale]

This issue has been automatically closed due to inactivity. If you have a good use case for this feature, please feel free to reopen the issue.