Support JUnit 4 RunListeners

[raulgd]

Expected Behavior

I'd like to be able to include my own JUnit RunListener classes for starting up and shutting down resources that are used across test suites to massively reduce resource setup overhead on test suites, the same way that it's supported in Maven surefire.

Current Behavior

There's no way to setup a run listener using Gradle

Context

This affects me in that, for testing, we used multiple "outside" resources like database, message queue, mail, etc. services, and we embed those for more real world validation testing.
As those services tend to take time to startup and shutdown, we setup test suites and a run listener that initializes such services only once, then executes all tests, and shuts them down, otherwise, the services get startup and shutdown per test, which consumes a lot of time and resources.

Your Environment

This is executed both in developers local machines with MacOS, Windows and Linux, and when executed by the CI server (Jenkins) it's tends to execute a lot of build pipelines in parallel, so this is where we notice the resource usage the most.

[w25r]

I'd like to explore the use cases in more depth.

Here's a list of questions and concerns that came up in internal discussions

• What do these custom listeners provide that current TestListeners do not?
• Why aren't these custom listeners added in the code, making them available in the IDE?
• What impact will running tests in parallel have on these listeners?
  • What impact will the forkEvery flag have?
• Will this strategy be compatible with JUnit 5?

[w25r]

It's also worth noting that a very similar use case was submitted in #701

[baev]

@w25r the problem is that currently there are no way to use custom reporting (for example - Allure Framework) with JUnit 4 and Gradle. And there are no way to get all the information about the test using TestListeners (they are kind of designed to only get list of the tests and their statuses). Allure can catch a much more such as steps, attachments, test parameters, custom groups, test fixtures etc.

[w25r]

@baev I see that you've already contributed to an example using Allure with Gradle. I think that is a valid workaround (if not a better alternative) to customizing test behavior in a build script.

Another possibility is performing these changes in a Gradle plugin that allows custom listeners to be added. That greatly reduces the risk, as only consumers of the plugin would be affected by the changes.

If we still do want to move forward with a change to Gradle proper, the next step we need is a well thought out Design Spec that addresses the concerns outlined above.

[baev]

I see that you've already contributed to an example using Allure with Gradle. I think that is a valid workaround (if not a better alternative) to customizing test behavior in a build script.

This requires to configure Runner for all the tests. Imagine to patch all custom runners (Parameterized, SpringRunner etc) to add the listener to tests. The problem is that we can not migrate test projects with thousands of tests from Maven to Gradle because of that.

The other workaround is use AspectJ to catch execution of internal JUnit methods and add the listener in there. But this smells bad

Another possibility is performing these changes in a Gradle plugin that allows custom listeners to be added. That greatly reduces the risk, as only consumers of the plugin would be affected by the changes.

What plugin do you have in mind? Do you mean to create a brand new plugin?

If we still do want to move forward with a change to Gradle proper, the next step we need is a well thought out Design Spec that addresses the concerns outlined above.

I can do all the work, this issue is kind of blocker for us.

[w25r]

What plugin do you have in mind? Do you mean to create a brand new plugin?

Yes, a brand new plugin would be perfect. apply plugin: 'junit-listeners' or something to that effect. Some investigation would be required to see if there is an opportunity to override existing behavior. Ideally, you could swap out the existing runner with the one you submitted in the pull request, but do it as a plugin.

If there doesn't seem to be a way to get what needs to be done in plugin form, then a design spec would be great.

[baev]

If there doesn't seem to be a way to get what needs to be done in plugin form

I already tried this but no luck.

[JonathonShields]

@baev Where you able to solve this issue? Where you looking to simply add an extra listener or override the OFTB Gradle one? I've come across a similar issue and I'd be interested to know how you have got on.

[baev]

I need to add one more JUnit listener. The only way I know is using the following aspect:

@Aspect
public class AllureRunListenerAspect {

    private static AllureRunListener allureRunListener = new AllureRunListener();

    @Pointcut("execution(void org.junit.runners.ParentRunner.run(org.junit.runner.notification.RunNotifier))")
    public void run() {
    }

    @Around("run()")
    public void run(ProceedingJoinPoint pjp) {
        Object[] args = pjp.getArgs();
        RunNotifier notifier = (RunNotifier) args[0];
        notifier.removeListener(allureRunListener);
        notifier.addListener(allureRunListener);
        notifier.fireTestRunStarted(Description.createSuiteDescription("Tests"));

        try {
            pjp.proceed(args);
        } catch (Throwable throwable) {
            throwable.printStackTrace();
        }

        notifier.fireTestRunFinished(new Result());
    }
}

I tried to create a brand new Gradle Plugin that adds listener, but can't find the way to do it.

[baev]

Is there any chances to get this processed?

[DreierF]

+1

[baev]

@kcooney @marcphilipp could you please help us with the issue? Currently there is no way to add a listener to JUnit4 in Gradle projects. Since we decided to not merge SPI support for listeners junit-team/junit4#1122 to JUnit 4 that is blocker for us

[w25r]

@baev as stated before, we'd like to see a design spec that addresses the aforementioned issues before this can be addressed.

How close did you get on the plugin side? If you push your repo, we could take a look at it and may be able to resolve your issues (or confirm that it can't be done from a plugin)

[marcphilipp]

The new JUnit Platform supports adding listeners via ServiceLoader. It can run old JUnit 4 ("Vintage") and new Jupiter tests. Is that a viable alternative for you?

[baev]

The new JUnit Platform supports adding listeners via ServiceLoader. It can run old JUnit 4 ("Vintage") and new Jupiter tests. Is that a viable alternative for you?

Nope, some our users can't migrate to JUnit 5 (banks, legacy projects etc)

How close did you get on the plugin side? If you push your repo, we could take a look at it and may be able to resolve your issues (or confirm that it can't be done from a plugin)

I didn't find a way how to override logic in JUnitTestClassExecuter, so there is no code at all

as stated before, we'd like to see a design spec that addresses the aforementioned issues before this can be addressed.

I'm not familiar with your dev process. Can you link some articles about design specs etc?

BTW I can do all the work but I need to be sure that this change gonna be merged some day. Also I don't know about your feature plans about JUnit 5 support in Gradle, how this will work together with JUnit4 etc. I pretty much sure that there are so many people that will use JUnit 4 for next few years and will not migrate to JUnit Platform, so It would be useful to have an ability to configure RunListeners anyway.

[kcooney]

Sorry for the late response. A few thoughts:

First of all, the JUnit RunListener API was intended to be used to notify code about the state of running tests (to drive a UI, produce an XML report, etc). It wasn't intended to be used to change the behavior of the running tests. By using it for this new purpose, you might get some confusing behavior. For example, if you do work in testFinished() that could throw an exception, the listeners could be notified about the failure after they were notified that the test was finished, and some listener notifications might not expect this. I realize that you have to work with the APIs you have available, but I thought I should mention this.

Secondly, I'm no longer comfortable with the idea of using an SPI interface to register listeners in JUnit. Doing so would allow any code in your classpath to add listeners, even if those listeners cause problems or are even hostile to your development environment. I could be convinced otherwise, but currently I'm thinking that if JUnit4 supported registering listeners via ServiceLoader there would need to be some way for a test in package com.example.project to opt-in for a listener that lives in org.evil.

The Gradle config could be one way to opt-in, but then you would only get the behavior if you run the tests in Gradle.

I'm confused about the concern about JUnit5. The users would not need to change their tests to use JUnit5. JUnit5 provides a newer API that IDEs and build tools can use to run JUnit4-style ("Vintage") and JUnit5-style ("Jupiter") tests (I believe JUnit3-style tests would run using the JUnit4 code, so those would be supported, too). I think Marc was suggesting that Gradle would eventually migrate to using the JUnit5 APIs to run JUnit tests, and you could take advantage of those APIs (which would also not require having your users upgrade their JUnit jars, since JUnit5 uses different maven packages).

[baev]

the JUnit RunListener API was intended to be used to notify code about the state of running tests (to drive a UI, produce an XML report, etc). It wasn't intended to be used to change the behavior of the running tests.

I agree, so all we need is produce extended XML report (steps, parameters, attachments, history, retries etc).

I'm no longer comfortable with the idea of using an SPI interface to register listeners

I got your point. The main idea of this conversation is that currently there no way to add listeners to JUnit using Gradle. And since Maven is kind of dead, our users start to migrate from Maven to Gradle. And they can't use Allure any more. That is the problem

I am totally fine with using JUnit 5 platform inside Gradle to run the tests as long as users will not require to change the code of tests. The only thing is that it would be so nice to get the problem resolved as soon as possible.

[kcooney]

the JUnit RunListener API was intended to be used to notify code about the state of running tests (to drive a UI, produce an XML report, etc). It wasn't intended to be used to change the behavior of the running tests.

I agree, so all we need is produce extended XML report (steps, parameters, attachments, history, retries etc).

Sorry for the confusion, @baev. @raulgd (who opened this issue) indicated that he wanted to shutdown resources in the listener. If all Allure needs is a test run report, then RunListener would work, and doing it in Gradle seems like the right path. It looks like TestListener might work as well. It might help to let the Gradle team know what you need to do that cannot be done with TestListener

[baev]

As I mentioned before TestListener designed to only get list of the tests and their statuses. Allure requires much more (an example of our RunListener you can find here).

[kcooney]

@baev I'm sure I'm missing something, but I still don't quite see what you need that TestListener doesn't provide. You are overriding testStarted() (which apparently maps to beforeTest()), testFinished() (== afterTest()) and testFailure() (== afterTest()). The fields you access from Description seem to have equivalents in Gradle's TestDescriptor (assuming you can map from the class name to the class). Is the problem that Gradle's TestListener is running in a different process than the test, and you need something in the same JVM?

[baev]

Is the problem that Gradle's TestListener is running in a different process than the test, and you need something in the same JVM?

Yep, we are using ThreadLocal to catch information about steps and attachments and reflection to get some metadata from test class/method (an example from annotations).

[w25r]

I think it's an intriguing idea to try to leverage JUnit 5's ServiceLoader mentioned by @marcphilipp. It looks like you can already use JUnit 5 with Gradle, so it shouldn't be a problem there. This is probably the best bet for getting this problem solved in the short term.

If that doesn't work, design specs can be found here. We already discussed some of the use cases and concerns that should be addressed in the design. As for promising that it's going to be merged someday, well, that is impossible to guarantee. But if the Gradle team determines that it is a safe, well-thought, well-tested change, then we are likely to merge it.

[baev]

shall I create an separate PR with design docs or continue the work in #1725?

[w25r]

A new one would probably be better, and we can reference this issue and the old PR.

[andreiadamian]

I believe the primary issue @baev refers to is the absence of TestDescriptor.getAnnotations().
It is present in JUnit's Description class, but gradle ignores this important piece of information when constructing DefaultTestDescriptor.

It is quite common to use test annotations for reporting purposes (allure is a great example).

Is there any chance we can get them into the TestDescriptor?

[pelizza]

Hey guys, any updates on this issue?

[sbabcoc]

I posted a potential solution in the Help/Discuss forum that could help JUnit 4 users who need to attach RunListener providers to their test suites.
https://discuss.gradle.org/t/how-to-attach-a-runlistener-to-your-junit-4-tests-in-gradle/30788
The post describes how to configure a Gradle project to use the JUnit Foundation library to attach RunListener classes to JUnit 4 test suites. Listeners are specified in a ServiceLoader provider configuration file, and the library automatically attaches them.

[pelizza]

Thanks @sbabcoc, I'll try that out!

[sbabcoc]

@pelizza Let me know if you run into any issues. I've fixed all of the problem I encountered myself, but my scenarios are very limited. Also, I've tried to provide clear, complete documentation, but it's likely that more details would be helpful here and there. Feedback is welcomed!

[sbabcoc]

@oehme I think the information I posted will be helpful to many folks still using JUnit 4. The library I refer to enables timeout management, automatic retry of failed tests, and event notifications in Gradle project tests. Where is the best place to publicize this information?

[oehme]

The forum or a blog post, so people will find it when they google for it.

[cah-daniel-adams01]

@sbabcoc I attempted to utilize this solution in my project and it is not working. I am not seeing any mention of the RunListener being picked up as the execution goes through it's processing. Any assistance would be welcome.

[sbabcoc]

@cah-daniel-adams01 The current release is 9.4.2. Try this first. Since this is a Gradle forum, we should probably continue any further discussion on the JUnit Foundation project repo. Open an issue with your scenario details and we'll see if we can get you going.

[mleegwt]

I can report a working RunListener using JUnitFoundation as reported by @sbabcoc. E.g. following https://stackoverflow.com/questions/54450475/junit-listener-configuration-in-gradle
This works fine for testRunStarted calls.

However: To also get the calls to testRunFinished I needed to switch to 'useJUnitPlatform()' (using the vintage engine).

[sbabcoc]

Yes, JUnit Foundation is implemented specifically to support JUnit 4. The testRunStarted notification is issued by JUnit Foundation when each RunListener is attached, but the testRunFinished notification comes from the JUnit 4 API.

[yavgel]

any update on this request?

[marcphilipp]

If you configure your test task to useJUnitPlatform(), add the Vintage engine to your test runtime classpath, and use JUnit 5.8.x or later, you can register a LauncherSessionListener via Java's Service Loader mechanism.

Docs: https://junit.org/junit5/docs/current/user-guide/#launcher-api-launcher-session-listeners-custom