Support JUnit platform reporting entries

[jlink]

When a failing test also publishes report entries, those should be displayed.

Expected Behavior

The junit platform TestExecutionListener has a method reportingEntryPublished which is called when tests report additional data. Those entries should be displayed/logged in gradle's output when a test fails. They should also show up in the generated html report.

Current Behavior

Nothing is shown.

Context

IntelliJ, Eclipse, Maven and the junit-platform-gradle-plugin actually report the entries.

Those entries usually give important additional information to interpret test results.
For example, jqwik (http://jqwik.net) uses those entries to report the random seed
which is needed to reproduce errors.

Steps to Reproduce (for bugs)

Running the following test through gradle should not only report a failure but also something
like timestamp = 2018-03-06T15:40:33.704, key = value (IntelliJ's output)

@Test
void failingTestWithReporting(TestReporter reporter) {
	reporter.publishEntry("key", "value");
	org.junit.jupiter.api.Assertions.fail("Fails with an additional report entry");
}

Your Environment

OSX, Java 8, Gradle 4.6

[oehme]

As mentioned on the #4716:

We should overhaul our test logging infrastructure holistically. We have already added display names, but only to the html report and console log. Now we are considering JUnit 5's reporting entries.

We should decide what things we want to support in a cross-cutting way (so that other test frameworks can be adapted to it). A display name makes a lot of sense. A Map<String, String> of "additional data" also seems like something other test frameworks might be interested in.

This information should be exposed to

• Build scans
• TestListeners
• Tooling API (so IDEs can display it)
• Console
• HTML report

@blindpirate before we continue here we should have a small design spec that addresses the above and proposes an API. We should make sure this API also makes sense for the native tests frameworks for instance.

We should not just add reporting entries as test output, because otherwise people might start depending on this implementation detail and try to parse that output. Then we'd be stuck if we wanted to change it to something more strongly modeled. Let's do some design first.

[sbrannen]

Please keep in mind that reporting features like display names and reporting entries (via a map of key-value pairs) will end up in the new reporting format from the Open Test Alliance. So that should make them mainstream... once that new format is supported.

[oehme]

@sbrannen is there already a spec for the format?

[sbrannen]

No, unfortunately not (yet).

That will be discussed here: ota4j-team/opentest4j#9

So feel free to chime in there with comments.

[jlink]

Any updates on this one?

With JUnit platform 1.3.0 looming, the old gradle plugin will no longer be supported and there's no way to run up-to-date junit 5 engines with reporting support.

[jlink]

Any updates on this one? JUnit platform 1.3.1 is already out.

[marcphilipp]

I'm afraid not. We're busy working on Gradle 5.0 at the moment. I hope to tackle this and other JUnit Platform issues (e.g. #6453) in the 5.1 timeframe.

[jlink]

What a bummer. Especially since I cannot think of any workaround.

[marcphilipp]

You could register a TestExecutionListener that prints them to stdout where Gradle should pick them up and add them to the test report.

[jlink]

Is there anyway to get stdout added to Gradle console output in case of a test failure? That's what I'd actually need since the test report is a few clicks away.

[sbrannen]

@jlink, I don't know if you can do that just on error, but have you tried testLogging.showStandardStreams = true in the test task?

[jlink]

@sbrannen @marcphilipp Both things together should make a simple mechanism possible. I'll see what I can figure out.

[rkraneis]

This is actually what I resorted to as Maven is also lacking support for report entries and we really like JUnit 5 and jqwik :)

[jlink]

There's a simpler way for jqwik: https://jqwik.net/docs/current/user-guide.html#jqwik-configuration

If you set useJunitPlatformReporter = false jqwik will do its own reporting to stdout.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. Given the limited bandwidth of the team, it will be automatically closed if no further activity occurs. If you're interested in how we try to keep the backlog in a healthy state, please read our blog post on how we refine our backlog. If you feel this is something you could contribute, please have a look at our Contributor Guide. Thank you for your contribution.

[jbduncan]

This issue sounds valuable to me.

[david-kariuki]

We need a solution for this.