New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support JUnit platform reporting entries #4605

Open
jlink opened this Issue Mar 6, 2018 · 12 comments

Comments

Projects
None yet
6 participants
@jlink

jlink commented Mar 6, 2018

When a failing test also publishes report entries, those should be displayed.

Expected Behavior

The junit platform TestExecutionListener has a method reportingEntryPublished which is called when tests report additional data. Those entries should be displayed/logged in gradle's output when a test fails. They should also show up in the generated html report.

Current Behavior

Nothing is shown.

Context

IntelliJ, Eclipse, Maven and the junit-platform-gradle-plugin actually report the entries.

Those entries usually give important additional information to interpret test results.
For example, jqwik (http://jqwik.net) uses those entries to report the random seed
which is needed to reproduce errors.

Steps to Reproduce (for bugs)

Running the following test through gradle should not only report a failure but also something
like timestamp = 2018-03-06T15:40:33.704, key = value (IntelliJ's output)

@Test
void failingTestWithReporting(TestReporter reporter) {
	reporter.publishEntry("key", "value");
	org.junit.jupiter.api.Assertions.fail("Fails with an additional report entry");
}

Your Environment

OSX, Java 8, Gradle 4.6

blindpirate added a commit that referenced this issue Mar 11, 2018

Supports report publishing API in JUnit Platform.
This fixes #4605

In JUnit Platform, users can use TestExecutionListener.reportingEntryPublished
to report some information in test. However, this API is ignored in previous
Gradle implementation. This PR adds support for this API.

blindpirate added a commit that referenced this issue Mar 11, 2018

Supports report publishing API in JUnit Platform.
This fixes #4605

In JUnit Platform, users can use TestExecutionListener.reportingEntryPublished
to report some information in test. However, this API is ignored in previous
Gradle implementation. This PR adds support for this API.

blindpirate added a commit that referenced this issue Mar 13, 2018

Supports report publishing API in JUnit Platform
This fixes #4605

In JUnit Platform, users can use TestExecutionListener.reportingEntryPublished
to report some information in test. However, this API is ignored in previous
Gradle implementation. This PR adds support for this API.

blindpirate added a commit that referenced this issue Mar 14, 2018

Supports report publishing API in JUnit Platform
This fixes #4605

In JUnit Platform, users can use TestExecutionListener.reportingEntryPublished
to report some information in test. However, this API is ignored in previous
Gradle implementation. This PR adds support for this API.

blindpirate added a commit that referenced this issue Mar 15, 2018

Supports report publishing API in JUnit Platform
This fixes #4605

In JUnit Platform, users can use TestExecutionListener.reportingEntryPublished
to report some information in test. However, this API is ignored in previous
Gradle implementation. This PR adds support for this API.

@blindpirate blindpirate self-assigned this Mar 15, 2018

@blindpirate blindpirate added this to the 4.7 RC1 milestone Mar 15, 2018

@oehme oehme modified the milestones: 4.7 RC1, 4.8 RC1 Mar 19, 2018

@oehme

This comment has been minimized.

Member

oehme commented Mar 19, 2018

As mentioned on the #4716:

We should overhaul our test logging infrastructure holistically. We have already added display names, but only to the html report and console log. Now we are considering JUnit 5's reporting entries.

We should decide what things we want to support in a cross-cutting way (so that other test frameworks can be adapted to it). A display name makes a lot of sense. A Map<String, String> of "additional data" also seems like something other test frameworks might be interested in.

This information should be exposed to

  • Build scans
  • TestListeners
  • Tooling API (so IDEs can display it)
  • Console
  • HTML report

@blindpirate before we continue here we should have a small design spec that addresses the above and proposes an API. We should make sure this API also makes sense for the native tests frameworks for instance.

We should not just add reporting entries as test output, because otherwise people might start depending on this implementation detail and try to parse that output. Then we'd be stuck if we wanted to change it to something more strongly modeled. Let's do some design first.

@sbrannen

This comment has been minimized.

sbrannen commented Mar 20, 2018

Please keep in mind that reporting features like display names and reporting entries (via a map of key-value pairs) will end up in the new reporting format from the Open Test Alliance. So that should make them mainstream... once that new format is supported.

@oehme

This comment has been minimized.

Member

oehme commented Mar 20, 2018

@sbrannen is there already a spec for the format?

@sbrannen

This comment has been minimized.

sbrannen commented Mar 20, 2018

No, unfortunately not (yet).

That will be discussed here: ota4j-team/opentest4j#9

So feel free to chime in there with comments.

@oehme oehme changed the title from JUnit Platform: Reporting entries should be displayed on failing tests to Support JUnit platform reporting entries Apr 5, 2018

@oehme oehme removed this from the 4.8 RC1 milestone Apr 10, 2018

@jlink

This comment has been minimized.

jlink commented Sep 3, 2018

Any updates on this one?

With JUnit platform 1.3.0 looming, the old gradle plugin will no longer be supported and there's no way to run up-to-date junit 5 engines with reporting support.

@jlink

This comment has been minimized.

jlink commented Sep 17, 2018

Any updates on this one? JUnit platform 1.3.1 is already out.

@marcphilipp

This comment has been minimized.

Member

marcphilipp commented Sep 18, 2018

I'm afraid not. We're busy working on Gradle 5.0 at the moment. I hope to tackle this and other JUnit Platform issues (e.g. #6453) in the 5.1 timeframe.

@jlink

This comment has been minimized.

jlink commented Sep 18, 2018

What a bummer. Especially since I cannot think of any workaround.

@marcphilipp

This comment has been minimized.

Member

marcphilipp commented Sep 18, 2018

You could register a TestExecutionListener that prints them to stdout where Gradle should pick them up and add them to the test report.

@jlink

This comment has been minimized.

jlink commented Sep 18, 2018

Is there anyway to get stdout added to Gradle console output in case of a test failure? That's what I'd actually need since the test report is a few clicks away.

@sbrannen

This comment has been minimized.

sbrannen commented Sep 18, 2018

@jlink, I don't know if you can do that just on error, but have you tried testLogging.showStandardStreams = true in the test task?

@jlink

This comment has been minimized.

jlink commented Sep 18, 2018

@sbrannen @marcphilipp Both things together should make a simple mechanism possible. I'll see what I can figure out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment