7903324: Improve per-class reporting of JUnit tests, in .jtr file

[sormuras]

Improve per-class reporting of JUnit tests by using a custom TestExecutionListener in JUnitRunner.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• CODETOOLS-7903324: Improve per-class reporting of JUnit tests, in .jtr file

Reviewers

• Jonathan Gibbons (@jonathan-gibbons - Reviewer) ⚠️ Review applies to 148dc5d3

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jtreg pull/127/head:pull/127
$ git checkout pull/127

Update a local copy of the PR:
$ git checkout pull/127
$ git pull https://git.openjdk.org/jtreg pull/127/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 127

View PR using the GUI difftool:
$ git pr show -t 127

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jtreg/pull/127.diff

[bridgekeeper]

👋 Welcome back cstein! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[sormuras]

Here are some sample output snippets copied from .jtr files collected by running self-tests if jtreg.

Regenerating...

[sormuras]

Here's an output sample for a modified JUnit Trace self-test with a failing case, namely using "123" as a non-blank data point:

SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[1] test('null')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[2] test('')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')'
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[4] test('  ')'
    FAILED: JupiterTests::nullEmptyAndBlankStrings '[5] test('123')'
org.opentest4j.AssertionFailedError: Input text "123" not blank ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
	at JupiterTests.nullEmptyAndBlankStrings(JupiterTests.java:63)
[...]
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)
SUCCESSFUL: JupiterTests::nullEmptyAndBlankStrings '[6] test('	')'
SUCCESSFUL: JupiterTests::succeedingTest 'succeedingTest()'
   ABORTED: JupiterTests::abortedTest 'abortedTest()'
org.opentest4j.TestAbortedException: Assumption failed: abort test execution mid-flight
JavaTest Message: JUnit Platform Failure(s): 1

[ JUnit Containers: found 4, started 4, succeeded 4, failed 0, aborted 0, skipped 0]
[ JUnit Tests: found 9, started 8, succeeded 6, failed 1, aborted 1, skipped 1]

java.lang.Exception: JUnit test failure
	at com.sun.javatest.regtest.agent.JUnitRunner.runWithJUnitPlatform(JUnitRunner.java:154)
	at com.sun.javatest.regtest.agent.JUnitRunner.main(JUnitRunner.java:96)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)

JavaTest Message: Test threw exception: java.lang.Exception
JavaTest Message: shutting down test

Note: I pruned a lot of lines from the exception stack trace.

[mlbridge]

Webrevs

• 03: Full - Incremental (d2268878)
• 02: Full - Incremental (0e606f2d)
• 01: Full - Incremental (148dc5d3)
• 00: Full (da0cafe2)

[sormuras]

The lines in the sample .jtr now read:

----------System.err:(31/1529)----------
STARTED    JupiterTests::nullEmptyAndBlankStrings '[1] test('null')'
SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[1] test('null')'
STARTED    JupiterTests::nullEmptyAndBlankStrings '[2] test('')'
SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[2] test('')'
STARTED    JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')'
SUCCESSFUL JupiterTests::nullEmptyAndBlankStrings '[3] test(' ')'
[...]
STARTED    JupiterTests::nullEmptyAndBlankStrings '[5] test('123')'
org.opentest4j.AssertionFailedError: Input text "123" not blank ==> expected: <true> but was: <false>
	at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
	at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:40)
	at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
	at JupiterTests.nullEmptyAndBlankStrings(JupiterTests.java:63)
[...]
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
	at java.base/java.lang.Thread.run(Thread.java:833)
FAILED      JupiterTests::nullEmptyAndBlankStrings '[5] test('123')'
STARTED    JupiterTests::succeedingTest 'succeedingTest()'
SUCCESSFUL JupiterTests::succeedingTest 'succeedingTest()'
SKIPPED    JupiterTests::skippedTest 'skippedTest()' for demonstration purposes
STARTED    JupiterTests::abortedTest 'abortedTest()'
org.opentest4j.TestAbortedException: Assumption failed: abort test execution mid-flight
ABORTED    JupiterTests::abortedTest 'abortedTest()'

[openjdk]

@sormuras This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

7903324: Improve per-class reporting of JUnit tests, in .jtr file

Reviewed-by: jjg

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 2 new commits pushed to the master branch:

• 1ad8188: 7903371: Use multi-catch where applicable
• b44b446: 7903370: Update openjdk.java.net URLs to openjdk.org

Please see this link for an up-to-date comparison between the source branch of this pull request and the master branch.
As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@jonathan-gibbons) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

[jonathan-gibbons]

Should there be an entry in the changelog for this?

[sormuras]

/integrate

[openjdk]

@sormuras
Your change (at version d226887) is now ready to be sponsored by a Committer.

[jonathan-gibbons]

/sponsor

[openjdk]

Going to push as commit adff8b0.
Since your change was applied there have been 2 commits pushed to the master branch:

• 1ad8188: 7903371: Use multi-catch where applicable
• b44b446: 7903370: Update openjdk.java.net URLs to openjdk.org

Your commit was automatically rebased without conflicts.

[openjdk]

@jonathan-gibbons @sormuras Pushed as commit adff8b0.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.