8294403: [REDO] make test should report only on executed tests

[shipilev]

This should help to speed up tests significantly. Currently, if we run "make test" with a subset of tests, JTReg would still read the entirety of test root to report on tests that were not run. Even with current suite of tests it gets expensive. If you add more tests to suite -- for example, mounting a large test archive like pre-generated fuzzer tests -- it starts to take a lot of time.

The reasonable default for older jtreg is -report:executed. My own CI -- that has millions of tests mounted in test/ -- was running with JTREG="OPTIONS=-report:executed" for years now. CODETOOLS-7903323 provides a new reporting option, -report:files, which makes this whole business easier. This requires new jtreg.

Motivational improvements:

$ time CONF=linux-x86_64-server-release make run-test TEST=gc/epsilon/TestHelloWorld.java

# Default JDK tree

# Baseline
real	0m9.086s
user	0m22.857s
sys	0m4.202s

# Patched
real	0m6.406s   ; +40% faster
user	0m17.156s
sys	0m3.193s

# +100K fuzzer tests in tree

# Baseline
real	3m1.997s
user	3m16.643s
sys	0m10.490s

# Patched
real	0m8.919s   ; 20x faster
user	0m17.904s
sys	0m4.860s

Additional testing:

• Ad-hoc timing tests (see above)
• Reproducer from CODETOOLS-7903331
• Eyeballing the test results on passing tests with REPEAT_COUNT > 1
• Eyeballing the test results on intermittently failing tests with RETRY_COUNT > 1

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8294403: [REDO] make test should report only on executed tests

Reviewers

• Erik Joelsson (@erikj79 - Reviewer) ⚠️ Review applies to a803958f
• David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to a803958f
• Jaikiran Pai (@jaikiran - Reviewer)
• Daniel Jeliński (@djelinski - Committer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/11824/head:pull/11824
$ git checkout pull/11824

Update a local copy of the PR:
$ git checkout pull/11824
$ git pull https://git.openjdk.org/jdk pull/11824/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 11824

View PR using the GUI difftool:
$ git pr show -t 11824

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/11824.diff

[bridgekeeper]

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@shipilev The following label will be automatically applied to this pull request:

• build

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (a82ac6cc)
• 00: Full (a803958f)

[shipilev]

This might need more exposure.

/label add hotspot
/label add core-libs

[openjdk]

@shipilev
The hotspot label was successfully added.

[openjdk]

@shipilev
The core-libs label was successfully added.

[erikj79]

Looks good from a build point of view.

[openjdk]

@shipilev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8294403: [REDO] make test should report only on executed tests

Reviewed-by: erikj, dholmes, jpai, djelinski

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 27 new commits pushed to the master branch:

• df1caf9: Merge
• a17f505: 8299476: PPC64 Zero build fails after JDK-8286302
• b743519: 8293824: gc/whitebox/TestConcMarkCycleWB.java failed "RuntimeException: assertTrue: expected true, was false"
• 8254cbb: 8299483: ProblemList java/text/Format/NumberFormat/CurrencyFormat.java
• 031829d: 8298324: Unable to run shell test with make
• 3eb85d1: 8299235: broken link referencing missing id
• 6a07fd0: 8155259: Suspicious buffer allocation in com.sun.tools.javac.file.BaseFileManager
• 4c0f24e: 8064931: tools/javac/scope/DupUnsharedTest.java needs to be updated to add the bug id
• c6588d5: 8299158: Improve MD5 intrinsic on AArch64
• c32a34c: 8299072: java_lang_ref_Reference::clear_referent should be GC agnostic
• ... and 17 more: https://git.openjdk.org/jdk/compare/67086ebf80109a623f3c2ad24d4e1a65de43d986...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[dholmes-ora]

This seems fine to me. Thanks.

I personally always run with -noreport. :)

[jaikiran]

Hello Aleksey, this looks good to me. Good to see the timing improvements on make test.

Please update the copyright years on the files before integrating.

I see that in the additional testing section you note, running tests with RETRY_COUNT. Is that intentional? Does retry count play a role in the report generation?

[shipilev]

Please update the copyright years on the files before integrating.

Updated.

I see that in the additional testing section you note, running tests with RETRY_COUNT. Is that intentional? Does retry count play a role in the report generation?

The logic for REPEAT/RETRY_COUNT merges the jtreg reports from "different" repeated runs. We have to check new reporting mode still produces sane results in that mode.

[jaikiran]

I see that in the additional testing section you note, running tests with RETRY_COUNT. Is that intentional? Does retry count play a role in the report generation?

The logic for REPEAT/RETRY_COUNT merges the jtreg reports from "different" repeated runs. We have to check new reporting mode still produces sane results in that mode.

Thank you for that detail, I wasn't aware of that.

[djelinski]

I like this!

$ time make run-test TEST=jdk/java/net/BindException/Test.java:

# baseline:
real    0m43.176s
user    0m12.453s
sys     1m13.797s
# patched:
real    0m22.558s
user    0m10.797s
sys     1m1.734s

[shipilev]

My longer testing passes well. I would like to integrate it soon, if there are no objections.

[shipilev]

/integrate

[openjdk]

Going to push as commit 5598acc.
Since your change was applied there have been 51 commits pushed to the master branch:

• 88f0ea7: 8299726: [cleanup] Some code cleanup in opto/compile.hpp
• 0234f81: 8298447: Unnecessary Vector usage in DocPrintJob implementations
• cc4936a: 8298720: Insufficient error handling when CodeBuffer is exhausted
• b5b7948: 8298240: Replace the usage of ImageLayoutException by the CMMException
• 99be740: 8299306: Test "javax/swing/JFileChooser/FileSystemView/CustomFSVLinkTest.java" fails on Windows 10 x64 because there are some buttons did not display button name
• 775da84: 8299412: JNI call of getAccessibleActionCount on a wrong thread
• 7845b0d: 8296934: Write a test to verify whether Undecorated Frame can be iconified or not
• 4b6809b: 8299693: Change to Xcode12.4+1.1 devkit for building on macOS at Oracle
• dfdbd0f: 8299343: Windows: Invalid thread_native_entry declaration
• 3e2314d: 8299254: Support dealing with standard assert macro
• ... and 41 more: https://git.openjdk.org/jdk/compare/67086ebf80109a623f3c2ad24d4e1a65de43d986...master

Your commit was automatically rebased without conflicts.

[openjdk]

@shipilev Pushed as commit 5598acc.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.