8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies #15631

roy-soumadipta · 2023-09-08T09:34:36Z

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies

serviceability/sa/TestJmapCoreMetaspace.java runs in hotspot:tier2, and takes about 330 seconds out of 670 seconds of the entire run on x86_64 fastdebug. The tier2 completion usually waits on it. Profiling shows we are spending time during classloading, checking dependencies. Since we load lots of classes, we do it many times, and each time the cost grows linearly to the number of classes. Some other tests that target metaspace explicitly disable VerifyDependencies to avoid this.

The commit includes adding "-XX:-VerifyDependencies" along with "-XX:+IgnoreUnrecognizedVMOptions" in TestJmapCore.java to make sure that release builds work with the test, as VerifyDependencies is a develop option and not available in release builds. The changes in the commit are resulting in considerable improvement for the test in fastdebug mode as can be seen below:

before: 130.41s user 14.11s system 118% cpu 2:01.85 total
after: 27.52s user 13.64s system 213% cpu 19.249 total

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies (Enhancement - P4)

Reviewers

Aleksey Shipilev (@shipilev - Reviewer)
Kevin Walls (@kevinjwalls - Committer) ⚠️ Review applies to a3aec144
Chris Plummer (@plummercj - Reviewer)
Leonid Mesnik (@lmesnik - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/15631/head:pull/15631
$ git checkout pull/15631

Update a local copy of the PR:
$ git checkout pull/15631
$ git pull https://git.openjdk.org/jdk.git pull/15631/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 15631

View PR using the GUI difftool:
$ git pr show -t 15631

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/15631.diff

Webrev

Link to Webrev Comment

… -XX:-VerifyDependencies serviceability/sa/TestJmapCoreMetaspace.java runs in hotspot:tier2, and takes about 330 seconds out of 670 seconds of the entire run on x86_64 fastdebug. The tier2 completion usually waits on it. Profiling shows we are spending time during classloading, checking dependencies. Since we load lots of classes, we do it many times, and each time the cost grows linearly to the number of classes. Some other tests that target metaspace explicitly disable `VerifyDependencies` to avoid this. The commit includes adding "-XX:-VerifyDependencies" along with "-XX:+IgnoreUnrecognizedVMOptions" in TestJmapCore.java to make sure that release builds work with the test, as VerifyDependencies is a develop option and not available in release builds. The changes in the commit is resulting in considerable improvement for the test in fastdebug mode as can be seen below: * before: **130.41s user 14.11s system 118% cpu 2:01.85 total** * after: **27.52s user 13.64s system 213% cpu 19.249 total**

bridgekeeper · 2023-09-08T09:35:31Z

👋 Welcome back roy-soumadipta! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2023-09-08T09:37:24Z

@roy-soumadipta The following label will be automatically applied to this pull request:

serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2023-09-08T09:41:56Z

Webrevs

shipilev · 2023-09-08T09:46:00Z

test/hotspot/jtreg/serviceability/sa/TestJmapCore.java

    static void test(String type) throws Throwable {
        ProcessBuilder pb = ProcessTools.createTestJvm("-XX:+CreateCoredumpOnCrash",
                "-Xmx512m", "-XX:MaxMetaspaceSize=64m", "-XX:+CrashOnOutOfMemoryError",
+                "-XX:-VerifyDependencies", "-XX:+IgnoreUnrecognizedVMOptions",


Have you tried this test with release? I think IgnoreUnrecognizedVMOptions is positional, and should go before VerifyDependencies to actually work.

I will try it and raise another commit if that is required

It would make sense to change the order, even if it works right now. This would specify what option is being ignored more clearly.

Hi Aleksey, release is working as well

shipilev

Looks good to me. Let's ask @plummercj to take a look as well.

openjdk · 2023-09-08T10:16:12Z

@roy-soumadipta This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies

Reviewed-by: shade, kevinw, cjplummer, lmesnik

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 58 new commits pushed to the master branch:

d06a564: 8315765: G1: Incorrect use of G1LastPLABAverageOccupancy
66b6a5a: 8277954: Replace use of monitors with explicit locks in the JDK LDAP provider implementation
4cb4637: 8315970: Big-endian issues after JDK-8310929
ae08143: 8315611: Open source swing text/html and tree test
7b3e697: 8315855: G1: Revise signature of set_humongous_candidate
1941290: 8315942: Sort platform enums and definitions after JDK-8304913 follow-ups
996b336: 8315781: Reduce the max value of GCDrainStackTargetSize
35bccac: 8315841: RISC-V: Check for hardware TSO support
a04c6c1: 8315609: Open source few more swing text/html tests
dab1c21: 8314491: Linux: jexec launched via PATH fails to find java
... and 48 more: https://git.openjdk.org/jdk/compare/024133b089d911dcc3ea70dfdaa6b150b14a9eb4...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@shipilev, @kevinjwalls, @plummercj, @lmesnik) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

kevinjwalls

Looks good.
I also assumed IgnoreUnrecognizedVMOptions would be positional, but it isn't. I checked various major Java versions and it works after an unknown option.
But I'm very pleased to see the update to have it appear first in the list, it just reads easier and removes that question.

plummercj

The changes look good, but I'd suggest adding a comment so the reader knows why these options were added.

lmesnik

It would be check if it is a debug build with 'Platform.isDebugBuild()' instead of IgnoreUnrecognizedVMOptions.
If test takes ~30 seconds it does it makes sense to reduce it's timeout to default and return it into tier1 group?

shipilev · 2023-09-11T08:58:44Z

The changes look good, but I'd suggest adding a comment so the reader knows why these options were added.

Agreed. Something like:

                "-Xmx512m", "-XX:MaxMetaspaceSize=64m", "-XX:+CrashOnOutOfMemoryError",
                // The test loads lots of small classes to exhaust Metaspace, skip method dependency
                // verification to improve performance in debug builds.
                "-XX:+IgnoreUnrecognizedVMOptions", "-XX:-VerifyDependencies",
                CoreUtils.getAlwaysPretouchArg(true),

roy-soumadipta · 2023-09-11T10:30:08Z

I'm adding a small comment @plummercj and 'Platform.isDebugBuild()' instead of IgnoreUnrecognizedVMOptions @lmesnik in the next commit. But I would like to defer the move to tier1 in a separate commit

…add comment about the change forbetter readability.

shipilev

Still good.

roy-soumadipta · 2023-09-11T16:28:17Z

/integrate

openjdk · 2023-09-11T16:29:35Z

@roy-soumadipta
Your change (at version 7b6af09) is now ready to be sponsored by a Committer.

shipilev · 2023-09-11T16:33:58Z

/sponsor

openjdk · 2023-09-11T16:34:58Z

Going to push as commit 877731d.
Since your change was applied there have been 58 commits pushed to the master branch:

d06a564: 8315765: G1: Incorrect use of G1LastPLABAverageOccupancy
66b6a5a: 8277954: Replace use of monitors with explicit locks in the JDK LDAP provider implementation
4cb4637: 8315970: Big-endian issues after JDK-8310929
ae08143: 8315611: Open source swing text/html and tree test
7b3e697: 8315855: G1: Revise signature of set_humongous_candidate
1941290: 8315942: Sort platform enums and definitions after JDK-8304913 follow-ups
996b336: 8315781: Reduce the max value of GCDrainStackTargetSize
35bccac: 8315841: RISC-V: Check for hardware TSO support
a04c6c1: 8315609: Open source few more swing text/html tests
dab1c21: 8314491: Linux: jexec launched via PATH fails to find java
... and 48 more: https://git.openjdk.org/jdk/compare/024133b089d911dcc3ea70dfdaa6b150b14a9eb4...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-09-11T16:35:12Z

@shipilev @roy-soumadipta Pushed as commit 877731d.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Backporting the fix for https://bugs.openjdk.org/browse/JDK-8315683 merged as part of openjdk/jdk#15631. https://github.com/openjdk/jdk/commit/877731d2a20249ce4724a071ba2da1faa56daca4.patch has been cleanly applied. Below are the test results: * after_release: **54.64s user 7.50s system 231% cpu 26.806 total** * after_fastdebug: **247.97s user 12.78s system 407% cpu 1:03.99 total** * before_release: **52.23s user 8.39s system 232% cpu 26.070 total** * before_fastdebug: **504.20s user 9.34s system 211% cpu 4:03.01 total**

Backporting the fix for https://bugs.openjdk.org/browse/JDK-8315770 merged as part of openjdk/jdk#15631. https://github.com/openjdk/jdk/commit/877731d2a20249ce4724a071ba2da1faa56daca4.patch apply failed due to differences in the file and the changes have been performed selectively. Below are the test results: * after_release: **99.63s user 10.34s system 171% cpu 1:04.25 total** * after_fastdebug: **433.45s user 17.35s system 401% cpu 1:52.15 total** * before_release: **102.65s user 10.50s system 172% cpu 1:05.65 total** * before_fastdebug: **516.82s user 16.68s system 253% cpu 3:30.51 total**

Backporting the fix for https://bugs.openjdk.org/browse/JDK-8315770 merged as part of openjdk/jdk#15631. https://github.com/openjdk/jdk/commit/877731d2a20249ce4724a071ba2da1faa56daca4.patch apply failed due to differences in the file and the changes have been performed selectively. Below are the test results: * after_release: **123.62s user 10.26s system 160% cpu 1:23.59 total** * after_fastdebug: **197.42s user 15.07s system 195% cpu 1:48.49 total** * before_release: **123.65s user 9.56s system 159% cpu 1:23.38 total** * before_fastdebug: **232.70s user 15.29s system 170% cpu 2:25.70 total**

openjdk bot added the rfr Pull request is ready for review label Sep 8, 2023

openjdk bot added the serviceability serviceability-dev@openjdk.org label Sep 8, 2023

shipilev reviewed Sep 8, 2023

View reviewed changes

Changing the order of flags to maintain consistency.

a3aec14

shipilev approved these changes Sep 8, 2023

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Sep 8, 2023

kevinjwalls reviewed Sep 8, 2023

View reviewed changes

kevinjwalls approved these changes Sep 8, 2023

View reviewed changes

plummercj approved these changes Sep 10, 2023

View reviewed changes

lmesnik reviewed Sep 10, 2023

View reviewed changes

Replace IgnoreUnrecognizedVMOptions with Platform.isDebugBuild() and …

7b6af09

…add comment about the change forbetter readability.

shipilev approved these changes Sep 11, 2023

View reviewed changes

lmesnik approved these changes Sep 11, 2023

View reviewed changes

plummercj approved these changes Sep 11, 2023

View reviewed changes

openjdk bot added the sponsor Pull request is ready to be sponsored label Sep 11, 2023

openjdk bot added the integrated Pull request has been integrated label Sep 11, 2023

openjdk bot closed this Sep 11, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 11, 2023

openjdk bot removed the sponsor Pull request is ready to be sponsored label Sep 11, 2023

roy-soumadipta deleted the TestJmapCoreMetaspace branch September 11, 2023 16:47

This was referenced Sep 25, 2023

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies openjdk/jdk21u#202

Closed

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies openjdk/jdk11u#85

Closed

roy-soumadipta mentioned this pull request Sep 25, 2023

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies openjdk/jdk17u-dev#1786

Closed

3 tasks

roy-soumadipta mentioned this pull request Sep 27, 2023

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies openjdk/jdk11u-dev#2150

Closed

3 tasks

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies #15631

8315770: serviceability/sa/TestJmapCoreMetaspace.java should run with -XX:-VerifyDependencies #15631

Uh oh!

Conversation

roy-soumadipta commented Sep 8, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented Sep 8, 2023

Uh oh!

openjdk bot commented Sep 8, 2023

Uh oh!

mlbridge bot commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

shipilev Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

roy-soumadipta Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

shipilev Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

roy-soumadipta Sep 8, 2023

Choose a reason for hiding this comment

Uh oh!

shipilev left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Sep 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kevinjwalls left a comment

Choose a reason for hiding this comment

Uh oh!

plummercj left a comment

Choose a reason for hiding this comment

Uh oh!

lmesnik left a comment

Choose a reason for hiding this comment

Uh oh!

shipilev commented Sep 11, 2023

Uh oh!

roy-soumadipta commented Sep 11, 2023

Uh oh!

shipilev left a comment

Choose a reason for hiding this comment

Uh oh!

roy-soumadipta commented Sep 11, 2023

Uh oh!

openjdk bot commented Sep 11, 2023

Uh oh!

shipilev commented Sep 11, 2023

Uh oh!

openjdk bot commented Sep 11, 2023

Uh oh!

openjdk bot commented Sep 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

roy-soumadipta commented Sep 8, 2023 •

edited by openjdk bot

Loading

mlbridge bot commented Sep 8, 2023 •

edited

Loading

openjdk bot commented Sep 8, 2023 •

edited

Loading