8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls #337

shipilev · 2024-03-06T19:12:00Z

This resolves potentially catastrophic bug in monitor deflation. Thread dumps are routinely requested by profilers, so it is a real in-production risk. It would be more prominent as we backport improvements in monitor deflation code like JDK-8319048. The interaction between deflation thread that can be stopped at safepoint in the middle of deflation and the VM op that deflates monitors itself may corrupt the VM state.

This series of backports moves all deflation to monitor deflation thread, avoiding the issue. There are 4 interconnected issues, which are backported here atomically:

JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls. The core part, deferring deflation to monitor deflation thread. The new test needs -XX:+UnlockExperimentalVMOptions to gain access to -XX:LockingMode. Otherwise applies cleanly. It needs JDK-8320515 as the followup.
JDK-8319896: Remove monitor deflation from final audit. Removes the remaining part that might call into deflation outside of monitor deflation thread. Applies cleanly. Makes JDK-8320515 backport clean. It needs JDK-8325437 as the followup.
JDK-8320515: assert(monitor->object_peek() != nullptr) failed. Owned monitors should not have a dead object. Follow-up for JDK-8318757, fixes the JNI interaction problem. Does not apply cleanly due to minor conflict in JtregNativeHotspot.gmk.
JDK-8325437: Safepoint polling in monitor deflation can cause massive logs. Followup for JDK-8319896, reducing the logs noise.

@stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)

Additional testing:

Linux x86_64 server fastdebug, all tests pass

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue
JDK-8318757 needs maintainer approval
JDK-8319896 needs maintainer approval
JDK-8320515 needs maintainer approval
JDK-8325437 needs maintainer approval

Issues

JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls (Bug - P2 - Approved)
JDK-8319896: Remove monitor deflation from final audit (Enhancement - P4 - Approved)
JDK-8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object (Bug - P3 - Approved)
JDK-8325437: Safepoint polling in monitor deflation can cause massive logs (Enhancement - P4 - Approved)

Reviewers

Stefan Karlsson (@stefank - Reviewer)
Andrew Dinn (@adinn - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk21u-dev.git pull/337/head:pull/337
$ git checkout pull/337

Update a local copy of the PR:
$ git checkout pull/337
$ git pull https://git.openjdk.org/jdk21u-dev.git pull/337/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 337

View PR using the GUI difftool:
$ git pr show -t 337

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk21u-dev/pull/337.diff

Webrev

Link to Webrev Comment

…or::deflate_monitor calls Needs -XX:+UnlockExperimentalVMOptions in new test Backport 87be6b69fe985eee01fc3344f9153d774db792c1

Backport 369bbecc0dab389b523c09bc332fe1cf6394cb26

…: Owned monitors should not have a dead object Conflicts in JtregNativeHotspot.gmk due to absent JDK-8311541 Reviewed-by: dholmes, ihse, sspitsyn, dcubed

…assive logs Backport 52d497619e58a5677bc4a015b1bd87f600f23837

bridgekeeper · 2024-03-06T19:12:48Z

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2024-03-06T19:13:14Z

This backport pull request has now been updated with issue from the original commit.

shipilev · 2024-03-07T09:27:09Z

/issue add JDK-8318757
/issue add JDK-8319896
/issue add JDK-8320515
/issue add JDK-8325437

openjdk · 2024-03-07T09:27:24Z

@shipilev This issue is referenced in the PR title - it will now be updated.

openjdk · 2024-03-07T09:27:33Z

@shipilev
Adding additional issue to issue list: 8319896: Remove monitor deflation from final audit.

openjdk · 2024-03-07T09:28:12Z

@shipilev
Adding additional issue to issue list: 8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object.

openjdk · 2024-03-07T09:28:53Z

@shipilev
Adding additional issue to issue list: 8325437: Safepoint polling in monitor deflation can cause massive logs.

shipilev · 2024-03-07T09:45:25Z

GHA RISC-V failure is known environmental issue.

mlbridge · 2024-03-07T10:15:31Z

Webrevs

01: Full - Incremental (0706b0a5)
00: Full (b01c8a16)

stefank

@stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)

Yes, these are essentially the same patches that I've been testing for 21.0.4-oracle.

openjdk · 2024-03-12T10:49:50Z

⚠️ @shipilev This change is now ready for you to apply for maintainer approval. This can be done directly in each associated issue or by using the /approval command.

shipilev · 2024-03-12T11:19:24Z

@stefank -- it looks like you are tasked with backporting some of this to 21.0.4-oracle. Maybe you want to review this PR, which would also allow you to pick it up as commit later? :)

Yes, these are essentially the same patches that I've been testing for 21.0.4-oracle.

OK, great, thanks for looking.

shipilev · 2024-03-12T11:36:15Z

/approval 8318757 request This resolves potentially catastrophic bug in monitor deflation due to interleaving with thread dumping code that is frequently exercised by profilers. This is a part of atomic 21u integration, see 21u-dev PR for more details. Applies cleanly, but the new test needs -XX:+UnlockExperimentalVMOptions to gain access to -XX:LockingMode. After that amendment, all tests pass. Was in mainline for several months. Small bugtail: needs JDK-8320515 as the followup.

/approval 8319896 request This resolves the last place that might lead to bugs due to interleaving with concurrent monitor deflation. Less important than JDK-8318757, since this only happens on JVM shutdown, but it makes relevant backports like JDK-8320515 clean. This is a part of atomic 21u integration, see 21u-dev PR for more details. Applies cleanly. All tests pass. Was in mainline for several months. Small bugtail: needs JDK-8325437 as the followup.

/approval 8320515 request Fixes the JNI interaction problem, followup for JDK-8318757. This is a part of atomic 21u integration, see 21u-dev PR for more details. Does not apply cleanly due to minor conflict in JtregNativeHotspot.gmk. All tests pass. Was in mainline for several months, without bugtail.

/approval 8325437 request Reduces the logs noise, followup for JDK-8319896. This is a part of atomic 21u integration, see 21u-dev PR for more details. Applies cleanly. All tests pass. Was in mainline for a month, without bugtail.

openjdk · 2024-03-12T11:37:22Z

@shipilev
8318757: The approval request has been created successfully.

openjdk · 2024-03-12T11:37:32Z

@shipilev
8319896: The approval request has been created successfully.

openjdk · 2024-03-12T11:37:43Z

@shipilev
8320515: The approval request has been created successfully.

openjdk · 2024-03-12T11:37:49Z

@shipilev
8325437: The approval request has been created successfully.

openjdk · 2024-03-13T20:03:27Z

@shipilev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls
8319896: Remove monitor deflation from final audit
8320515: assert(monitor->object_peek() != nullptr) failed: Owned monitors should not have a dead object
8325437: Safepoint polling in monitor deflation can cause massive logs

Reviewed-by: stefank, adinn

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 21 new commits pushed to the master branch:

cc65d0d: 8325372: Shenandoah: SIGSEGV crash in unnecessary_acquire due to LoadStore split through phi
6d8d049: 8324632: Update Zlib Data Compression Library to Version 1.3.1
5431b82: 8322239: [macos] a11y : java.lang.NullPointerException is thrown when focus is moved on the JTabbedPane
5762df6: 8321489: Update LCMS to 2.16
a69c437: 8316559: Refactor some util/Calendar tests to JUnit
390e647: 8313612: Use JUnit in lib-test/jdk tests
539b04e: 8314832: Few runtime/os tests ignore vm flags
0d3a4e2: 8310818: Refactor more Locale tests to use JUnit
166aad0: 8310731: Configure a javax.net.ssl.SNIMatcher for the HTTP/1.1 test servers in java/net/httpclient tests
58b3403: 8310234: Refactor Locale tests to use JUnit
... and 11 more: https://git.openjdk.org/jdk21u-dev/compare/7505194e6b86fbda8af2eafb18b7667eec2d55ce...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

jerboaa · 2024-03-14T16:08:10Z

@adinn Could you please review this as well? Thanks.

adinn · 2024-03-14T16:12:58Z

@jerboaa Sure, I'll take a look.

adinn

Backport looks good. The only thing to note is that we might want to follow this up with whatever is done to resolve JDK-8320720.

jerboaa · 2024-03-15T14:33:26Z

Thanks!

jerboaa · 2024-03-15T14:39:02Z

/approve yes

openjdk · 2024-03-15T14:40:00Z

@jerboaa
8318757: The approval request has been approved.
8319896: The approval request has been approved.
8320515: The approval request has been approved.
8325437: The approval request has been approved.

shipilev · 2024-03-18T08:57:56Z

Thanks all!

/integrate

openjdk · 2024-03-18T08:59:14Z

Going to push as commit d1af31b.
Since your change was applied there have been 21 commits pushed to the master branch:

cc65d0d: 8325372: Shenandoah: SIGSEGV crash in unnecessary_acquire due to LoadStore split through phi
6d8d049: 8324632: Update Zlib Data Compression Library to Version 1.3.1
5431b82: 8322239: [macos] a11y : java.lang.NullPointerException is thrown when focus is moved on the JTabbedPane
5762df6: 8321489: Update LCMS to 2.16
a69c437: 8316559: Refactor some util/Calendar tests to JUnit
390e647: 8313612: Use JUnit in lib-test/jdk tests
539b04e: 8314832: Few runtime/os tests ignore vm flags
0d3a4e2: 8310818: Refactor more Locale tests to use JUnit
166aad0: 8310731: Configure a javax.net.ssl.SNIMatcher for the HTTP/1.1 test servers in java/net/httpclient tests
58b3403: 8310234: Refactor Locale tests to use JUnit
... and 11 more: https://git.openjdk.org/jdk21u-dev/compare/7505194e6b86fbda8af2eafb18b7667eec2d55ce...master

Your commit was automatically rebased without conflicts.

openjdk · 2024-03-18T08:59:22Z

@shipilev Pushed as commit d1af31b.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

shipilev added 4 commits March 6, 2024 13:30

Unclean JDK-8318757: VM_ThreadDump asserts in interleaved ObjectMonit…

a573b63

…or::deflate_monitor calls Needs -XX:+UnlockExperimentalVMOptions in new test Backport 87be6b69fe985eee01fc3344f9153d774db792c1

Clean JDK-8319896: Remove monitor deflation from final audit

2a52d89

Backport 369bbecc0dab389b523c09bc332fe1cf6394cb26

Unclean JDK-8320515: assert(monitor->object_peek() != nullptr) failed…

95e7827

…: Owned monitors should not have a dead object Conflicts in JtregNativeHotspot.gmk due to absent JDK-8311541 Reviewed-by: dholmes, ihse, sspitsyn, dcubed

Clean JDK-8325437: Safepoint polling in monitor deflation can cause m…

b01c8a1

…assive logs Backport 52d497619e58a5677bc4a015b1bd87f600f23837

openjdk bot changed the title ~~Backport 87be6b69fe985eee01fc3344f9153d774db792c1~~ 8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls Mar 6, 2024

openjdk bot added the backport label Mar 6, 2024

shipilev marked this pull request as ready for review March 7, 2024 10:10

openjdk bot added the rfr Pull request is ready for review label Mar 7, 2024

Merge branch 'master' into JDK-MULTI-deflation

0706b0a

stefank approved these changes Mar 12, 2024

View reviewed changes

openjdk bot added the approval label Mar 12, 2024

adinn reviewed Mar 15, 2024

View reviewed changes

adinn approved these changes Mar 15, 2024

View reviewed changes

openjdk bot added ready Pull request is ready to be integrated and removed approval labels Mar 15, 2024

openjdk bot added the integrated Pull request has been integrated label Mar 18, 2024

openjdk bot closed this Mar 18, 2024

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Mar 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls #337

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls #337

shipilev commented Mar 6, 2024 •

edited by openjdk bot

Loading

bridgekeeper bot commented Mar 6, 2024

openjdk bot commented Mar 6, 2024

shipilev commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

shipilev commented Mar 7, 2024

mlbridge bot commented Mar 7, 2024 •

edited

Loading

stefank left a comment

openjdk bot commented Mar 12, 2024

shipilev commented Mar 12, 2024

shipilev commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 13, 2024 •

edited

Loading

jerboaa commented Mar 14, 2024

adinn commented Mar 14, 2024

adinn left a comment

jerboaa commented Mar 15, 2024

jerboaa commented Mar 15, 2024

openjdk bot commented Mar 15, 2024

shipilev commented Mar 18, 2024

openjdk bot commented Mar 18, 2024

openjdk bot commented Mar 18, 2024

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls #337

8318757: VM_ThreadDump asserts in interleaved ObjectMonitor::deflate_monitor calls #337

Conversation

shipilev commented Mar 6, 2024 • edited by openjdk bot Loading

Progress

Issues

Reviewers

Reviewing

Webrev

bridgekeeper bot commented Mar 6, 2024

openjdk bot commented Mar 6, 2024

shipilev commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

openjdk bot commented Mar 7, 2024

shipilev commented Mar 7, 2024

mlbridge bot commented Mar 7, 2024 • edited Loading

Webrevs

stefank left a comment

Choose a reason for hiding this comment

openjdk bot commented Mar 12, 2024

shipilev commented Mar 12, 2024

shipilev commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 12, 2024

openjdk bot commented Mar 13, 2024 • edited Loading

jerboaa commented Mar 14, 2024

adinn commented Mar 14, 2024

adinn left a comment

Choose a reason for hiding this comment

jerboaa commented Mar 15, 2024

jerboaa commented Mar 15, 2024

openjdk bot commented Mar 15, 2024

shipilev commented Mar 18, 2024

openjdk bot commented Mar 18, 2024

openjdk bot commented Mar 18, 2024

shipilev commented Mar 6, 2024 •

edited by openjdk bot

Loading

mlbridge bot commented Mar 7, 2024 •

edited

Loading

openjdk bot commented Mar 13, 2024 •

edited

Loading