8270085: Suspend during block transition may deadlock if lock held #257

pchilano · 2021-07-20T20:06:12Z

Hi all,

The following patch fixes deadlocks issues that could occur when checking for suspension while holding VM locks. See the bug description for a concrete case. The solution is to avoid checking for suspend requests when using the TBIVM wrapper. The original patch was actually written by @robehn(now on vacations) and I only made small changes.

Tested in mach5 tiers1-7. I also verified that the new added test SuspendBlocked.java deadlocks with the current bits and passes with this patch.

Thanks,
 Patricio

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8270085: Suspend during block transition may deadlock if lock held

Reviewers

Daniel D. Daugherty (@dcubed-ojdk - Reviewer)
David Holmes (@dholmes-ora - Reviewer)
Coleen Phillimore (@coleenp - Reviewer) ⚠️ Review applies to 6ad7c4e

Contributors

Robbin Ehn <rehn@openjdk.org>
Patricio Chilano Mateo <pchilanomate@openjdk.org>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk17 pull/257/head:pull/257
$ git checkout pull/257

Update a local copy of the PR:
$ git checkout pull/257
$ git pull https://git.openjdk.java.net/jdk17 pull/257/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 257

View PR using the GUI difftool:
$ git pr show -t 257

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk17/pull/257.diff

bridgekeeper · 2021-07-20T20:07:28Z

👋 Welcome back pchilanomate! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-07-20T20:09:13Z

@pchilano The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

pchilano · 2021-07-20T21:19:54Z

/label add hotspot-runtime

pchilano · 2021-07-20T21:20:05Z

/label remove hotspot

openjdk · 2021-07-20T21:20:36Z

@pchilano
The hotspot-runtime label was successfully added.

openjdk · 2021-07-20T21:21:47Z

@pchilano
The hotspot label was successfully removed.

mlbridge · 2021-07-20T21:33:06Z

Webrevs

dcubed-ojdk

Thumbs up.

openjdk · 2021-07-20T21:57:04Z

@pchilano This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8270085: Suspend during block transition may deadlock if lock held

Co-authored-by: Robbin Ehn <rehn@openjdk.org>
Co-authored-by: Patricio Chilano Mateo <pchilanomate@openjdk.org>
Reviewed-by: dcubed, dholmes, coleenp

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 23 new commits pushed to the master branch:

39b486d: 8271126: ProblemList runtime/InvocationTests/invokevirtualTests.java
4119a52: 8270461: ZGC: Invalid oop passed to ZBarrierSetRuntime::load_barrier_on_oop_array
89f7998: 8266347: assert(Dependencies::is_concrete_root_method(fm, ctxk) == Dependencies::is_concrete_method(m, ctxk)) failed: mismatch
e1051ae: 8264066: Enhance compiler validation
add995b: 8265201: JarFile.getInputStream not validating invalid signed jars
ca6b222: 8258432: Improve File Transfers
a3acce5: 8264079: Improve abstractions
138f598: 8262380: Enhance XML processing passes
0e6c011: 8262967: Improve Zip file support
e48af00: 8264460: Improve NTLM support
... and 13 more: https://git.openjdk.java.net/jdk17/compare/845c31dc4b49dfbed25238a398c80b8cdd0a3997...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

pchilano · 2021-07-20T22:16:33Z

Thumbs up.

Thanks for the review Dan!

dholmes-ora

Looks good.

Sorry I missed the fact this should have been based on 17 to begin with.

coleenp

Just some comments and questions. Looks good!

coleenp · 2021-07-21T00:01:12Z

src/hotspot/share/runtime/handshake.cpp

+  // Handshakes cannot safely safepoint.
+  // The exception to this rule is the asynchronous suspension handshake.
+  // It by-passes the NSV by manually doing the transition.
+  NoSafepointVerifier nsv;


I have no idea what this comment means in this context. Below we take out a lock with _no_safepoint_check which is essentially a NSV. You just moved this comment so I don't suggest changing it at this time.

Right, it's created with _allow_vm_block = true so the NSV is redundant. The _no_safepoint_check actually means we don't check for safepoints if there is contention while acquiring the monitor but we don't do inc_no_safepoint_count() to enforce no safepoints later though.

A side effect of the no_safepoint_check locking is that it has an implicit NSV though. We do call inc_no_safepoint_count when setting the owner, but I remembered this wrong. It's not unconditional:

// NSV implied with locking allow_vm_block flag. // The tty_lock is special because it is released for the safepoint by // the safepoint mechanism. if (new_owner->is_Java_thread() && _allow_vm_block && this != tty_lock) { JavaThread::cast(new_owner)->inc_no_safepoint_count(); }

coleenp · 2021-07-21T00:01:30Z

src/hotspot/share/runtime/handshake.cpp

+    // actual suspend since Handshake::execute() above only installed
+    // the asynchronous handshake.
+    SafepointMechanism::process_if_requested(self);
+  }


Is this an optimization? Or can the thread escape the suspend request?

For self-suspend requests coming from Java we will suspend when going back to Java. In the case of JVMTI self-suspend requests I think Robbin found some JVMTI tests that were expecting to not return back to native without suspending.

We should document the order of events why this code is needed in a future RFE, because this comment doesn't really explain it to me and knowing where to put all these process_if_requested() and process() calls is far from obvious. I believe you and Robbin, but there are too many mysteries.

I commented the line and run all the serviceability/ tests to look for issues. The only test that failed was SuspendWithCurrentThread.java due to the recent 8264663 change (openjdk/jdk@b5c6351). The patch aimed to verify that the self-suspending thread actually suspended and didn't exited, but the check is too restrictive because it doesn't consider that the JT will still suspend when going back to Java. That check can be moved to Java to verify the suspender didn't return from suspendTestedThreads().
Maybe I should do that in another RFE along with removing this code?

Yes, that would be good to clear up in a future RFE.

coleenp · 2021-07-21T00:02:59Z

test/hotspot/jtreg/runtime/handshake/SuspendBlocked.java

+/*
+ * @test SuspendBlocked
+ * @bug 8270085
+ * @library /testlibrary /test/lib


I think this is referring to /testlibrary that I removed. Can you see if it's still needed?

Not needed, removed.

pchilano · 2021-07-21T01:42:49Z

Looks good.

Sorry I missed the fact this should have been based on 17 to begin with.

Thanks for the review David!

pchilano · 2021-07-21T01:46:59Z

Just some comments and questions. Looks good!

Thanks for the review Coleen!

coleenp

Still good.

coleenp · 2021-07-21T12:37:14Z

src/hotspot/share/runtime/handshake.cpp

+  // Handshakes cannot safely safepoint.
+  // The exception to this rule is the asynchronous suspension handshake.
+  // It by-passes the NSV by manually doing the transition.
+  NoSafepointVerifier nsv;


A side effect of the no_safepoint_check locking is that it has an implicit NSV though. We do call inc_no_safepoint_count when setting the owner, but I remembered this wrong. It's not unconditional:

// NSV implied with locking allow_vm_block flag. // The tty_lock is special because it is released for the safepoint by // the safepoint mechanism. if (new_owner->is_Java_thread() && _allow_vm_block && this != tty_lock) { JavaThread::cast(new_owner)->inc_no_safepoint_count(); }

coleenp · 2021-07-21T12:41:40Z

src/hotspot/share/runtime/handshake.cpp

+    // actual suspend since Handshake::execute() above only installed
+    // the asynchronous handshake.
+    SafepointMechanism::process_if_requested(self);
+  }


We should document the order of events why this code is needed in a future RFE, because this comment doesn't really explain it to me and knowing where to put all these process_if_requested() and process() calls is far from obvious. I believe you and Robbin, but there are too many mysteries.

dcubed-ojdk

Still good.

pchilano · 2021-07-21T21:21:28Z

While chatting with Coleen I realized the comment in should_process_no_suspend() wasn't accurate so I fixed it. Not necessarily there will be a suspend request when reaching that path. The poll could be armed just because a safepoint or a non-suspend handshake operation was executed while the thread was safepoint safe.

dcubed-ojdk

Thanks for adjusting the comment. Still good.

openjdk · 2021-07-22T14:00:15Z

@pchilano Could not parse @robehn @pchilano as a valid contributor.
Syntax: /contributor (add|remove) [@user | openjdk-user | Full Name <email@address>]. For example:

/contributor add @openjdk-bot
/contributor add duke
/contributor add J. Duke <duke@openjdk.org>

pchilano · 2021-07-22T14:01:46Z

/contributor add @robehn

pchilano · 2021-07-22T14:01:57Z

/contributor add @pchilano

openjdk · 2021-07-22T14:02:45Z

@pchilano
Contributor Robbin Ehn <rehn@openjdk.org> successfully added.

openjdk · 2021-07-22T14:03:52Z

@pchilano
Contributor Patricio Chilano Mateo <pchilanomate@openjdk.org> successfully added.

pchilano · 2021-07-22T14:28:06Z

Thanks @dholmes-ora, @dcubed-ojdk and @coleenp for the reviews!

pchilano · 2021-07-22T14:29:26Z

/integrate

openjdk · 2021-07-22T14:30:29Z

Going to push as commit e7f9009.
Since your change was applied there have been 23 commits pushed to the master branch:

39b486d: 8271126: ProblemList runtime/InvocationTests/invokevirtualTests.java
4119a52: 8270461: ZGC: Invalid oop passed to ZBarrierSetRuntime::load_barrier_on_oop_array
89f7998: 8266347: assert(Dependencies::is_concrete_root_method(fm, ctxk) == Dependencies::is_concrete_method(m, ctxk)) failed: mismatch
e1051ae: 8264066: Enhance compiler validation
add995b: 8265201: JarFile.getInputStream not validating invalid signed jars
ca6b222: 8258432: Improve File Transfers
a3acce5: 8264079: Improve abstractions
138f598: 8262380: Enhance XML processing passes
0e6c011: 8262967: Improve Zip file support
e48af00: 8264460: Improve NTLM support
... and 13 more: https://git.openjdk.java.net/jdk17/compare/845c31dc4b49dfbed25238a398c80b8cdd0a3997...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-07-22T14:30:42Z

@pchilano Pushed as commit e7f9009.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

v1

744162e

openjdk bot added the hotspot hotspot-dev@openjdk.java.net label Jul 20, 2021

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.java.net label Jul 20, 2021

openjdk bot removed the hotspot hotspot-dev@openjdk.java.net label Jul 20, 2021

pchilano marked this pull request as ready for review July 20, 2021 21:28

openjdk bot added the rfr Pull request is ready for review label Jul 20, 2021

dcubed-ojdk approved these changes Jul 20, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jul 20, 2021

pchilano mentioned this pull request Jul 20, 2021

8270085: Suspend during block transition may deadlock if lock held openjdk/jdk#4828

Closed

3 tasks

pchilano closed this Jul 20, 2021

pchilano reopened this Jul 20, 2021

dholmes-ora approved these changes Jul 20, 2021

View reviewed changes

coleenp approved these changes Jul 21, 2021

View reviewed changes

remove /testlibrary from SuspendBlocked.java

6ad7c4e

coleenp approved these changes Jul 21, 2021

View reviewed changes

dcubed-ojdk approved these changes Jul 21, 2021

View reviewed changes

fix comment in should_process_no_suspend()

6cfa771

dcubed-ojdk approved these changes Jul 21, 2021

View reviewed changes

dholmes-ora approved these changes Jul 22, 2021

View reviewed changes

openjdk bot closed this Jul 22, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jul 22, 2021

8270085: Suspend during block transition may deadlock if lock held #257

8270085: Suspend during block transition may deadlock if lock held #257

Conversation

pchilano commented Jul 20, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Contributors

Reviewing

bridgekeeper bot commented Jul 20, 2021

openjdk bot commented Jul 20, 2021

pchilano commented Jul 20, 2021

pchilano commented Jul 20, 2021

openjdk bot commented Jul 20, 2021

openjdk bot commented Jul 20, 2021

mlbridge bot commented Jul 20, 2021 • edited Loading

Webrevs

dcubed-ojdk left a comment

Choose a reason for hiding this comment

openjdk bot commented Jul 20, 2021 • edited Loading

pchilano commented Jul 20, 2021

dholmes-ora left a comment

Choose a reason for hiding this comment

coleenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pchilano commented Jul 21, 2021

pchilano commented Jul 21, 2021

coleenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcubed-ojdk left a comment

Choose a reason for hiding this comment

pchilano commented Jul 21, 2021

dcubed-ojdk left a comment

Choose a reason for hiding this comment

openjdk bot commented Jul 22, 2021

pchilano commented Jul 22, 2021

pchilano commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

pchilano commented Jul 22, 2021

pchilano commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

openjdk bot commented Jul 22, 2021

pchilano commented Jul 20, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Jul 20, 2021 •

edited

Loading

openjdk bot commented Jul 20, 2021 •

edited

Loading