8270085: Suspend during block transition may deadlock if lock held #4828

pchilano · 2021-07-19T14:34:55Z

Hi all,

The following patch fixes deadlocks issues that could occur when checking for suspension while holding VM locks. See the bug description for a concrete case. The solution is to avoid checking for suspend requests when using the TBIVM wrapper. The original patch was actually written by @robehn(now on vacations) and I only made small changes.

Run tiers1-6 in mach5 and currently running tier7. I also verified that the new added test SuspendBlocked.java deadlocks with the current bits and passes with this patch.

Thanks,
 Patricio

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8270085: Suspend during block transition may deadlock if lock held

Reviewers

David Holmes (@dholmes-ora - Reviewer) ⚠️ Review applies to 603c62a
Coleen Phillimore (@coleenp - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4828/head:pull/4828
$ git checkout pull/4828

Update a local copy of the PR:
$ git checkout pull/4828
$ git pull https://git.openjdk.java.net/jdk pull/4828/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4828

View PR using the GUI difftool:
$ git pr show -t 4828

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4828.diff

bridgekeeper · 2021-07-19T14:35:38Z

👋 Welcome back pchilanomate! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-07-19T14:38:12Z

@pchilano The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

pchilano · 2021-07-19T14:38:55Z

/label add hotspot-runtime

pchilano · 2021-07-19T14:39:06Z

/label remove hotspot

openjdk · 2021-07-19T14:40:19Z

@pchilano
The hotspot-runtime label was successfully added.

openjdk · 2021-07-19T14:41:48Z

@pchilano
The hotspot label was successfully removed.

mlbridge · 2021-07-19T15:57:23Z

Webrevs

dholmes-ora

Hi Patricio,

Thanks for taking over this fix from @robehn! And thanks Robbin for the initial fix.

So in simple terms we must never suspend whilst holding a VM internal lock (or any critical resource). So the ThreadBlockInVM destructor, which runs after we acquired a contended lock, must not suspend, but the ThreadBlockInVMPreProcess that we use for ObjectMonitors must suspend (and will release the lock).

I took a look at the other uses for ThreadBlockInVM. There are quite a few places where ThreadBlockInVM is used, some of which relate to threads that can't be suspended, but others where it looks very suspicious that we might have suspended before this change - so I think this is addressing a number of potential bugs.

I do wonder (perhaps separate RFE as it touches a number of files) whether ThreadBlockInVM should now be renamed ThreadBlockInVMNoSuspend to make it very clear the thread can't suspend? Though that might also mislead people as some of the logic is executed by threads that don't allow for suspension (not visible to Java code in general).

Overall I think this fix looks good, but I can't help wonder if the switch to using handshakes for suspend, is not still causing suspension checks in far more places than we used to check, and in potentially inappropriate (or at best unnecessary) places ... I'll take that thought up off line.

Thanks,
David

dholmes-ora · 2021-07-20T01:31:24Z

src/hotspot/share/runtime/interfaceSupport.inline.hpp

@@ -289,7 +291,7 @@ class ThreadBlockInVM {
  ThreadBlockInVMPreprocess<InFlightMutexRelease> _tbivmpp;
 public:
  ThreadBlockInVM(JavaThread* thread, Mutex** in_flight_mutex_addr = NULL)
-    : _ifmr(in_flight_mutex_addr), _tbivmpp(thread, _ifmr) {}
+    : _ifmr(in_flight_mutex_addr), _tbivmpp(thread, _ifmr, false) {}


Please add comment:

false /* no suspend */) { }

dholmes-ora · 2021-07-20T01:34:24Z

test/hotspot/jtreg/runtime/handshake/SuspendBlocked.java

@@ -0,0 +1,58 @@
+/*
+ * Copyright (c) 2020, 2021, Oracle and/or its affiliates. All rights reserved.


New file so should only have one copyright year.

dholmes-ora · 2021-07-20T01:38:59Z

test/hotspot/jtreg/runtime/handshake/SuspendBlocked.java

+    public static void run_loop() {
+        WhiteBox wb = WhiteBox.getWhiteBox();
+        for (int i = 0; i < 100; i++) {
+            wb.lockAndBlock(false /* suspender */);


There are two ways to document an argument like this:

What it logically means e.g. (false /* not suspender */)

What parameter it refers to - in which case, to avoid confusion with comments of type 1, I suggest we use a convention (already in use) (/* suspender= */ false)

openjdk · 2021-07-20T04:40:01Z

@pchilano This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8270085: Suspend during block transition may deadlock if lock held

Reviewed-by: dholmes, coleenp

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 12 new commits pushed to the master branch:

38694aa: 8270939: ProblemList java/lang/invoke/RicochetTest.java until JDK-8251969 is fixed
754352f: 8270340: Base64 decodeBlock intrinsic for Power64 needs cleanup
8cd0769: 8270875: Deprecate the FilterSpuriousWakeups flag so it can be removed
534f005: 8268284: javax/swing/JComponent/7154030/bug7154030.java fails with "Exception: Failed to hide opaque button"
00195b8: 8265604: Support unlinked classes in dynamic CDS archive
7f35e5b: 8270869: G1ServiceThread may not terminate
c3519c3: Merge
c130451: 8269752: C2: assert(false) failed: Bad graph detected in build_loop_late
f644365: 8269689: Update --release 17 symbol information for JDK 17 build 31
3fc761d: 8269032: Stringdedup tests are failing if the ergonomically select GC does not support it
... and 2 more: https://git.openjdk.java.net/jdk/compare/e7cdfebbeebb274b28495b469f39d5874af45e65...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

pchilano · 2021-07-20T16:19:41Z

Hi David,

Hi Patricio,

Thanks for taking over this fix from @robehn! And thanks Robbin for the initial fix.

So in simple terms we must never suspend whilst holding a VM internal lock (or any critical resource). So the ThreadBlockInVM destructor, which runs after we acquired a contended lock, must not suspend, but the ThreadBlockInVMPreProcess that we use for ObjectMonitors must suspend (and will release the lock).

I took a look at the other uses for ThreadBlockInVM. There are quite a few places where ThreadBlockInVM is used, some of which relate to threads that can't be suspended, but others where it looks very suspicious that we might have suspended before this change - so I think this is addressing a number of potential bugs.

I do wonder (perhaps separate RFE as it touches a number of files) whether ThreadBlockInVM should now be renamed ThreadBlockInVMNoSuspend to make it very clear the thread can't suspend? Though that might also mislead people as some of the logic is executed by threads that don't allow for suspension (not visible to Java code in general).

In general I think this details of what things get processed when transitioning from one state to another should not be exposed to the users unless is necessary. In this case I think most users of TBIVM don't really care about processing suspend requests or not other than TBIVM should do whatever is right, which here is just never process them. For the special cases of java monitors and jvmti raw monitors where we specifically want to honor suspend requests we have the special ThreadBlockInVMPreprocess. I actually looked into having a single TBIVM with an additional parameter to allow for suspend in those special cases. It wasn't straightforward because of the template but I'll try to revisit that.

Thanks for reviewing this David!

Patricio

dcubed-ojdk · 2021-07-20T18:36:53Z

This fix appears to be baselined on https://github.com/openjdk/jdk
instead of JDK17. The bug is targeted at JDK17 so you probably need
to reparent the PR (if possible) or recreate it (ouch).

I'm going to go ahead and start my review anyway.

pchilano · 2021-07-20T18:44:10Z

This fix appears to be baselined on https://github.com/openjdk/jdk
instead of JDK17. The bug is targeted at JDK17 so you probably need
to reparent the PR (if possible) or recreate it (ouch).

I'm going to go ahead and start my review anyway.

Ok, I thought I was supposed to integrate it in 18 first and then backport it (?).

dcubed-ojdk · 2021-07-20T18:56:33Z

It's a P2 so you should request permission to integrate it in JDK17 (follow
the RDP2 rules) and integrate there first. It will be automatically forward
ported when @JesperIRL syncs from JDK17 -> JDK18.

dcubed-ojdk · 2021-07-20T18:39:43Z

src/hotspot/share/prims/whitebox.cpp

+    // suspended in ~ThreadBlockInVM. This verifies we only suspend
+    // at the right place.
+    while (Atomic::cmpxchg(&_emulated_lock, 0, 1) != 0) {}
+    assert(_emulated_lock == 1, "Not locked");


s/"Not locked"/"Must be locked"/

dcubed-ojdk · 2021-07-20T18:40:55Z

src/hotspot/share/prims/whitebox.cpp

+  }
+  Atomic::store(&_emulated_lock, 0);
+WB_END
+
 //Some convenience methods to deal with objects from java


nit typo: please add a space before 'Some' (not your bug, but since you are here...)

dcubed-ojdk · 2021-07-20T18:41:50Z

src/hotspot/share/prims/whitebox.cpp

+
+  {CC"lockAndBlock", CC"(Z)V", (void*)&WB_LockAndBlock},
+


Not sure why you added a blank line above and below...

dcubed-ojdk · 2021-07-20T18:43:45Z

src/hotspot/share/runtime/handshake.cpp

@@ -441,6 +451,11 @@ bool HandshakeState::have_non_self_executable_operation() {
  return _queue.contains(non_self_queue_filter);
 }

+bool HandshakeState::has_none_suspend_operation() {


This name is bugging me: has_none_suspend_operation()

I think the function is supposed to return true if the queue contains
at least one non-suspend operation. If that's the case, then perhaps
it should be named:

has_a_non_suspend_operation()

dcubed-ojdk · 2021-07-20T18:49:05Z

src/hotspot/share/runtime/interfaceSupport.inline.hpp

@@ -289,7 +291,7 @@ class ThreadBlockInVM {
  ThreadBlockInVMPreprocess<InFlightMutexRelease> _tbivmpp;
 public:
  ThreadBlockInVM(JavaThread* thread, Mutex** in_flight_mutex_addr = NULL)
-    : _ifmr(in_flight_mutex_addr), _tbivmpp(thread, _ifmr) {}
+    : _ifmr(in_flight_mutex_addr), _tbivmpp(thread, _ifmr, false /* no suspend */) {}


@dholmes-ora asked in a different place for a change like this:

/* allow_suspend= */ false

I think that makes sense here too.

No I said there were two styles for doing this kind of commenting. I like this style and suggested the comment.

coleenp

Just some comments and questions. Looks good!

coleenp · 2021-07-20T20:52:55Z

src/hotspot/share/runtime/handshake.cpp

-}
+  // Handshakes cannot safely safepoint.
+  // The exception to this rule is the asynchronous suspension handshake.
+  // It by-passes the NSV by manually doing the transition.


I have no idea what this comment means in this context. Below we take out a lock with _no_safepoint_check which is essentially a NSV. You just moved this comment so I don't suggest changing it at this time.

coleenp · 2021-07-20T21:01:05Z

src/hotspot/share/runtime/handshake.cpp

+    // actual suspend since Handshake::execute() above only installed
+    // the asynchronous handshake.
+    SafepointMechanism::process_if_requested(self);
+  }


Is this an optimization? Or can the thread escape the suspend request?

coleenp · 2021-07-20T21:06:40Z

test/hotspot/jtreg/runtime/handshake/SuspendBlocked.java

+/*
+ * @test SuspendBlocked
+ * @bug 8270085
+ * @library /testlibrary /test/lib


I think this is referring to /testlibrary that I removed. Can you see if it's still needed?

pchilano · 2021-07-20T22:37:55Z

jdk17 PR: openjdk/jdk17#257

pchilano · 2021-07-20T22:42:07Z

Closing and using jdk17 PR: openjdk/jdk17#257

v1

7bf7e5b

openjdk bot added the hotspot hotspot-dev@openjdk.org label Jul 19, 2021

openjdk bot added the hotspot-runtime hotspot-runtime-dev@openjdk.org label Jul 19, 2021

openjdk bot removed the hotspot hotspot-dev@openjdk.org label Jul 19, 2021

add bug reference to SuspendBlocked.java

603c62a

pchilano marked this pull request as ready for review July 19, 2021 15:51

openjdk bot added the rfr Pull request is ready for review label Jul 19, 2021

dholmes-ora approved these changes Jul 20, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jul 20, 2021

address David's comments

374b7ee

dcubed-ojdk suggested changes Jul 20, 2021

View reviewed changes

address Dan's comments

3fe9666

coleenp approved these changes Jul 20, 2021

View reviewed changes

pchilano closed this Jul 20, 2021

		@@ -0,0 +1,58 @@
		/*
		* Copyright (c) 2020, 2021, Oracle and/or its affiliates. All rights reserved.

8270085: Suspend during block transition may deadlock if lock held #4828

8270085: Suspend during block transition may deadlock if lock held #4828

Conversation

pchilano commented Jul 19, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Jul 19, 2021

openjdk bot commented Jul 19, 2021

pchilano commented Jul 19, 2021

pchilano commented Jul 19, 2021

openjdk bot commented Jul 19, 2021

openjdk bot commented Jul 19, 2021

mlbridge bot commented Jul 19, 2021 • edited Loading

Webrevs

dholmes-ora left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Jul 20, 2021 • edited Loading

pchilano commented Jul 20, 2021

dcubed-ojdk commented Jul 20, 2021

pchilano commented Jul 20, 2021 • edited Loading

dcubed-ojdk commented Jul 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coleenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pchilano commented Jul 20, 2021

pchilano commented Jul 20, 2021

pchilano commented Jul 19, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Jul 19, 2021 •

edited

Loading

openjdk bot commented Jul 20, 2021 •

edited

Loading

pchilano commented Jul 20, 2021 •

edited

Loading