8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load #4602

lmesnik · 2021-06-26T17:48:15Z

The crash happens because nmethod might become a zombie before it is enqueued in JvmtiDeferredEventQueue or after it is dequeued from it. The crash is reproduced by serviceability/jvmti/CompiledMethodLoad/Zombie.java. However, it takes ~3K runs to hit it. I verified the fix by running this test >100K on each platform. Also, I verified that protecting in 'void JvmtiDeferredEventQueue::post(JvmtiEnv* env)' is not enough.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load

Reviewers

Serguei Spitsyn (@sspitsyn - Reviewer)
David Holmes (@dholmes-ora - Reviewer)
Coleen Phillimore (@coleenp - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4602/head:pull/4602
$ git checkout pull/4602

Update a local copy of the PR:
$ git checkout pull/4602
$ git pull https://git.openjdk.java.net/jdk pull/4602/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4602

View PR using the GUI difftool:
$ git pr show -t 4602

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4602.diff

bridgekeeper · 2021-06-26T17:48:21Z

👋 Welcome back lmesnik! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-06-26T17:50:11Z

@lmesnik The following labels will be automatically applied to this pull request:

hotspot
serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

openjdk · 2021-06-28T18:13:48Z

@lmesnik this pull request can not be integrated into master due to one or more merge conflicts. To resolve these merge conflicts and update this pull request you can run the following commands in the local repository for your personal fork:

git checkout 8245877
git fetch https://git.openjdk.java.net/jdk master
git merge FETCH_HEAD
# resolve conflicts and follow the instructions given by git merge
git commit -m "Merge master"
git push

mlbridge · 2021-06-28T18:44:24Z

Webrevs

sspitsyn

Hi Leonid,
I looks good to me.
Thank you for addressing it!
Thanks,
Serguei

openjdk · 2021-06-28T19:00:34Z

@lmesnik This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load

Reviewed-by: sspitsyn, dholmes, coleenp

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 17 new commits pushed to the master branch:

1ac8dee: 8269615: Fix for 8263640 broke Windows build
7ca753b: 8269268: JDWP: Properly fix thread lookup assert in findThread()
7a23c9c: 8260540: serviceability/jdwp/AllModulesCommandTest.java failed with "Debuggee error: 'ERROR: transport error 202: bind failed: Address already in use'"
c1e2a29: 8263640: hs_err improvement: handle class path longer than O_BUFLEN
bb42d75: 8269417: Minor clarification on NonblockingQueue utility
e238cbd: 8269530: runtime/ParallelLoad/ParallelSuperTest.java timeout
3f2c372: 8269126: Rename G1AllowPreventiveGC option to G1UsePreventiveGC
a977157: 8261579: AArch64: Support for weaker memory ordering in Atomic
ee1e202: 8268821: Split systemDictionaryShared.cpp
03d54e6: Merge
... and 7 more: https://git.openjdk.java.net/jdk/compare/d0d26f5c550e44bf22d9155e4219351104be4635...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

dholmes-ora

Hi Leonid,

I'm not clear on the details here - please see comments below.

Thanks,
David

dholmes-ora · 2021-06-29T02:23:25Z

src/hotspot/share/prims/jvmtiImpl.cpp

+  for (QueueNode* node = _queue_head; node != NULL; node = node->next()) {
+    node->event().post_compiled_method_load_event(env);
+  }


Can't you dequeue() immediately after calling post_compiled_method_load_event()?

Seems that dequeue in for-loop deletes the node which posted. It is possible to update the loop to have update dequeue in the same iteration however, I don't think it is a good idea to mix iterator/deletion in the same loop. What is the reason for this change?

Just to save iterating the queue twice. And that is what the original loop did - you just have to switch the order of processing and deleting.

dholmes-ora · 2021-06-29T02:23:58Z

src/hotspot/share/code/nmethod.cpp

@@ -1608,6 +1608,7 @@ void nmethod::post_compiled_method_load_event(JvmtiThreadState* state) {
    if (is_not_entrant() && can_convert_to_zombie()) {
      return;
    }
+    mark_as_seen_on_stack();


Not obvious what this actually does in relation to the dequeuing problem.

updated comments

I'm not 100% this is needed because we don't handshake this thread and we just checked that it can't be converted to zombie, but I don't think this hurts anything and I thought it might be needed.

lmesnik · 2021-06-29T06:12:29Z

Replied to [@dholmes-ora] comments but don't see any notifications from GitHub yet. Not clear if it is my PR/GitHub/mail issues.

dholmes-ora · 2021-06-29T07:42:17Z

I see the comment update in the PR and email, but no response to my comment about dequeuing within the for-loop.

lmesnik

replied to comments

lmesnik · 2021-06-29T07:46:39Z

src/hotspot/share/code/nmethod.cpp

@@ -1608,6 +1608,7 @@ void nmethod::post_compiled_method_load_event(JvmtiThreadState* state) {
    if (is_not_entrant() && can_convert_to_zombie()) {
      return;
    }
+    mark_as_seen_on_stack();


updated comments

lmesnik · 2021-06-29T16:36:01Z

[@dholmes-ora] I've updated the loop in the post. Seems your comments were removed because I remove the first loop.

coleenp

Looks good!

coleenp · 2021-06-29T17:57:09Z

src/hotspot/share/code/nmethod.cpp

@@ -1608,6 +1608,7 @@ void nmethod::post_compiled_method_load_event(JvmtiThreadState* state) {
    if (is_not_entrant() && can_convert_to_zombie()) {
      return;
    }
+    mark_as_seen_on_stack();


I'm not 100% this is needed because we don't handshake this thread and we just checked that it can't be converted to zombie, but I don't think this hurts anything and I thought it might be needed.

coleenp · 2021-06-29T17:58:33Z

src/hotspot/share/prims/jvmtiImpl.cpp

-     JvmtiDeferredEvent event = dequeue();
-     event.post_compiled_method_load_event(env);
+    _queue_head->event().post_compiled_method_load_event(env);
+    dequeue();


Good find! So we can zombie the nmethod after we take it off the list. Makes a lot of sense. Thank you for your perseverance in tracking this down!

dholmes-ora

Looks good!

Thanks,
David

lmesnik · 2021-06-30T02:03:59Z

/integrate

openjdk · 2021-06-30T02:05:40Z

Going to push as commit b969136.
Since your change was applied there have been 26 commits pushed to the master branch:

ee526a2: Merge
0d745ae: 8269034: AccessControlException for SunPKCS11 daemon threads
d042029: 8269529: javax/swing/reliability/HangDuringStaticInitialization.java fails in Windows debug build
401cb0a: 8269232: assert(!is_jweak(handle)) failed: wrong method for detroying jweak
b8a16e9: 8268884: C2: Compile::remove_speculative_types must iterate top-down
25f9f19: 8249646: Runtime.exec(String, String[], File) documentation contains literal {@link ...}
0d83dc2: 8268699: Shenandoah: Add test for JDK-8268127
7010dfd: 8269517: compiler/loopopts/TestPartialPeelingSinkNodes.java crashes with -XX:+VerifyGraphEdges
fc0fd96: 8269126: Rename G1AllowPreventiveGC option to G1UsePreventiveGC
1ac8dee: 8269615: Fix for 8263640 broke Windows build
... and 16 more: https://git.openjdk.java.net/jdk/compare/d0d26f5c550e44bf22d9155e4219351104be4635...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-06-30T02:06:02Z

@lmesnik Pushed as commit b969136.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

fix

d17e4ce

openjdk bot added serviceability serviceability-dev@openjdk.org hotspot hotspot-dev@openjdk.org labels Jun 26, 2021

lmesnik added 2 commits June 26, 2021 13:21

test unproblemlisted.

70159cb

ident fixed.

9cc4d00

openjdk bot added the merge-conflict Pull request has merge conflict with target branch label Jun 28, 2021

lmesnik added 2 commits June 28, 2021 11:17

Merge branch 'master' of https://github.com/openjdk/jdk into 8245877

01afc5f

fix in problemlist.txt

35d5848

openjdk bot removed the merge-conflict Pull request has merge conflict with target branch label Jun 28, 2021

lmesnik marked this pull request as ready for review June 28, 2021 18:38

openjdk bot added the rfr Pull request is ready for review label Jun 28, 2021

sspitsyn approved these changes Jun 28, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jun 28, 2021

dholmes-ora reviewed Jun 29, 2021

View reviewed changes

Added comment.

9ea2cb9

lmesnik commented Jun 29, 2021

View reviewed changes

post updated.

2e222f0

coleenp approved these changes Jun 29, 2021

View reviewed changes

dholmes-ora approved these changes Jun 29, 2021

View reviewed changes

sspitsyn approved these changes Jun 30, 2021

View reviewed changes

openjdk bot closed this Jun 30, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jun 30, 2021

lmesnik deleted the 8245877 branch February 12, 2022 21:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load #4602

8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load #4602

lmesnik commented Jun 26, 2021 •

edited by openjdk bot

Loading

bridgekeeper bot commented Jun 26, 2021

openjdk bot commented Jun 26, 2021

openjdk bot commented Jun 28, 2021

mlbridge bot commented Jun 28, 2021 •

edited

Loading

sspitsyn left a comment

openjdk bot commented Jun 28, 2021 •

edited

Loading

dholmes-ora left a comment

dholmes-ora Jun 29, 2021

lmesnik Jun 29, 2021

dholmes-ora Jun 29, 2021

dholmes-ora Jun 29, 2021

lmesnik Jun 29, 2021

coleenp Jun 29, 2021

lmesnik commented Jun 29, 2021 •

edited

Loading

dholmes-ora commented Jun 29, 2021

lmesnik left a comment

lmesnik Jun 29, 2021

lmesnik commented Jun 29, 2021

coleenp left a comment

coleenp Jun 29, 2021

coleenp Jun 29, 2021

dholmes-ora left a comment

lmesnik commented Jun 30, 2021

openjdk bot commented Jun 30, 2021

openjdk bot commented Jun 30, 2021

8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load #4602

8245877: assert(_value != __null) failed: resolving NULL _value in JvmtiExport::post_compiled_method_load #4602

Conversation

lmesnik commented Jun 26, 2021 • edited by openjdk bot Loading

Progress

Issue

Reviewers

Reviewing

bridgekeeper bot commented Jun 26, 2021

openjdk bot commented Jun 26, 2021

openjdk bot commented Jun 28, 2021

mlbridge bot commented Jun 28, 2021 • edited Loading

Webrevs

sspitsyn left a comment

Choose a reason for hiding this comment

openjdk bot commented Jun 28, 2021 • edited Loading

dholmes-ora left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmesnik commented Jun 29, 2021 • edited Loading

dholmes-ora commented Jun 29, 2021

lmesnik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmesnik commented Jun 29, 2021

coleenp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dholmes-ora left a comment

Choose a reason for hiding this comment

lmesnik commented Jun 30, 2021

openjdk bot commented Jun 30, 2021

openjdk bot commented Jun 30, 2021

lmesnik commented Jun 26, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Jun 28, 2021 •

edited

Loading

openjdk bot commented Jun 28, 2021 •

edited

Loading

lmesnik commented Jun 29, 2021 •

edited

Loading