8266593: vmTestbase/nsk/jvmti/PopFrame/popframe011 fails with "assert(java_thread == _state->get_thread()) failed: Must be" by sspitsyn · Pull Request #6440 · openjdk/jdk

sspitsyn · 2021-11-17T22:21:33Z

The test fails when the target JavaThread has is_exiting() status. In such a case the JvmtiExport::cleanup_thread(this) has already made a clean up of its jvmtiThreadState, so the JavaThread address returned by _state->get_thread() is 0xbabababababababa.
The fix is to add a check for is_exiting() status into handshake closure do_thread() early.
There following handshake closures are fixed by this update:

UpdateForPopTopFrameClosure
SetForceEarlyReturn
SetFramePopClosure

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8266593: vmTestbase/nsk/jvmti/PopFrame/popframe011 fails with "assert(java_thread == _state->get_thread()) failed: Must be"

Reviewers

Martin Doerr (@TheRealMDoerr - Reviewer)
Leonid Mesnik (@lmesnik - Reviewer)
Daniel D. Daugherty (@dcubed-ojdk - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/6440/head:pull/6440
$ git checkout pull/6440

Update a local copy of the PR:
$ git checkout pull/6440
$ git pull https://git.openjdk.java.net/jdk pull/6440/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 6440

View PR using the GUI difftool:
$ git pr show -t 6440

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/6440.diff

bridgekeeper · 2021-11-17T22:23:01Z

👋 Welcome back sspitsyn! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-11-17T22:25:49Z

@sspitsyn The following labels will be automatically applied to this pull request:

hotspot
serviceability

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-11-17T22:37:01Z

Webrevs

TheRealMDoerr

LGTM. Thanks for fixing it!

openjdk · 2021-11-17T23:01:05Z

@sspitsyn This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8266593: vmTestbase/nsk/jvmti/PopFrame/popframe011 fails with "assert(java_thread == _state->get_thread()) failed: Must be"

Reviewed-by: mdoerr, lmesnik, dcubed

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 46 new commits pushed to the master branch:

0a9e76c: 8277485: Zero: Fix fast{i,f}access_0 bytecodes handling
1c215f3: 8272773: Configurable card table card size
1d7cef3: 8276662: Scalability bottleneck in SymbolTable::lookup_common()
c79a485: 8277494: [BACKOUT] JDK-8276150 Quarantined jpackage apps are labeled as "damaged"
2ab43ec: 8273544: Increase test coverage for snippets
2d4af22: 8277370: configure script cannot distinguish WSL version
a3406a1: 8277092: TestMetaspaceAllocationMT2.java#ndebug-default fails with "RuntimeException: Committed seems high: NNNN expected at most MMMM"
e47cc81: 8275386: Change nested classes in jdk.jlink to static nested classes
f609b8f: 8274946: Cleanup unnecessary calls to Throwable.initCause() in java.rmi
36b59f3: 8274333: Redundant null comparison after Pattern.split
... and 36 more: https://git.openjdk.java.net/jdk/compare/91607436b79126ccb999deaf38a98209dbfe6ec1...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

dholmes-ora

Hi Serguei, (sorry for initially addressing to Leonid)

Something seems amiss to me.

First the checks for java_thread->threadObj() == NULL should not be necessary as the threadObj can never be NULL once it has been started and a non-started thread should not be possible by the time you reach the code doing the checks. Even if we nulled out threadObj for a terminated thread the is_exiting check would already handle that case.

Second, if the target thread is exiting then surely the suspension check should return false and so we would already give a JVMTI_ERROR_THREAD_NOT_SUSPENDED error?

Thanks,
David

sspitsyn · 2021-11-18T06:49:17Z

Martin and Leonid, thank you for quick review!

sspitsyn · 2021-11-18T06:55:49Z

Hi David,

Thank you for reviewing this!
I was also thinking about getting rid of the check java_thread->threadObj() == NULL.
Then I've decided it is safe to keep it as it was in the original UpdateForPopTopFrameClosure implementation (but later in the code). I will remove it and retest the fix.

Second, if the target thread is exiting then surely the suspension check should return
false and so we would already give a JVMTI_ERROR_THREAD_NOT_SUSPENDED error?

The assert
assert(java_thread == _state->get_thread(), "Must be");
is fired one line before the JVMTI_ERROR_THREAD_NOT_SUSPENDED code is returned.

dholmes-ora · 2021-11-18T07:35:06Z

Wouldn't it suffice to just move the assert then?

sspitsyn · 2021-11-18T08:26:49Z

It does not look right to check other conditions if the JavaThread is exiting.
So, I think, the java_thread->is_exiting() has to be checked first.
Please, let me know if I miss anything.

…= NULL

dholmes-ora · 2021-11-18T09:56:05Z

IIUC these cases all require that the target is suspended else it is an error. If the target is_exiting then it is not suspended and therefore there is an error. The suspension check should already handle an exiting thread and so there is no need to explicitly add an is_exiting check.

sspitsyn · 2021-11-18T10:31:12Z

It is not correct.
At least, there is this case:

/* non suspended and exiting thread */
    case 6:
        set_watch_ev(1); /* watch JVMTI events */
        popframe_err = (jvmti->PopFrame(frameThr)); /* explode the bomb */
        set_watch_ev(0); /* ignore again JVMTI events */
        if (popframe_err != JVMTI_ERROR_THREAD_NOT_SUSPENDED &&
            popframe_err != JVMTI_ERROR_THREAD_NOT_ALIVE) {
            printf("TEST FAILED: the function PopFrame() returned the error %d: %s\n",
                popframe_err, TranslateError(popframe_err));
            printf("\tBut it should return the error JVMTI_ERROR_THREAD_NOT_SUSPENDED or JVMTI_ERROR_THREAD_NOT_ALIVE.\n");
            return STATUS_FAILED;
        }
        break;
    }

In other cases, the test constructs cases so that the tested thread is alive when expected.
The test was easily failing before in 10th of runs but now it does not fail in 100 runs.
I'll try to run this test 1000 times on all platforms.

sspitsyn · 2021-11-18T10:35:17Z

Also, if the target thread is exiting then the PopFrame should return error code JVMTI_ERROR_THREAD_NOT_ALIVE, but not JVMTI_ERROR_THREAD_NOT_SUSPENDED. It does not matter what this test is expecting.

dholmes-ora · 2021-11-18T12:53:14Z

It seems somewhat subjective whether a thread that is exiting and thus still on its way to becoming "not alive" needs to report "not alive" versus "not suspended". As there appears to be no synchronization with the target in this case what stops it from transitioning to "is_exiting" the moment after the "is_exiting" check returns false, but before you hit the assertion?

dholmes-ora · 2021-11-18T13:08:35Z

Ignore that last question - the target is in a handshake so can't change state.

dcubed-ojdk · 2021-11-18T17:15:06Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+
+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  assert(java_thread == _state->get_thread(), "Must be");


This assert() is the site of the original test failure. I haven't yet
looked at the locations of the other changes.

The is_exiting() check is made under the protection of the
JvmtiThreadState_lock so an unsuspended target thread that is
exiting cannot reach the point where the _state is updated to
clear the JavaThread* so we can't fail the assert() if the
is_exiting() check has returned false.

Dan,
Thank you for reviewing this!
I'm not sure, I correctly understand you here.
Are you saying that you agree with this change?
In fact, the thread state can not be changed (and the assert fired) after the is_exiting() check is made even without JvmtiThreadState_lock protection because it is inside of a handshake execution.

I agree with the is_exiting() check addition.

I forgot that we're executing a Handshake doit() function. So we have a couple
of reasons why an unsuspended target thread can't change from !is_exiting()
to is_exiting() while we are in this function.

Again this introduces a more precise state check but also changes the behaviour by now reporting NOT_ALIVE instead of NOT_SUSPENDED. The assertion failure can be fixed by simply moving the assertion to after the suspension check.

dcubed-ojdk

I don't see a reason for the change in SetForceEarlyReturn::doit(),
but I'm okay with the other changes.

dcubed-ojdk · 2021-11-18T17:18:23Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  if (!self) {
    if (!java_thread->is_suspended()) {
      _result = JVMTI_ERROR_THREAD_NOT_SUSPENDED;


I don't see an obvious reason for this is_exiting() check.

Okay. I see similar check in the force_early_return() function:

if (state == NULL) { return JVMTI_ERROR_THREAD_NOT_ALIVE; }

Would it better to replace it with this check instead? :

if (java_thread->is_exiting()) { return JVMTI_ERROR_THREAD_NOT_ALIVE; }

Removing this check and keep the one inside the handshake would be even better.

I would also add this line for symmetry with two other cases:

+ MutexLocker mu(JvmtiThreadState_lock); SetForceEarlyReturn op(state, value, tos);

My point is that I don't see why you added the is_exiting() check
since I don't see a race in that function, i.e., there's no assert() in
this function that you need to protect.

As for adding the MutexLocker mu(JvmtiThreadState_lock), you'll
have to analyze and justify why you would need to add that lock grab
independent of this fix. I'm not seeing a bug there, but I haven't looked
very closely.

The is_exiting check changes the behaviour from reporting JVMTI_ERROR_THREAD_NOT_SUSPENDED to JVMTI_ERROR_THREAD_NOT_ALIVE. Arguably it is a more precise answer, but it is somewhat splitting hairs. To me it might be clearer to the developer what their logic error is if they get NOT_SUSPENDED rather than NOT_ALIVE. Either way this change is not needed to fix any known bug and the change is behaviour seems questionable.

dcubed-ojdk · 2021-11-18T17:21:54Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  assert(_state->get_thread() == java_thread, "Must be");


The assert() on L1625 is subject to the same race as the original site.
This is_exiting() check is made under the protection of the
JvmtiThreadState_lock so it is sufficient to protect that assert().

Okay, thanks!

Same comment as above.

dcubed-ojdk · 2021-11-19T17:04:42Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  if (!self) {
    if (!java_thread->is_suspended()) {
      _result = JVMTI_ERROR_THREAD_NOT_SUSPENDED;


My point is that I don't see why you added the is_exiting() check
since I don't see a race in that function, i.e., there's no assert() in
this function that you need to protect.

As for adding the MutexLocker mu(JvmtiThreadState_lock), you'll
have to analyze and justify why you would need to add that lock grab
independent of this fix. I'm not seeing a bug there, but I haven't looked
very closely.

dcubed-ojdk · 2021-11-19T17:11:16Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+
+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  assert(java_thread == _state->get_thread(), "Must be");


I agree with the is_exiting() check addition.

I forgot that we're executing a Handshake doit() function. So we have a couple
of reasons why an unsuspended target thread can't change from !is_exiting()
to is_exiting() while we are in this function.

sspitsyn · 2021-11-19T18:21:32Z

Dan, thank you for review!

sspitsyn · 2021-11-19T18:47:44Z

David,
Thank you for your questions.
I'm not sure if all of them are resolved though. :)
Please, let me know if it is the case.

TheRealMDoerr

Still good. Thumbs up from my side.

dholmes-ora

Hi Serguei,

I still feel the bug here can be fixed simply by moving assertions, rather than by introducing a change in behaviour as to what error code would be returned.

But I'll leave to serviceability folk to decide.

Thanks,
David

dholmes-ora · 2021-11-22T02:00:05Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  if (!self) {
    if (!java_thread->is_suspended()) {
      _result = JVMTI_ERROR_THREAD_NOT_SUSPENDED;


The is_exiting check changes the behaviour from reporting JVMTI_ERROR_THREAD_NOT_SUSPENDED to JVMTI_ERROR_THREAD_NOT_ALIVE. Arguably it is a more precise answer, but it is somewhat splitting hairs. To me it might be clearer to the developer what their logic error is if they get NOT_SUSPENDED rather than NOT_ALIVE. Either way this change is not needed to fix any known bug and the change is behaviour seems questionable.

dholmes-ora · 2021-11-22T02:02:19Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+
+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  assert(java_thread == _state->get_thread(), "Must be");


Again this introduces a more precise state check but also changes the behaviour by now reporting NOT_ALIVE instead of NOT_SUSPENDED. The assertion failure can be fixed by simply moving the assertion to after the suspension check.

dholmes-ora · 2021-11-22T02:04:01Z

src/hotspot/share/prims/jvmtiEnvBase.cpp

+  if (java_thread->is_exiting()) {
+    return; /* JVMTI_ERROR_THREAD_NOT_ALIVE (default) */
+  }
  assert(_state->get_thread() == java_thread, "Must be");


Same comment as above.

sspitsyn · 2021-11-22T09:21:51Z

Hi David,
Thank you for looking at this and your comments.
Exiting thread should not be in suspended state.
If PopFrame reports THREAD_NOT_SUSPENDED instead of THREAD_NOT_ALIVE then it will never report THREAD_NOT_ALIVE as the spec says.
So, I'm pretty sure that the THREAD_NOT_ALIVE error code should normally take priority.
It is why I prefer current fix over moving the assert.
But I kind of understand your concern. Thank you for sharing it!
Thanks,
Serguei

sspitsyn · 2021-11-22T10:46:54Z

/integrate

openjdk · 2021-11-22T10:47:57Z

Going to push as commit 32839ba.
Since your change was applied there have been 50 commits pushed to the master branch:

8051041: 8277534: Remove unused ReferenceProcessor::has_discovered_references
3f847fe: 8277385: Zero: Enable CompactStrings support
ca31ed5: 8275448: [REDO] AArch64: Implement string_compare intrinsic in SVE
4ff4301: 8224922: Access JavaFileObject from Element(s)
0a9e76c: 8277485: Zero: Fix fast{i,f}access_0 bytecodes handling
1c215f3: 8272773: Configurable card table card size
1d7cef3: 8276662: Scalability bottleneck in SymbolTable::lookup_common()
c79a485: 8277494: [BACKOUT] JDK-8276150 Quarantined jpackage apps are labeled as "damaged"
2ab43ec: 8273544: Increase test coverage for snippets
2d4af22: 8277370: configure script cannot distinguish WSL version
... and 40 more: https://git.openjdk.java.net/jdk/compare/91607436b79126ccb999deaf38a98209dbfe6ec1...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-11-22T10:48:14Z

@sspitsyn Pushed as commit 32839ba.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

fix 8266593: vmTestbase/nsk/jvmti/PopFrame/popframe011 fails with assert

abd69aa

openjdk bot added serviceability serviceability-dev@openjdk.org hotspot hotspot-dev@openjdk.org labels Nov 17, 2021

remove vmTestbase/nsk/jvmti/PopFrame/popframe011 from ProblemList.txt

64f2294

openjdk bot added the rfr Pull request is ready for review label Nov 17, 2021

TheRealMDoerr approved these changes Nov 17, 2021

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Nov 17, 2021

lmesnik approved these changes Nov 18, 2021

View reviewed changes

dholmes-ora reviewed Nov 18, 2021

View reviewed changes

Merge

60e784e

get rid of the checks in jvmti handshakes: java_thread->threadObj() =…

435ab51

…= NULL

dcubed-ojdk reviewed Nov 18, 2021

View reviewed changes

dcubed-ojdk suggested changes Nov 18, 2021

View reviewed changes

dcubed-ojdk approved these changes Nov 19, 2021

View reviewed changes

TheRealMDoerr approved these changes Nov 19, 2021

View reviewed changes

lmesnik approved these changes Nov 21, 2021

View reviewed changes

dholmes-ora reviewed Nov 22, 2021

View reviewed changes

openjdk bot closed this Nov 22, 2021

openjdk bot added integrated Pull request has been integrated and removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Nov 22, 2021

sspitsyn deleted the br3 branch November 22, 2021 10:52

Conversation

sspitsyn commented Nov 17, 2021 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Uh oh!

bridgekeeper bot commented Nov 17, 2021

Uh oh!

openjdk bot commented Nov 17, 2021

Uh oh!

mlbridge bot commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

TheRealMDoerr left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented Nov 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sspitsyn commented Nov 18, 2021

Uh oh!

sspitsyn commented Nov 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora commented Nov 18, 2021

Uh oh!

sspitsyn commented Nov 18, 2021

Uh oh!

dholmes-ora commented Nov 18, 2021

Uh oh!

sspitsyn commented Nov 18, 2021

Uh oh!

sspitsyn commented Nov 18, 2021

Uh oh!

dholmes-ora commented Nov 18, 2021

Uh oh!

dholmes-ora commented Nov 18, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dcubed-ojdk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sspitsyn Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sspitsyn commented Nov 19, 2021

Uh oh!

sspitsyn commented Nov 19, 2021

Uh oh!

sspitsyn commented Nov 17, 2021 •

edited by openjdk bot

Loading

mlbridge bot commented Nov 17, 2021 •

edited

Loading

openjdk bot commented Nov 17, 2021 •

edited

Loading

dholmes-ora left a comment •

edited

Loading

sspitsyn commented Nov 18, 2021 •

edited

Loading

sspitsyn Nov 19, 2021 •

edited

Loading

sspitsyn commented Nov 22, 2021 •

edited

Loading