Fix to handle suspend/resume of virtual/carrier threads #16943

dipak-bagadiya · 2023-03-17T08:54:50Z

JVMTI treats a virtual thread and its carrier thread as two separate threads.
If JVMTI suspends a virtual thread, then its carrier thread
should not be suspended and vice-versa. Similarly, if JVMTI resumes
a virtual thread, then its carrier thread should not be resumed and
vice-versa.
In OpenJ9, currently, a mounted virtual thread and its carrier thread
shared the same J9VMThread. The thread state is derived from the
J9VMThread. Due to the sharing of the J9VMThread in the mounted
state, if a mounted virtual thread is suspended, then the JVMTI
functions will also reflect that its carrier thread is suspended.

Currently, "isSuspendedByJVMTI" is the hidden field that holds
the suspend status for the virtual thread only, now it made
available for carrier thread too, i.e., the scope has changed
from "java/lang/VirtualThread" to "java/lang/Thread" threads
Also, it is renamed to "isSuspendedInternal" as this hidden field is
not just specific to JVMTI.

The following functions has changed to set/reset and verify
"isSuspendedInternal" status and halt flag for threads: -

1) SUSPEND THREAD:-
suspendhelper.cpp:suspendThread:
jvmtiThread.c:jvmtiSuspendResumeCallBack
thread.cpp:Java_java_lang_Thread_suspendImpl
->In case of suspend the halt flag
(J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND) is set if the thread is
mounted, i.e., threadObject == targetThread->threadObject in
J9VMThread's publicFlags also the hidden field "isSuspendedInternal"
set to a non-zero value for the thread instance in all cases
regardless of thread being mounted or unmounted.
->If any of the public flags are already set, then the relevant
failure message is assigned.

2) RESUME THREAD:-
jvmtiThread.c:resumeThread:
jvmtiThread.c:jvmtiSuspendResumeCallBack
thread.cpp:Java_java_lang_Thread_resumeImpl
->In case of resume the halt flag
(J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND) is cleared if the thread
is mounted, i.e., threadObject == targetThread->threadObject in
J9VMThread's publicFlags also the hidden field "isSuspendedInternal"
set to a zero value for the thread instance in all cases regardless
of thread being mounted or unmounted.
->If any of the public flags are already set, then the relevant
failure message is assigned.

3) GETSTATE:-
jvmtiHelpers.c:getThreadState
jvmtiHelpers.c:getVirtualThreadState
->The getThreadState function is changed in such a way that will
verify the "isSuspendedInternal" filed to check the thread is
suspend or not.

The basic concept behind the new changes:-
[mounted + unmounted] set the "isSuspendedInternal" field
[mounted] the halt flag is only modified if the thread object is
stored in targetThread->threadObject
[unmounted] Delay setting the halt flag until the thread mount.

Related: #16689
Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

babsingh · 2023-03-24T17:03:53Z

@dipak-bagadiya After verifying that sanity.openjdk and extended.openjdk pass, please change PR's state to "Ready for review".

dipak-bagadiya · 2023-03-28T18:20:31Z

@babsingh I made some changes to the code and ran the build. I'll check the results for sanity. openjdk and extended openjdk, once passed, will modify PR.
Requesting you to Please review the code section.

babsingh

Two key points:

Code changes should not break older JDK versions; and
Avoid redundant code via sharing.

runtime/jcl/common/thread.cpp

runtime/jvmti/jvmtiThread.c

runtime/jvmti/suspendhelper.cpp

babsingh · 2023-03-30T15:27:43Z

Also, please review the commit message in MS Word to fix typos, sentence structure and grammar issues.

babsingh · 2023-04-05T05:11:31Z

@dipak-bagadiya The revised suspend, resume and get thread state logic is posted in #16689 (comment).

babsingh

@dipak-bagadiya

Please review the logic thoroughly for functional correctness.
Some of the previous feedback was unaddressed.
The commit message and PR description need to reflect the new design.

runtime/j9vm/javanextvmi.cpp

runtime/jcl/common/thread.cpp

runtime/jvmti/jvmtiThread.c

runtime/jvmti/suspendhelper.cpp

babsingh

There are still issues in the logic which will cause functional failures.
Typo in the first word of the PR description and commit message: VMTI -> JVMTI.

runtime/j9vm/javanextvmi.cpp

runtime/jcl/common/thread.cpp

runtime/jvmti/jvmtiHelpers.c

runtime/jvmti/jvmtiThread.c

runtime/jvmti/suspendhelper.cpp

babsingh

Do the JVMTI tests in extended.openjdk pass for JDK19?
Also, test compilation with JDK17.

runtime/jcl/common/thread.cpp

runtime/jvmti/jvmtiHelpers.c

runtime/jvmti/jvmtiThread.c

runtime/jvmti/suspendhelper.cpp

babsingh · 2023-04-25T15:58:54Z

jenkins build sanity,extended.openjdk zlinux,win jdk19,jdk20

babsingh · 2023-04-25T15:59:44Z

jenkins test sanity,extended.openjdk zlinux,win jdk19,jdk20

babsingh · 2023-04-25T16:00:08Z

jenkins test sanity win jdk8

dipak-bagadiya · 2023-04-27T16:20:54Z

Failures from the below issues are seen in the PR builds:

Further investigation is required for the below failure. The below failure happens in JDK19 and not in JDK20.
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal/101/tapResults/

18:05:59.728 0x1d2fe00    j9vm.228    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK19_s390x_linux_Personal/openj9/runtime/vm/VMAccess.cpp:355: (0 == ((vmThread)->publicFlags & J9_PUBLIC_FLAGS_VM_ACCESS))
JVMDUMP039I Processing dump event "traceassert", detail "" at 2023/04/25 18:05:59 - please wait.
JVMDUMP032I JVM requested System dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/core.20230425.180559.731559.0001.dmp' in response to an event
JVMDUMP010I System dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/core.20230425.180559.731559.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/javacore.20230425.180559.731559.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/javacore.20230425.180559.731559.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/Snap.20230425.180559.731559.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/jenkins/workspace/Test_openjdk19_j9_sanity.functional_s390x_linux_Personal_testList_0/aqa-tests/TKG/output_16824422641535/JCL_Test_none_SCC_0/Snap.20230425.180559.731559.0003.trc
JVMDUMP013I Processed dump event "traceassert", detail "".

Below are failures that look like the known ones, but I did not find any open issues for the same reason, and they seem unrelated to the changes in this PR.

sun/security/mscapi/NonAsciiAlias.java.NonAsciiAlias
com/sun/jndi/dns/ConfigTests/Timeout.java.Timeout

JVMTI treats a virtual thread and its carrier thread as two separate threads. If JVMTI suspends a virtual thread, then its carrier thread should not be suspended and vice-versa. Similarly, if JVMTI resumes a virtual thread, then its carrier thread should not be resumed and vice-versa. In OpenJ9, currently, a mounted virtual thread and its carrier thread shared the same J9VMThread. The thread state is derived from the J9VMThread. Due to the sharing of the J9VMThread in the mounted state,if a mounted virtual thread is suspended, then the JVMTI functions will also reflect that its carrier thread is suspended. Currently, "isSuspendedByJVMTI" is the hidden field that holds the suspend status for the virtual thread only, now it made available for carrier thread too, i.e. the scope has changed from "java/lang/VirtualThread" to "java/lang/Thread" threads Also, it is renamed to "isSuspendedInternal" as this hidden field is not just specific to JVMTI. The following functions has changed to set/reset and verify "isSuspendedInternal" status for threads:- 1) SUSPEND THREAD:- suspendhelper.cpp:suspendThread: jvmtiThread.c:jvmtiSuspendResumeCallBack thread.cpp:Java_java_lang_Thread_suspendImpl ->In case of suspend the halt flag (J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND) is set if the thread is mounted, i.e. threadObject == targetThread->threadObject in J9VMThread's publicFlags also the hidden field "isSuspendedInternal" set to a non-zero value for the thread instance in all cases regardless of thread being mounted or unmounted. ->If any of the public flags are already set, then the relevant failure message is assigned. 2) RESUME THREAD:- jvmtiThread.c:resumeThread: jvmtiThread.c:jvmtiSuspendResumeCallBack thread.cpp:Java_java_lang_Thread_resumeImpl ->In case of resume the halt flag (J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND) is cleared if the thread is mounted, i.e. threadObject == targetThread->threadObject in J9VMThread's publicFlags also the hidden field "isSuspendedInternal" set to a zero value for the thread instance in all cases regardless of thread being mounted or unmounted. ->If any of the public flags are already set, then the relevant failure message is assigned. 3) GETSTATE:- jvmtiHelpers.c:getThreadState jvmtiHelpers.c:getVirtualThreadState ->The getThreadState function is changed in such a way that will verify the "isSuspendedInternal" filed to check the thread is suspend or not. The basic concept behind the new changes: [mounted + unmounted] set the "isSuspendedInternal" field [mounted] the halt flag is only modified if the thread object is stored in targetThread->threadObject [unmounted] Delay setting the halt flag until the thread mount. Related: eclipse-openj9#16689 Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

babsingh · 2023-05-03T02:23:19Z

jenkins test sanity alinux64 jdk20

babsingh · 2023-05-03T02:23:47Z

jenkins compile win jdk8

babsingh

Overall, the code changes LGTM. re #16943 (comment): In the earlier PR builds, mostly known failures are seen. The VM-Access assertion is seen on JDK19 and not on JDK20; this is because JDK19 is out-of-service and has become stale.

Test had been fixed by eclipse-openj9/openj9#16943 Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

Test has been fixed by eclipse-openj9/openj9#16943. Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

Test has been fixed by eclipse-openj9/openj9#16943. Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

babsingh · 2023-05-04T03:13:49Z

runtime/jcl/common/thread.cpp

-					while (J9_ARE_ALL_BITS_SET(targetThread->publicFlags, J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND | J9_PUBLIC_FLAGS_VM_ACCESS)) {
-						omrthread_monitor_wait(targetThread->publicFlagsMutex);
+					vmFuncs->setHaltFlag(targetThread, J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND);
+				} else {


During the review, I was looking at the code with white-space hidden. So, I missed that the else block is indented incorrectly. It needs to include the monitor exit and goto calls.

@dipak-bagadiya This might be the potential fix for the assertion seen in #17334.

fyi @pshipton, we will thoroughly test the code before merging again.

Revised code:

... #if JAVA_SPEC_VERSION >= 19 J9OBJECT_U32_STORE(currentThread, receiverObject, vm->isSuspendedInternalOffset, 1); if (receiverObject == targetThread->threadObject) #endif /* JAVA_SPEC_VERSION >= 19 */ { if (currentThread == targetThread) { vmFuncs->setHaltFlag(targetThread, J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND); } else { vmFuncs->internalExitVMToJNI(currentThread); omrthread_monitor_enter(targetThread->publicFlagsMutex); VM_VMAccess::setHaltFlagForVMAccessRelease(targetThread, J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND); if (VM_VMAccess::mustWaitForVMAccessRelease(targetThread)) { while (J9_ARE_ALL_BITS_SET(targetThread->publicFlags, J9_PUBLIC_FLAGS_HALT_THREAD_JAVA_SUSPEND | J9_PUBLIC_FLAGS_VM_ACCESS)) { omrthread_monitor_wait(targetThread->publicFlagsMutex); } } omrthread_monitor_exit(targetThread->publicFlagsMutex); goto vmAccessReleased; } } ...

dipak-bagadiya marked this pull request as draft March 17, 2023 08:55

babsingh self-assigned this Mar 17, 2023

dipak-bagadiya force-pushed the suspend_resume_status branch 6 times, most recently from 2362f58 to ca18ba7 Compare March 24, 2023 06:39

dipak-bagadiya force-pushed the suspend_resume_status branch 3 times, most recently from cb0afcb to cdb97e6 Compare March 28, 2023 18:08

dipak-bagadiya force-pushed the suspend_resume_status branch from cdb97e6 to 63b7237 Compare March 30, 2023 14:25

babsingh requested changes Mar 30, 2023

View reviewed changes

dipak-bagadiya force-pushed the suspend_resume_status branch from 63b7237 to 22d11a7 Compare April 3, 2023 13:17

dipak-bagadiya force-pushed the suspend_resume_status branch 2 times, most recently from 1a19b45 to 57ec6f7 Compare April 6, 2023 13:59

babsingh mentioned this pull request Apr 6, 2023

JDK19 serviceability_jvmti_j9_0_FAILED serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java Fatal error: Carrier thread is NOT expected to be suspended #17123

Closed

babsingh requested changes Apr 7, 2023

View reviewed changes

dipak-bagadiya force-pushed the suspend_resume_status branch from 57ec6f7 to c204919 Compare April 10, 2023 12:00

babsingh requested changes Apr 12, 2023

View reviewed changes

dipak-bagadiya force-pushed the suspend_resume_status branch from c204919 to b74a222 Compare April 13, 2023 04:41

babsingh requested changes Apr 14, 2023

View reviewed changes

runtime/jcl/common/thread.cpp Outdated Show resolved Hide resolved

runtime/jvmti/jvmtiHelpers.c Outdated Show resolved Hide resolved

runtime/jvmti/jvmtiThread.c Outdated Show resolved Hide resolved

runtime/jvmti/suspendhelper.cpp Show resolved Hide resolved

dipak-bagadiya force-pushed the suspend_resume_status branch 4 times, most recently from 7a5b7b7 to 36b28c2 Compare April 20, 2023 16:17

dipak-bagadiya force-pushed the suspend_resume_status branch 3 times, most recently from 5ab4901 to e8e22fc Compare April 24, 2023 17:54

dipak-bagadiya force-pushed the suspend_resume_status branch from e8e22fc to 5a37cb1 Compare May 2, 2023 07:21

babsingh marked this pull request as ready for review May 2, 2023 22:27

babsingh approved these changes May 3, 2023

View reviewed changes

babsingh merged commit aa5727e into eclipse-openj9:master May 3, 2023
6 checks passed

babsingh mentioned this pull request May 3, 2023

JVMTI WaitNotifySuspendedVThreadTest fails with JVMTI_ERROR_THREAD_SUSPENDED #16689

Closed

dipak-bagadiya added a commit to dipak-bagadiya/aqa-tests that referenced this pull request May 3, 2023

Re-enable JVMTI test WaitNotifySuspendedVThreadTest

f590703

Test had been fixed by eclipse-openj9/openj9#16943 Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

dipak-bagadiya mentioned this pull request May 3, 2023

Re-enable JVMTI test WaitNotifySuspendedVThreadTest adoptium/aqa-tests#4558

Merged

dipak-bagadiya added a commit to dipak-bagadiya/aqa-tests that referenced this pull request May 3, 2023

Re-enable JVMTI test WaitNotifySuspendedVThreadTest

6987bbc

Test has been fixed by eclipse-openj9/openj9#16943. Signed-off-by: Dipak Bagadiya dipak.bagadiya@ibm.com

pshipton mentioned this pull request May 4, 2023

zLinux ASSERTION FAILED VMAccess.cpp:355: (0 == ((vmThread)->publicFlags & J9_PUBLIC_FLAGS_VM_ACCESS)) #17334

Closed

babsingh mentioned this pull request May 4, 2023

Revert "Fix to handle suspend/resume of virtual/carrier threads" #17335

Merged

pshipton added the comp:vm label May 4, 2023

babsingh reviewed May 4, 2023

View reviewed changes

pshipton mentioned this pull request May 4, 2023

OpenJDK jdk8 java/util/stream failures #17337

Closed

0xdaryl mentioned this pull request May 4, 2023

Stop failing compilations due to final Object invokeinterface #17317

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix to handle suspend/resume of virtual/carrier threads #16943

Fix to handle suspend/resume of virtual/carrier threads #16943

dipak-bagadiya commented Mar 17, 2023 •

edited

babsingh commented Mar 24, 2023

dipak-bagadiya commented Mar 28, 2023

babsingh left a comment

babsingh commented Mar 30, 2023 •

edited

babsingh commented Apr 5, 2023

babsingh left a comment •

edited

babsingh left a comment •

edited

babsingh left a comment •

edited

babsingh commented Apr 25, 2023

babsingh commented Apr 25, 2023

babsingh commented Apr 25, 2023

dipak-bagadiya commented Apr 27, 2023 •

edited by babsingh

babsingh commented May 3, 2023

babsingh commented May 3, 2023

babsingh left a comment •

edited

babsingh May 4, 2023 •

edited

Fix to handle suspend/resume of virtual/carrier threads #16943

Fix to handle suspend/resume of virtual/carrier threads #16943

Conversation

dipak-bagadiya commented Mar 17, 2023 • edited

babsingh commented Mar 24, 2023

dipak-bagadiya commented Mar 28, 2023

babsingh left a comment

Choose a reason for hiding this comment

babsingh commented Mar 30, 2023 • edited

babsingh commented Apr 5, 2023

babsingh left a comment • edited

Choose a reason for hiding this comment

babsingh left a comment • edited

Choose a reason for hiding this comment

babsingh left a comment • edited

Choose a reason for hiding this comment

babsingh commented Apr 25, 2023

babsingh commented Apr 25, 2023

babsingh commented Apr 25, 2023

dipak-bagadiya commented Apr 27, 2023 • edited by babsingh

babsingh commented May 3, 2023

babsingh commented May 3, 2023

babsingh left a comment • edited

Choose a reason for hiding this comment

babsingh May 4, 2023 • edited

Choose a reason for hiding this comment

dipak-bagadiya commented Mar 17, 2023 •

edited

babsingh commented Mar 30, 2023 •

edited

babsingh left a comment •

edited

babsingh left a comment •

edited

babsingh left a comment •

edited

dipak-bagadiya commented Apr 27, 2023 •

edited by babsingh

babsingh left a comment •

edited

babsingh May 4, 2023 •

edited