Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenJDK serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest #18738

Open
pshipton opened this issue Jan 11, 2024 · 8 comments
Open

OpenJDK serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest #18738

pshipton opened this issue Jan 11, 2024 · 8 comments
Labels
comp:jvmti comp:vm project:loom Used to track Project Loom related work test failure
Milestone
Release 0.48 (Jav...

Comments

@pshipton
Copy link
Member

From the grinder run for #18730
Internal test on ubu22lert-1
serviceability_jvmti_j9_1
serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

failed 3/10 on this machine. Didn't fail on the other 4 machines in this 50x grinder.

17:04:10  ACTION: main -- Failed. Unexpected exit from test [exit code: 87]

STDOUT:
17:04:10  check_jvmti_status: JVMTI function returned error: JVMTI_ERROR_THREAD_NOT_ALIVE (15)

STDERR:
17:04:10  Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread
@pshipton pshipton added test failure project:loom Used to track Project Loom related work comp:jvmti labels Jan 11, 2024
@pshipton pshipton mentioned this issue Jan 11, 2024
Open
@tajila tajila added the comp:vm label Jan 29, 2024
@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Mar 8, 2024 •
edited
Loading

JDK21 ppc64_aix(paix816)

java version "21.0.3-beta" 2024-04-16
IBM Semeru Runtime Certified Edition 21.0.3+6-202403080141 (build 21.0.3-beta+6-202403080141)
Eclipse OpenJ9 VM 21.0.3+6-202403080141 (build master-ae0c76e4f, JRE 21 AIX ppc64-64-Bit Compressed References 20240308_121 (JIT enabled, AOT enabled)
OpenJ9   - ae0c76e4f
OMR      - 107d389e6
JCL      - abc7b7a2f based on jdk-21.0.3+6)

[2024-03-08T07:58:42.559Z] variation: Mode150
[2024-03-08T07:58:42.559Z] JVM_OPTIONS:  -XX:+UseCompressedOops -Xverbosegclog 

[2024-03-08T08:05:05.611Z] TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

[2024-03-08T08:05:05.612Z] STDERR:
[2024-03-08T08:05:05.612Z] 
[2024-03-08T08:05:05.612Z] Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread

[2024-03-08T08:05:05.613Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
[2024-03-08T08:05:05.613Z] --------------------------------------------------
[2024-03-08T08:11:47.115Z] Test results: passed: 155; failed: 1
[2024-03-08T08:11:50.250Z] Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64_aix_testList_0/jvmtest/openjdk/report/html/report.html
[2024-03-08T08:11:50.250Z] Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64_aix_testList_0/aqa-tests/TKG/output_17098847212710/serviceability_jvmti_j9_0/work
[2024-03-08T08:11:50.250Z] Error: Some tests failed or other problems occurred.
[2024-03-08T08:11:50.250Z] -----------------------------------
[2024-03-08T08:11:50.250Z] serviceability_jvmti_j9_0_FAILED

50x grinder 2/50 failed
Fatal error: Incorrect frame count(paix821) and one timeout

@pshipton
Copy link
Member Author

serviceability/jvmti/stress/StackTrace/NotSuspended/GetStackTraceNotSuspendedStressTest.java
one failure on paix821.

20:14:58  Synchronization point checkStatus(0) called.
20:14:58  Data 100103d5638 100207b1158
20:14:58  Agent: wait for thread to start
20:14:58  Agent: started
20:14:58  Agent: Stacktrace in virtual thread is incorrect: count: 0
20:14:58  Thread: 10023444408, name: VThread-Producer-75, state(2):  TERMINATED, attrs: virtual daemon
20:14:58  JVMTI Stack Trace: frame count: 0
20:14:58  
20:14:58  STDERR:
20:14:58  
20:14:58  Fatal error: Incorrect frame count

There was also a different failure #18675 on paix827
serviceability/jvmti/vthread/SuspendResume2/SuspendResume2.java#no-vmcontinuations

@JasonFengJ9
Copy link
Member

JDK22 aarch64_linux(ubu20-aarch64-3)

[2024-03-16T20:11:25.269Z] variation: Mode650
[2024-03-16T20:11:25.269Z] JVM_OPTIONS:  -XX:-UseCompressedOops -Xverbosegclog 

[2024-03-16T20:17:09.257Z] TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

[2024-03-16T20:17:09.258Z] STDERR:
[2024-03-16T20:17:09.258Z] 
[2024-03-16T20:17:09.259Z] Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread

[2024-03-16T20:17:09.259Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
[2024-03-16T20:17:09.259Z] --------------------------------------------------
[2024-03-16T20:20:07.817Z] Test results: passed: 158; failed: 1
[2024-03-16T20:20:13.956Z] Report written to /home/jenkins/workspace/Test_openjdk22_j9_extended.openjdk_aarch64_linux_testList_1/jvmtest/openjdk/report/html/report.html
[2024-03-16T20:20:13.956Z] Results written to /home/jenkins/workspace/Test_openjdk22_j9_extended.openjdk_aarch64_linux_testList_1/aqa-tests/TKG/output_17106198831576/serviceability_jvmti_j9_1/work
[2024-03-16T20:20:13.956Z] Error: Some tests failed or other problems occurred.
[2024-03-16T20:20:13.956Z] -----------------------------------
[2024-03-16T20:20:13.956Z] serviceability_jvmti_j9_1_FAILED

@JasonFengJ9
Copy link
Member

JDK21 aarch64_linux(sles15-aarch64-2)

[2024-04-13T17:34:38.105Z] check_jvmti_status: JVMTI function returned error: JVMTI_ERROR_THREAD_NOT_ALIVE (15)

[2024-04-13T17:34:38.105Z] STDERR:
[2024-04-13T17:34:38.105Z] 
[2024-04-13T17:34:38.105Z] Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread

[2024-04-13T17:34:38.105Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
[2024-04-13T17:34:38.105Z] --------------------------------------------------
[2024-04-13T17:37:32.973Z] Test results: passed: 155; failed: 1
[2024-04-13T17:37:52.029Z] Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_linux_testList_1/jvmtest/openjdk/report/html/report.html
[2024-04-13T17:37:52.029Z] Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_aarch64_linux_testList_1/aqa-tests/TKG/output_17130294852200/serviceability_jvmti_j9_1/work
[2024-04-13T17:37:52.029Z] Error: Some tests failed or other problems occurred.
[2024-04-13T17:37:52.029Z] -----------------------------------
[2024-04-13T17:37:52.029Z] serviceability_jvmti_j9_1_FAILED

@JasonFengJ9
Copy link
Member

JDK21 ppc64le_linux(sles12le-svl-rt2-1)

java version "21.0.3" 2024-04-16 LTS
IBM Semeru Runtime Certified Edition 21.0.3.0-rc1 (build 21.0.3+9-LTS)
Eclipse OpenJ9 VM 21.0.3.0-rc1 (build v0.44.0-release-8d018d6d4, JRE 21 Linux ppc64le-64-Bit Compressed References 20240416_151 (JIT enabled, AOT enabled)
OpenJ9   - 8d018d6d4
OMR      - 254af5a04
JCL      - 6d89fe50d69 based on jdk-21.0.3+9)

[2024-04-22T14:24:54.720Z] variation: Mode150
[2024-04-22T14:24:54.720Z] JVM_OPTIONS:  -XX:+UseCompressedOops -Xverbosegclog 

[2024-04-22T14:28:00.004Z] TEST: serviceability/jvmti/stress/StackTrace/Suspended/GetStackTraceSuspendedStressTest.java

[2024-04-22T14:28:00.006Z] STDERR:
[2024-04-22T14:28:00.006Z] 
[2024-04-22T14:28:00.006Z] Fatal error: jvmti_common get_carrier_thread: Error in JVMTI extension GetCarrierThread

[2024-04-22T14:28:00.006Z] TEST RESULT: Failed. Unexpected exit from test [exit code: 87]
[2024-04-22T14:28:00.006Z] --------------------------------------------------
[2024-04-22T14:31:06.962Z] Test results: passed: 155; failed: 1
[2024-04-22T14:31:08.524Z] Report written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64le_linux_testList_0/jvmtest/openjdk/report/html/report.html
[2024-04-22T14:31:08.524Z] Results written to /home/jenkins/workspace/Test_openjdk21_j9_extended.openjdk_ppc64le_linux_testList_0/aqa-tests/TKG/output_17137958933002/serviceability_jvmti_j9_0/work
[2024-04-22T14:31:08.524Z] Error: Some tests failed or other problems occurred.
[2024-04-22T14:31:08.524Z] -----------------------------------
[2024-04-22T14:31:08.524Z] serviceability_jvmti_j9_0_FAILED

@babsingh
Copy link
Contributor

babsingh commented May 27, 2024 •
edited
Loading

GetCarrierThread is a JVMTI extension API (not regulated by the JVMTI specification), which doesn't block this release. Moving this issue to the next release.

@babsingh babsingh assigned fengxue-IS and unassigned fengxue-IS May 27, 2024
@pshipton
Copy link
Member Author

pshipton commented Jul 9, 2024 •
edited
Loading

https://trssrtp1.fyre.ibm.com/output/test?id=668ae6bc6b34225c2bda7712 - paix825
https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk22_j9_extended.openjdk_ppc64_aix_testList_3/10/
serviceability/jvmti/stress/StackTrace/NotSuspended/GetStackTraceNotSuspendedStressTest.java

[2024-07-07T03:59:53.189Z] STDOUT:
[2024-07-07T03:59:53.189Z] Agent_OnLoad started
[2024-07-07T03:59:53.189Z] Agent_OnLoad finished
[2024-07-07T03:59:53.189Z] Synchronization point checkStatus(0) called.
[2024-07-07T03:59:53.189Z] Data 10010422398 100205973d0
[2024-07-07T03:59:53.189Z] Agent: wait for thread to start
[2024-07-07T03:59:53.189Z] Agent: started
[2024-07-07T03:59:53.189Z] Agent: Stacktrace in virtual thread is incorrect: count: 0
[2024-07-07T03:59:53.189Z] Thread: 100241100d8, name: VThread-Consumer-69, state(2):  TERMINATED, attrs: virtual daemon
[2024-07-07T03:59:53.189Z] JVMTI Stack Trace: frame count: 0
[2024-07-07T03:59:53.189Z] 
[2024-07-07T03:59:53.189Z] STDERR:
[2024-07-07T03:59:53.189Z] 
[2024-07-07T03:59:53.189Z] Fatal error: Incorrect frame count

@fengxue-IS
Copy link
Contributor

https://trssrtp1.fyre.ibm.com/output/test?id=668ae6bc6b34225c2bda7712 - paix825 https://hyc-runtimes-jenkins.swg-devops.com/job/Test_openjdk22_j9_extended.openjdk_ppc64_aix_testList_3/10/ serviceability/jvmti/stress/StackTrace/NotSuspended/GetStackTraceNotSuspendedStressTest.java

This is a different test to the original issue reported.

Upon code review, this could be caused by a timing gap between vthread's last unmount and setting the vthread state to TERMINATED in afterDone() call https://github.com/ibmruntimes/openj9-openjdk-jdk22/blob/openj9/src/java.base/share/classes/java/lang/VirtualThread.java#L251-L255

The jvmtiGetStackTrace rely on getVMThread to ensure thread is alive, which checks the vthread's java state field, due to the time gap mentioned above, we could report a just finished vthread as alive but without a java stack
https://github.com/eclipse-openj9/openj9/blob/master/runtime/jvmti/jvmtiHelpers.cpp#L157-L181

I'll draft a PR to also check the JVM's internal state on the vthread to determine if thread is alive in getVMThread, this should fix the problem seen on this test, but mismatch of vthread state between java and jvmti due to the way vthread state is updated in java may still be exposed.

This was referenced Jul 10, 2024
Open
Open
@fengxue-IS fengxue-IS mentioned this issue Jul 10, 2024
Merged
fengxue-IS added a commit to fengxue-IS/openj9 that referenced this issue Jul 18, 2024
@fengxue-IS
Update VThread alive check to check continuation state
b14cd71 
Check if the continuation is finished to determine more accurately
if a virtual thread is alive.

Related: eclipse-openj9#18738 (comment)

Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>
fengxue-IS added a commit to fengxue-IS/openj9 that referenced this issue Jul 22, 2024
@fengxue-IS
Update VThread alive check to check continuation state
f96c6e0 
Check if the continuation is finished to determine more accurately
if a virtual thread is alive.

Related: eclipse-openj9#18738 (comment)

Signed-off-by: Jack Lu <Jack.S.Lu@ibm.com>
@fengxue-IS fengxue-IS mentioned this issue Jul 22, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:jvmti comp:vm project:loom Used to track Project Loom related work test failure
Projects
None yet
Milestone
Release 0.48 (Java 8, 11, 17, 21, 23)...
Development

No branches or pull requests
5 participants
@babsingh @fengxue-IS @tajila @JasonFengJ9 @pshipton