Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

j9mm.479 * ** ASSERTION FAILED ** at openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env))) #13369

Closed
JasonFengJ9 opened this issue Aug 24, 2021 · 13 comments

Comments

@JasonFengJ9
Copy link
Member

JasonFengJ9 commented Aug 24, 2021

JDK16 Windows build has same assertion error at Test_openjdk16_j9_special.system_x86-64_windows_Personal/6/tapResults/

openjdk version "16.0.2-internal" 2021-07-20
OpenJDK Runtime Environment (build 16.0.2-internal+0-adhoc.jenkins.buildjdk16x86-64windowspersonal)
Eclipse OpenJ9 VM (build master-2ba48511f18, JRE 16 Windows Server 2016 amd64-64-Bit Compressed References 20210822_24 (JIT enabled, AOT enabled)
OpenJ9   - 2ba48511f18
OMR      - 6eec759cd2d
JCL      - 44316ee515b based on jdk-16.0.2+7)

===============================================
Running test LambdaLoadTest_special_J9_5m_1 ...
===============================================
LambdaLoadTest_special_J9_5m_1 Start Time: Sun Aug 22 22:36:32 2021 Epoch Time (ms): 1629696992224
variation: Mode103
JVM_OPTIONS:  -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs 

LT  22:37:24.802 - Test failed. Details recorded in execution log.
LT  stderr 0000021794419D00: Object neither in heap nor stack-allocated in thread load-5
LT  stderr 0000021794419D00:	O-Slot=000000274B41F2B8
LT  stderr 0000021794419D00:	O-Slot value=0074007300650074
LT  stderr 0000021794419D00:	PC=00007FFD6F6E1C32
LT  stderr 0000021794419D00:	framesWalked=4
LT  stderr 0000021794419D00:	arg0EA=0000021794A39258
LT  stderr 0000021794419D00:	walkSP=0000021794A39138
LT  stderr 0000021794419D00:	literals=0000000000000010
LT  stderr 0000021794419D00:	jitInfo=0000021795A7EC68
LT  stderr 0000021794419D00:	method=0000021794500420 (net/adoptopenjdk/loadTest/adaptors/JUnitAdaptor$1.addResult(Ljava/lang/String;Ljava/lang/String;Lorg/junit/runner/Description;Ljava/lang/String;Ljava/lang/Throwable;)V) (JIT)
LT  stderr 0000021794419D00:	stack=0000021794A328A0-0000021794A39930
LT  stderr 05:37:24.846 0x21793de9400    j9mm.479    *   ** ASSERTION FAILED ** at c:\users\jenkins\workspace\build_jdk16_x86-64_windows_personal\openj9\runtime\gc_glue_java\MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2021/08/22 22:37:24 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using 'C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\core.20210822.223724.5288.0001.dmp' in response to an event
STF 22:37:25.228 - Found dump at: C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\core.20210822.223724.5288.0001.dmp
LT  stderr core file generated - C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\core.20210822.223724.5288.0001.dmp
LT  stderr 88.0001.dmp
LT  stderr JVMDUMP032I JVM requested Java dump using 'C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\javacore.20210822.223724.5288.0002.txt' in response to an event
STF 22:37:32.218 - Found dump at: C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\javacore.20210822.223724.5288.0002.txt
LT  stderr javacore file generated - C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\javacore.20210822.223724.5288.0002.txt
LT  stderr 2.txt
LT  stderr JVMDUMP032I JVM requested Snap dump using 'C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\Snap.20210822.223724.5288.0003.trc' in response to an event
LT  stderr JVMDUMP010I Snap dump written to C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\Snap.20210822.223724.5288.0003.trc
LT  stderr JVMDUMP013I Processed dump event "traceassert", detail "".
STF 22:37:35.342 - Found dump at: C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\Snap.20210822.223724.5288.0003.trc
LT  stderr Snap file generated - C:\Users\jenkins\workspace\Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2\aqa-tests\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1\20210822-223635-LambdaLoadTest\results\Snap.20210822.223724.5288.0003.trc
STF 22:37:35.907 - **FAILED** Process LT  ended with exit code (4294967295) and not the expected exit code/s (0)
STF 22:37:35.908 - Monitoring Report Summary:
STF 22:37:35.908 -   o Process LT  has crashed unexpectedly
STF 22:37:35.909 - Killing processes: LT 
STF 22:37:35.909 -   o Process LT  pid 5288 is not running
**FAILED** at step 1 (Run lambda and stream load test). Expected return value=0 Actual=1 at C:/Users/jenkins/workspace/Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2/aqa-tests/\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1/20210822-223635-LambdaLoadTest/execute.pl line 97.
STF 22:37:36.435 - **FAILED** execute script failed. Expected return value=0 Actual=1
STF 22:37:36.435 - 
STF 22:37:36.435 - ====================   T E A R D O W N   ====================
STF 22:37:36.435 - Running teardown: perl C:/Users/jenkins/workspace/Test_openjdk16_j9_special.system_x86-64_windows_Personal_testList_2/aqa-tests/\TKG\output_16296893109699\LambdaLoadTest_special_J9_5m_1/20210822-223635-LambdaLoadTest/tearDown.pl
STF 22:37:36.752 - TEARDOWN stage completed
STF 22:37:36.782 - 
STF 22:37:36.782 - =====================   R E S U L T S   =====================
STF 22:37:36.782 - Stage results:
STF 22:37:36.782 -   setUp:     pass
STF 22:37:36.782 -   execute:  *fail*
STF 22:37:36.782 -   teardown:  pass
STF 22:37:36.782 - 
STF 22:37:36.782 - Overall result: **FAILED**

LambdaLoadTest_special_J9_5m_1_FAILED

Re-run the Grinder

Edit: assigning this to 0.28 release since JDK16 head stream w/ ojdk_mh is used for JDK17 test.

Originally posted by @JasonFengJ9 in #13017 (comment)

There is a similar assertion at aarch64_linux #13017. Created this issue to track Windows platform as per #13017 (comment)

@JasonFengJ9
Copy link
Member Author

Also observed at MacOS Test_openjdk16_j9_special.system_x86-64_mac_Personal/6/tapResults/

openjdk version "16.0.2-internal" 2021-07-20
OpenJDK Runtime Environment (build 16.0.2-internal+0-adhoc.jenkins.BuildJDK16x86-64macPersonal)
Eclipse OpenJ9 VM (build master-5d2495e99a7, JRE 16 Mac OS X amd64-64-Bit Compressed References 20210823_100 (JIT enabled, AOT enabled)
OpenJ9   - 5d2495e99a7
OMR      - 6eec759cd2d
JCL      - 44316ee515b based on jdk-16.0.2+7)

===============================================
Running test LambdaLoadTest_special_J9_5m_4 ...
===============================================
LambdaLoadTest_special_J9_5m_4 Start Time: Mon Aug 23 17:09:47 2021 Epoch Time (ms): 1629752987453
variation: Mode112
JVM_OPTIONS:  -Xgcpolicy:gencon -Xjit:count=0 -Xnocompressedrefs 

LT  17:10:05.764 - Starting thread. Suite=0 thread=5
LT  stderr 00007FFB26968700: Object neither in heap nor stack-allocated in thread load-4
LT  stderr 00007FFB26968700:	O-Slot=000070000D3AFB98
LT  stderr 00007FFB26968700:	O-Slot value=0074007300650074
LT  stderr 00007FFB26968700:	PC=000000011EE13FB2
LT  stderr 00007FFB26968700:	framesWalked=4
LT  stderr 00007FFB26968700:	arg0EA=00007FFB2D170F38
LT  stderr 00007FFB26968700:	walkSP=00007FFB2D170E18
LT  stderr 00007FFB26968700:	literals=0000000000000010
LT  stderr 00007FFB26968700:	jitInfo=00007FFB2CBE71E8
LT  stderr 00007FFB26968700:	method=00007FFB2CA31320 (net/adoptopenjdk/loadTest/adaptors/JUnitAdaptor$1.addResult(Ljava/lang/String;Ljava/lang/String;Lorg/junit/runner/Description;Ljava/lang/String;Ljava/lang/Throwable;)V) (JIT)
LT  stderr 00007FFB26968700:	stack=00007FFB2D14E040-00007FFB2D1715D0
LT  stderr 21:10:11.901 0x7ffb2689df00    j9mm.479    *   ** ASSERTION FAILED ** at /Users/jenkins/workspace/Build_JDK16_x86-64_mac_Personal/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))
LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2021/08/23 17:10:11 - please wait.

STF 17:10:20.361 - =====================   R E S U L T S   =====================
STF 17:10:20.361 - Stage results:
STF 17:10:20.361 -   setUp:     pass
STF 17:10:20.361 -   execute:  *fail*
STF 17:10:20.361 -   teardown:  pass
STF 17:10:20.361 - 
STF 17:10:20.361 - Overall result: **FAILED**

LambdaLoadTest_special_J9_5m_4_FAILED

Re-run the Grinder

@JasonFengJ9 JasonFengJ9 changed the title JDK16 Windows : j9mm.479 * ** ASSERTION FAILED ** at openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env))) JDK16 : j9mm.479 * ** ASSERTION FAILED ** at openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env))) Aug 24, 2021
@pshipton
Copy link
Member

@dmitripivkine fyi

@dmitripivkine
Copy link
Contributor

As mentioned above this is exactly the same issue as #13017,
see my comment #13017 (comment)

The problem seems is not limited to ARM platform

@0xdaryl FYI

@dmitripivkine
Copy link
Contributor

@pshipton I can triage this if you like to have more information about frame where the bad o-slot is discovered and compare between Windows/Mac/Arm cases (I assuming system cores are available). These failures are identical including O-Slot value=0074007300650074 - tset chars in it

@dmitripivkine
Copy link
Contributor

dmitripivkine commented Aug 24, 2021

I found Windows build but there is artefacts stored
The same for Mac build
Trying to reproduce problem in Grinder

@pshipton
Copy link
Member

If it helps the JIT team.

@hzongaro
Copy link
Member

I've not been able to reproduce the failures so far.

@pshipton
Copy link
Member

I'll go ahead and close, we can reopen if seen again.

@hzongaro
Copy link
Member

Re-opening - I was able to reproduce the failure.

@hzongaro hzongaro reopened this Sep 14, 2021
@JasonFengJ9
Copy link
Member Author

Observed at JDK8 aarch64_linux job/Test_openjdk8_j9_special.system_aarch64_linux_testList_4/81/

[2021-09-14T09:52:54.854Z] openjdk version "1.8.0_312-ea"
[2021-09-14T09:52:54.854Z] IBM Semeru Runtime Open Edition (build 1.8.0_312-ea-202109140243-b05)
[2021-09-14T09:52:54.854Z] Eclipse OpenJ9 VM (build master-87c301e9d, JRE 1.8.0 Linux aarch64-64-Bit Compressed References 20210914_177 (JIT enabled, AOT enabled)
[2021-09-14T09:52:54.854Z] OpenJ9   - 87c301e9d
[2021-09-14T09:52:54.854Z] OMR      - 17c439ee7
[2021-09-14T09:52:54.854Z] JCL      - 8d7af23b based on jdk8u312-b05)
[2021-09-14T09:48:00.956Z] Running test MauveMultiThrdLoad_special_5m_3 ...
[2021-09-14T09:48:00.956Z] ===============================================
[2021-09-14T09:48:00.956Z] MauveMultiThrdLoad_special_5m_3 Start Time: Tue Sep 14 05:47:59 2021 Epoch Time (ms): 1631612879386
[2021-09-14T09:48:00.956Z] "/home/jenkins/workspace/Test_openjdk8_j9_special.system_aarch64_linux_testList_4/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:destroyAll; "/home/jenkins/workspace/Test_openjdk8_j9_special.system_aarch64_linux_testList_4/openjdkbinary/j2sdk-image/bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
[2021-09-14T09:48:00.956Z] JVMSHRC005I No shared class caches available
[2021-09-14T09:48:00.956Z] JVMSHRC005I No shared class caches available
[2021-09-14T09:48:00.956Z] cache cleanup done
[2021-09-14T09:48:00.956Z] variation: Mode107
[2021-09-14T09:48:00.956Z] JVM_OPTIONS:  -Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:count=0 -Xnocompressedrefs 

[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00: Object neither in heap nor stack-allocated in thread load-3
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	O-Slot=0000FFFEC8A82388
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	O-Slot value=0000FFFF901BB000
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	PC=0000FFFF02693FDC
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	framesWalked=2
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	arg0EA=0000FFFE8C008158
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	walkSP=0000FFFE8C008068
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	literals=0000000000000000
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	jitInfo=0000FFFEFDB5FDF8
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	method=0000FFFE940D4D38 (gnu/testlet/java/lang/System/arraycopy.copy(Ljava/lang/Object;ILjava/lang/Object;II)Ljava/lang/Object;) (JIT)
[2021-09-14T09:52:48.708Z] LT  stderr 0000FFFF90AE5A00:	stack=0000FFFE8C002E60-0000FFFE8C008710
[2021-09-14T09:52:48.708Z] LT  stderr 09:52:45.719 0xfffebc002400    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/build-scripts/jobs/jdk8u/jdk8u-linux-aarch64-openj9/workspace/build/src/openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env)))
[2021-09-14T09:52:48.708Z] LT  stderr JVMDUMP039I Processing dump event "traceassert", detail "" at 2021/09/14 05:52:45 - please wait.

[2021-09-14T09:52:50.602Z] MauveMultiThrdLoad_special_5m_3_FAILED

@JasonFengJ9 JasonFengJ9 changed the title JDK16 : j9mm.479 * ** ASSERTION FAILED ** at openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env))) j9mm.479 * ** ASSERTION FAILED ** at openj9/runtime/gc_glue_java/MarkingSchemeRootMarker.cpp:53: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, object, stackLocation, walkState).validate(_env))) Sep 14, 2021
@0xdaryl
Copy link
Contributor

0xdaryl commented Sep 15, 2021

Failures like this on AArch64 may be #13507. I'm not saying that definitively, but it is a possiblity. That PR was just merged yesterday.

@hzongaro
Copy link
Member

So far I've seen failures on MacOS, but they haven't exhibited the O-Slot assertion failure symptom. Instead I've seen NullPointerExceptions like the following, which bears a strong resemblance to issue #13472. I don't know whether they are two different symptoms of the same problem or two distinct problems, so I'll continue to investigate.

20:10:14.287 - Parsing inventory file. Root=/Users/jenkins/workspace/Grinder_testList_0/aqa-tests/TKG/output_1631664579426/LambdaLoadTest_special_J9_5m_4/20210914-200939-LambdaLoadTest/results/1.LT.inventory File=openjdk.test.load/config/inventories/lambdasAndStreams/lambda.xml
20:10:14.312 - Failed to initialise LoadTest
java.lang.NullPointerException: Cannot read field "member" because "mh" is null
	at java.lang.invoke.DirectMethodHandle.internalMemberName(DirectMethodHandle.java:330) ~[?:?]
	at java.lang.invoke.LambdaForm$DMH/0x0000000000000000.invokeStatic(LambdaForm$DMH) ~[?:?]
	at java.lang.invoke.LambdaForm$MH/0x0000000000000000.invoke(LambdaForm$MH) ~[?:?]
	at java.lang.invoke.LambdaForm$MH/0x0000000000000000.linkToTargetMethod(LambdaForm$MH) ~[?:?]
	at net.adoptopenjdk.stf.environment.DirectoryRef.childFile(DirectoryRef.java:112) ~[bin/:?]
	at net.adoptopenjdk.loadTest.InventoryData.readInventoryFile(InventoryData.java:135) ~[bin/:?]
	at net.adoptopenjdk.loadTest.InventoryData.<init>(InventoryData.java:96) ~[bin/:?]
	at net.adoptopenjdk.loadTest.LoadTest.createSuite(LoadTest.java:478) ~[bin/:?]
	at net.adoptopenjdk.loadTest.LoadTest.<init>(LoadTest.java:329) ~[bin/:?]
	at net.adoptopenjdk.loadTest.LoadTest.main(LoadTest.java:106) [bin/:?]

@0xdaryl
Copy link
Contributor

0xdaryl commented Sep 22, 2021

Closing, as the original problem reported in this issue is a likely dup of #13180. Other issues encountered during the investigation seem to be tracked by other issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants