aarch64 mac LambdaLoadTest_J9_5m_0 crash vmState=0x00000000 Invalid class pointer in stack allocated object

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/48
LambdaLoadTest_J9_5m_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/48/system_test_output.tar.gz

LT  stderr Type=Segmentation error vmState=0x00000000
LT  stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
LT  stderr Handler1=00000001025D7060 Handler2=000000010232E1A4 InaccessibleAddress=7C3F344A00000010
LT  stderr x0=0000000146991700 x1=0000000146991700 x2=0000000148405900 x3=000000016EB4E920
LT  stderr x4=000000016EB4E8E8 x5=000000016EB4E8D8 x6=000000016EB4E8D0 x7=000000016EB4E8C8
LT  stderr x8=7C3F344A00000000 x9=0000000000000010 x10=000000016EB4E8D0 x11=000000016EB4E8D8
LT  stderr x12=000000016EB4E8E8 x13=000000016EB4E8C0 x14=00000001026203C8 x15=0000000000000010
LT  stderr x16=000000010261FE64 x17=000000020F0FDAB0 x18=0000000379B1C710 x19=0000000040000000
LT  stderr x20=00000002800D07D0 x21=000000016EB4E8C0 x22=000000016EB4E8D0 x23=0000000115F7A740
LT  stderr x24=000000016EB4E8E8 x25=000000016EB4E8D8 x26=000000016EB4E8D0 x27=000000016EB4E8C8
LT  stderr x28=00000001026F6DA4 x29(FP)=000000016EB4E8B0 x30(LR)=000000010261FE58 x31(SP)=000000016EB4E620
LT  stderr PC=0000000102634DEC SP=000000016EB4E620
LT  stderr v0 0000000148771679 (f: 1215764096.000000, d: 2.722663e-314)
LT  stderr v1 0000000128aad8d0 (f: 682285248.000000, d: 2.459090e-314)
LT  stderr v2 0000000040800000 (f: 1082130432.000000, d: 5.346435e-315)
LT  stderr v3 bfc6f0109b7566fb (f: 2608162560.000000, d: -1.792012e-01)
LT  stderr v4 0000000000000003 (f: 3.000000, d: 1.482197e-323)
LT  stderr v5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
LT  stderr v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
LT  stderr v7 0000000000000006 (f: 6.000000, d: 2.964394e-323)
LT  stderr v8 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
LT  stderr v9 401fff480604a506 (f: 100967688.000000, d: 7.999298e+00)
LT  stderr v10 401fff480604a506 (f: 100967688.000000, d: 7.999298e+00)
LT  stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
LT  stderr v17 3fd561ba0f8e9555 (f: 261002576.000000, d: 3.340898e-01)
LT  stderr v18 bf68d312fbea7c16 (f: 4226448384.000000, d: -3.030336e-03)
LT  stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
LT  stderr v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
LT  stderr v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
LT  stderr v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
LT  stderr v25 4010000000000000 (f: 0.000000, d: 4.000000e+00)
LT  stderr v26 4010000000000000 (f: 0.000000, d: 4.000000e+00)
LT  stderr v27 4014000000000000 (f: 0.000000, d: 5.000000e+00)
LT  stderr v28 4014000000000000 (f: 0.000000, d: 5.000000e+00)
LT  stderr v29 4010000000000000 (f: 0.000000, d: 4.000000e+00)
LT  stderr v30 4005555555555555 (f: 1431655808.000000, d: 2.666667e+00)
LT  stderr v31 4008000000000000 (f: 0.000000, d: 3.000000e+00)
LT  stderr Module=/Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
LT  stderr Module_base_address=00000001025B4000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
LT  stderr Symbol_address=000000010261FE64
LT  stderr Target=2_90_20221110_119 (Mac OS X 11.7.1)
LT  stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
LT  stderr ----------- Stack Backtrace -----------
LT  stderr ---------------------------------------
LT  stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2022/11/10 14:13:45 - please wait.
LT  stderr JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16680473003139/LambdaLoadTest_J9_5m_0/20221110-140858-LambdaLoadTest/results/core.20221110.141345.80268.0001.dmp' in response to an event
STF 14:13:46.108 - Found dump at: /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16680473003139/LambdaLoadTest_J9_5m_0/20221110-140858-LambdaLoadTest/results/core.20221110.141345.80268.0001.dmp
LT  stderr core file generated - /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16680473003139/LambdaLoadTest_J9_5m_0/20221110-140858-LambdaLoadTest/results/core.20221110.141345.80268.0001.dmp
LT  stderr JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk19_j9_sanity.system_aarch64_mac_Nightly_testList_0/aqa-tests/TKG/output_16680473003139/LambdaLoadTest_J9_5m_0/20221110-140858-LambdaLoadTest/results/core.20221110.141345.80268.0001.dmp
LT  stderr 0000000146991700: Invalid class pointer in stack allocated object in thread load-5
LT  stderr 0000000146991700:	O-Slot=0000000148771570
LT  stderr 0000000146991700:	O-Slot value=0000000148771638
LT  stderr 0000000146991700:	PC=000000012672029B
LT  stderr 0000000146991700:	framesWalked=0
LT  stderr 0000000146991700:	arg0EA=0000000148771638
LT  stderr 0000000146991700:	walkSP=0000000148771560
LT  stderr 0000000146991700:	literals=0000000127216548
LT  stderr 0000000146991700:	jitInfo=0000000000000000
LT  stderr 0000000146991700:	method=0000000127216548 (java/lang/invoke/LambdaForm$NFI/0x00000000267201a0.invoke_LL_L(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;) (Interpreted)
LT  stderr 0000000146991700:	stack=0000000148708048-0000000148771E50
STF 14:13:56.270 - **FAILED** Process LT  ended with exit code (255) and not the expected exit code/s (0)

Possible dup of #15170

[tajila]

The invalid O-slot is in a J2I frame

[knn-k]

2 failures in 30x Grinder. https://openj9-jenkins.osuosl.org/job/Grinder/1523/

[knn-k]

1 failure in 30x Grinder with the latest nightly build: https://openj9-jenkins.osuosl.org/job/Grinder/1524/

[knn-k]

The failure in https://openj9-jenkins.osuosl.org/job/Grinder/1524/ looks the same as #15251.
O-Slot value=000000017000A5B0 is j9class-> ramStatics for java/lang/Thread$ThreadNumbering as described in #15251 (comment).

From 1.LT.stderr:

000000013188A300: Object neither in heap nor stack-allocated in thread load-3
000000013188A300:       O-Slot=00000003A0121248
000000013188A300:       O-Slot value=000000017000A5B0 <---
000000013188A300:       PC=0000000117F56AD8

From jdmpview:

> !allclasses | grep java/lang/Thread$ThreadNumbering
0x000000017000A600      0x0000000000000328      0x0000000141870668      0x00000001502BC480      0       java/lang/Thread$ThreadNumbering
0x00000001502BC480      0x00000220      0x0000000141870668      0x00000020      0x0C804000      java/lang/Thread$ThreadNumbering
> !j9class 0x000000017000A600
J9Class at 0x17000a600 {
  Fields for J9Class:
        (... snip ...)
        0x40: struct J9Method* ramMethods = !j9method 0x0000000170008030 // java/lang/Thread$ThreadNumbering.<init>()V
        0x48: UDATA* ramStatics = !j9x 0x000000017000A5B0 <---
        0x50: struct J9Class* arrayClass = !j9class 0x0000000000000000
        (... snip ...)
}

[knn-k]

The same for the two failures in Grinder job 1523.
O-Slot value is the same as j9class-> ramStatics for java/lang/Thread$ThreadNumbering.

[knn-k]

Issue #15251 has been fixed. I ran a 30x Grinder job for LambdaLoadTest_J9_5m_0, and it finished with no failures.
https://openj9-jenkins.osuosl.org/job/Grinder/1611/

[knn-k]

Another 30x Grinder job finished successfully. I think this issue can be closed as a dup of #15251.
https://openj9-jenkins.osuosl.org/job/Grinder/1613/