MauveSingleThrdLoad_J9_5m Misaligned stack-allocated object

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk20_j9_sanity.system_aarch64_linux_Nightly_testList_1/78
MauveSingleThrdLoad_J9_5m_0

LT  01:50:52.575 - Completed 93.7%. Number of tests started=3227545 (+299950)
LT  stderr 0000FFFF8C252500: Misaligned stack-allocated object in thread load-0
LT  stderr 0000FFFF8C252500:	O-Slot=0000FFFE880098A0
LT  stderr 0000FFFF8C252500:	O-Slot value=0000FFFE8800970C
LT  stderr 0000FFFF8C252500:	PC=0000FFFEF4DA51E4
LT  stderr 0000FFFF8C252500:	framesWalked=5
LT  stderr 0000FFFF8C252500:	arg0EA=0000FFFE88009E78
LT  stderr 0000FFFF8C252500:	walkSP=0000FFFE880092C8
LT  stderr 0000FFFF8C252500:	literals=0000000000000010
LT  stderr 0000FFFF8C252500:	jitInfo=0000FFFEB8911678
LT  stderr 0000FFFF8C252500:	method=0000FFFE88520BE8 (gnu/testlet/java/lang/Float/compareTo.test2(Lgnu/testlet/TestHarness;)V) (JIT)
LT  stderr 0000FFFF8C252500:	stack=0000FFFE88004B28-0000FFFE8800A340
LT  stderr 01:50:57.054 0xfffe8c002400    j9mm.479    *   ** ASSERTION FAILED ** at /home/jenkins/workspace/Build_JDK20_aarch64_linux_Nightly/openj9/runtime/gc_glue_java/ScavengerRootScanner.hpp:109: ((MM_StackSlotValidator(MM_StackSlotValidator::NOT_ON_HEAP, *slotPtr, stackLocation, walkState).validate(_env)))

There are some other occurrences of this.
#16510
#17052
#17171

@dmitripivkine fyi

[dmitripivkine]

This failure looks like duplicate of #16510:
bad O-slot in JIT frame:

<ffff8c252500> JIT frame: bp = 0x0000FFFE88009E68, pc = 0x0000FFFEF4DA51E4, unwindSP = 0x0000FFFE88009300, cp = 0x0000FFFE885209D0, arg0EA = 0x0000FFFE88009E78, jitInfo = 0x0000FFFEB8911678
<ffff8c252500> 	Method: gnu/testlet/java/lang/Float/compareTo.test2(Lgnu/testlet/TestHarness;)V !j9method 0x0000FFFE88520BE8
<ffff8c252500> 	Bytecode index = 5, inlineDepth = 0, PC offset = 0x000000000000FC88
<ffff8c252500> 	stackMap=0x0000FFFEB891A6F4, slots=I16(0x0002) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0002), localBaseOffset=I16(0xF8E8)
<ffff8c252500> 	Described JIT args starting at 0x0000FFFE88009E70 for U16(0x0002) slots
<ffff8c252500> 		O-Slot: : a1[0x0000FFFE88009E70] = 0x0000FFFF8A9F5E00
<ffff8c252500> 		O-Slot: : a0[0x0000FFFE88009E78] = 0x0000FFFF8A9F5E28
<ffff8c252500> 	Described JIT temps starting at 0x0000FFFE88009750 for IDATA(0x00000000000000E3) slots
<ffff8c252500> Address 0x0000FFFEB891B2C5
<ffff8c252500> Num internal ptr map bytes U8(0x0D)
<ffff8c252500> Address 0x0000FFFEB891B2C6
<ffff8c252500> Index of first internal ptr I16(0x00DF)
<ffff8c252500> Address 0x0000FFFEB891B2C8
<ffff8c252500> Offset of first internal ptr I16(0xFFD0)
<ffff8c252500> Address 0x0000FFFEB891B2CA
<ffff8c252500> Num distinct pinning arrays U8(0x06)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E60 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E60 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x05) num internal pointer stack slots U8(0x00)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E58 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E58 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x04) num internal pointer stack slots U8(0x00)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E50 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E50 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x03) num internal pointer stack slots U8(0x00)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E48 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E48 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x02) num internal pointer stack slots U8(0x00)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E38 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E38 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x00) num internal pointer stack slots U8(0x00)
<ffff8c252500> Before object slot walk &address : 0x0000FFFE88009E40 address : 0x0000000000000000 bp 0x0000FFFE88009E68 offset of first internal ptr I16(0xFFD0)
<ffff8c252500> After object slot walk for pinning array with &address : 0x0000FFFE88009E40 old address 0x0000000000000000 new address 0x0000000000000000 displacement IDATA(0x0000000000000000)
<ffff8c252500> For pinning array U8(0x01) num internal pointer stack slots U8(0x00)
<ffff8c252500> 		I-Slot: : t226[0x0000FFFE88009750] = 0x0000FFFF8A94B938
<ffff8c252500> 		I-Slot: : t225[0x0000FFFE88009758] = 0x0000FFFF0000001F
<ffff8c252500> 		I-Slot: : t224[0x0000FFFE88009760] = 0x0000FFFF8A949678
<ffff8c252500> 		I-Slot: : t223[0x0000FFFE88009768] = 0x0000FFFF8A949920
<ffff8c252500> 		I-Slot: : t222[0x0000FFFE88009770] = 0x0000FFFF8A949BC8
<ffff8c252500> 		I-Slot: : t221[0x0000FFFE88009778] = 0x0000FFFF8A949E70
<ffff8c252500> 		I-Slot: : t220[0x0000FFFE88009780] = 0x0000FFFF8A94A118
<ffff8c252500> 		I-Slot: : t219[0x0000FFFE88009788] = 0x0000FFFF8A94A3C0
<ffff8c252500> 		I-Slot: : t218[0x0000FFFE88009790] = 0x0000FFFF8A94A668
<ffff8c252500> 		I-Slot: : t217[0x0000FFFE88009798] = 0x0000FFFF8A94A910
<ffff8c252500> 		I-Slot: : t216[0x0000FFFE880097A0] = 0x0000FFFF8A94ABF0
<ffff8c252500> 		I-Slot: : t215[0x0000FFFE880097A8] = 0x0000FFFF8A94B0B8
<ffff8c252500> 		I-Slot: : t214[0x0000FFFE880097B0] = 0x0000FFFF8A94B3A0
<ffff8c252500> 		I-Slot: : t213[0x0000FFFE880097B8] = 0x0000FFFF8A94B688
<ffff8c252500> 		I-Slot: : t212[0x0000FFFE880097C0] = 0x0000FFFF8A94B970
<ffff8c252500> 		I-Slot: : t211[0x0000FFFE880097C8] = 0x0000FFFF8A94B990
<ffff8c252500> 		I-Slot: : t210[0x0000FFFE880097D0] = 0x0000FFFF8A94B6A8
<ffff8c252500> 		I-Slot: : t209[0x0000FFFE880097D8] = 0x0000FFFF8A94B3C0
<ffff8c252500> 		I-Slot: : t208[0x0000FFFE880097E0] = 0x0000FFFF8A94B0D8
<ffff8c252500> 		I-Slot: : t207[0x0000FFFE880097E8] = 0x0000FFFF8A94AC10
<ffff8c252500> 		I-Slot: : t206[0x0000FFFE880097F0] = 0x0000FFFF8A94A930
<ffff8c252500> 		I-Slot: : t205[0x0000FFFE880097F8] = 0x0000FFFF8A94A688
<ffff8c252500> 		I-Slot: : t204[0x0000FFFE88009800] = 0x0000FFFF8A94A3E0
<ffff8c252500> 		I-Slot: : t203[0x0000FFFE88009808] = 0x0000FFFF8A94A138
<ffff8c252500> 		I-Slot: : t202[0x0000FFFE88009810] = 0x0000FFFF8A949E90
<ffff8c252500> 		I-Slot: : t201[0x0000FFFE88009818] = 0x0000FFFF8A949BE8
<ffff8c252500> 		I-Slot: : t200[0x0000FFFE88009820] = 0x0000FFFF8A949940
<ffff8c252500> 		I-Slot: : t199[0x0000FFFE88009828] = 0x0000FFFF8A949698
<ffff8c252500> 		I-Slot: : t198[0x0000FFFE88009830] = 0x0000FFFF8A9493F0
<ffff8c252500> 		I-Slot: : t197[0x0000FFFE88009838] = 0x0000000100000000
<ffff8c252500> 		I-Slot: : t196[0x0000FFFE88009840] = 0x000000010F6D51F8
<ffff8c252500> 		I-Slot: : t195[0x0000FFFE88009848] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t194[0x0000FFFE88009850] = 0x0000000000000765
<ffff8c252500> 		I-Slot: : t193[0x0000FFFE88009858] = 0x00000000F3DA49C0
<ffff8c252500> 		I-Slot: : t192[0x0000FFFE88009860] = 0x0000FFFF0F6D11E8
<ffff8c252500> 		I-Slot: : t191[0x0000FFFE88009868] = 0x0000FFFF0F6D11E8
<ffff8c252500> 		I-Slot: : t190[0x0000FFFE88009870] = 0x0000FFFF0EE3D620
<ffff8c252500> 		I-Slot: : t189[0x0000FFFE88009878] = 0x0000FFFF10765718
<ffff8c252500> 		I-Slot: : t188[0x0000FFFE88009880] = 0x0000FFFF10765718
<ffff8c252500> 		I-Slot: : t187[0x0000FFFE88009888] = 0x0000FFFEF3F75038
<ffff8c252500> 		I-Slot: : t186[0x0000FFFE88009890] = 0x0000FFFF106A8418
<ffff8c252500> 		I-Slot: : t185[0x0000FFFE88009898] = 0x0000FFFF8A8F1430
<ffff8c252500> 		O-Slot: : t184[0x0000FFFE880098A0] = 0x0000FFFE8800970C <-----
<ffff8c252500> 		I-Slot: : t183[0x0000FFFE880098A8] = 0x0000FFFF8C20D700
<ffff8c252500> 		I-Slot: : t182[0x0000FFFE880098B0] = 0x0000FFFF0F6D51F8
<ffff8c252500> 		I-Slot: : t181[0x0000FFFE880098B8] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t180[0x0000FFFE880098C0] = 0x0000FFFF00000000
.....

[dmitripivkine]

There are more similar bad O-slots in this frame contain the same misaligned value 0x0000FFFE8800970C:

<ffff8c252500> 		I-Slot: : t13[0x0000FFFE88009DF8] = 0x0000000000000001
<ffff8c252500> 		I-Slot: : t12[0x0000FFFE88009E00] = 0x0000000000000001
<ffff8c252500> 		I-Slot: : t11[0x0000FFFE88009E08] = 0x0000000000000001
<ffff8c252500> 		O-Slot: : t10[0x0000FFFE88009E10] = 0x0000FFFE8800970C <-----
<ffff8c252500> 		O-Slot: : t9[0x0000FFFE88009E18] = 0x0000FFFE8800970C <-----
<ffff8c252500> 		I-Slot: : t8[0x0000FFFE88009E20] = 0x0000FFFF0FAF0410
<ffff8c252500> 		O-Slot: : t7[0x0000FFFE88009E28] = 0x0000FFFE8800970C <-----
<ffff8c252500> 		O-Slot: : t6[0x0000FFFE88009E30] = 0x0000FFFE8800970C <-----
<ffff8c252500> 		I-Slot: : t5[0x0000FFFE88009E38] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t4[0x0000FFFE88009E40] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t3[0x0000FFFE88009E48] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t2[0x0000FFFE88009E50] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t1[0x0000FFFE88009E58] = 0x0000000000000000
<ffff8c252500> 		I-Slot: : t0[0x0000FFFE88009E60] = 0x0000000000000000
<ffff8c252500> 	JIT-RegisterMap = UDATA(0x0000000000000000)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFE88009328] = UDATA(0x0000000000000000) (jit_r21)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F2F0] = UDATA(0x0000FFFE8802FE00) (jit_r22)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F2F8] = UDATA(0x0000FFFF8A9EC650) (jit_r23)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F300] = UDATA(0x0000FFFE8802FE00) (jit_r24)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F308] = UDATA(0x0000FFFE8802FE00) (jit_r25)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F310] = UDATA(0x0000FFFE8802FE00) (jit_r26)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F318] = UDATA(0x0000FFFE8802FE00) (jit_r27)
<ffff8c252500> 	JIT-Frame-RegisterMap[0x0000FFFEF0A1F320] = UDATA(0x0000000000000002) (jit_r28)

Most of this "numbered" slots are I-slots, there are only five O-slots and all of them points to the same bad value. This value points to the address in the same java thread stack:

0xFFFE880096E0 :  0000000100000001 0000000100000001 [ ................ ]
0xFFFE880096F0 :  0000000100000001 0000000100000001 [ ................ ]
0xFFFE88009700 :  0000000100000001 0000ffffff800000 [ ................ ] <-----
0xFFFE88009710 :  0000000000000000 0000000100000001 [ ................ ]
0xFFFE88009720 :  0000000100000001 0000000100000001 [ ................ ]

For the record: GC uses bit 0x4 to recognize Forwarded Pointers. However this bit can be installed to the header of the objects located in Evacuate part of the Nursery. I can not see how it can apply to stack slots. So, I think it is not a case here.