jdk19 Compiling method: java/lang/Thread.genThreadName() crash vmstate 0005FFFF

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/3/

jdk_util_1\work\java\util\concurrent\FutureTask\ExplicitSet\
jdk_util_1\work\java\util\concurrent\Semaphore\RacingReleases
jdk_util_1\work\java\util\jar\JarFile\jarVerification\MultiProviderTest\

contains a javacore and core file for a gpf.

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/3/openjdk_test_output.tar.gz

1XHEXCPMODULE  Compiling method: java/lang/Thread.genThreadName()Ljava/lang/String;
NULL
1XHFLAGS       VM flags:000000000005FFFF

Changes from previous build:
2712974...0ed4cfa
eclipse-openj9/openj9-omr@16e09fb...560f16b

@tajila

[pshipton]

Maybe related to #15251

I only started running regular testing on jdk19 two nights ago.

[tajila]

@r30shah ^^

[r30shah]

Looking at the core-dump from one of the failure from the diagnostic files (.../aqa-tests/TKG/output_16626899173765/jdk_util_1/work/java/util/concurrent/Semaphore/RacingReleases/core.20220908.225950.46458.0001.dmp), I see following stack trace.

{libj9vm29.so}{hashTableFind}
{libj9vm29.so}{isAllowedReadAccessToModule}
{libj9vm29.so}{checkModuleAccess}
{libj9vm29.so}{checkVisibility}
{libj9vm29.so}{resolveStaticFieldRefInto}
{libj9jit29.so}{jitCTResolveStaticFieldRefWithMethod}
{libj9jit29.so}{TR_RelocationRecordDataAddress::findDataAddress(TR_RelocationRuntime*, TR_RelocationTarget*)}
{libj9jit29.so}{TR_RelocationRecordDataAddress::applyRelocation(TR_RelocationRuntime*, TR_RelocationTarget*, unsigned char*)}
{libj9jit29.so}{TR_RelocationRecord::applyRelocationAtAllOffsets(TR_RelocationRuntime*, TR_RelocationTarget*, unsigned char*)}
{libj9jit29.so}{TR_RelocationRecordGroup::applyRelocations(TR_RelocationRuntime*, TR_RelocationTarget*, unsigned char*)}
{libj9jit29.so}{TR_RelocationRuntime::relocateAOTCodeAndData(unsigned char*, unsigned char*, unsigned char*, unsigned char*)}
{libj9jit29.so}{TR_RelocationRuntime::prepareRelocateAOTCodeAndData(J9VMThread*, TR_FrontEnd*, TR::CodeCache*, J9JITDataCacheHeader const*, J9Method*, bool, TR::Options*, TR::Compilation*, TR_ResolvedMethod*, unsigned char*)}
{libj9jit29.so}{TR::CompilationInfoPerThreadBase::installAotCachedMethod(J9VMThread*, void const*, J9Method*, TR_FrontEnd*, TR::Options*, TR_ResolvedMethod*, TR_MethodToBeCompiled*, TR::Compilation*)}

So the failure happens when we are installing AOT cached method. Segmentation fault happens trying to find out an entry in the hash table, where we call the hashTableFind [1] with NULL table. That call to hashTableFind was made from isAllowedReadAccessToModule [2] and the NULL argument toModule->readAccessHashTable is passed in. From the core-dump, by inspecting the toModule variable in that function following are the contents.

0x000003ff9814e000 J9Class - {java/io/Serializable}
Memory Contents
0x3ff9814e000: 0x0000000099669966 Unknown!
0x3ff9814e008: 0x000003fedd46d6e8 Ptr Unknown!
0x3ff9814e010: 0x000003ff9814dfb8 Ptr J9Class - {java/lang/Object}
0x3ff9814e018: 0x00000000001e0001 Unknown!
0x3ff9814e020: 0x0000000000000000 Unknown!
0x3ff9814e028: 0x000003ff980938e8 MemBlk - /home/jenkins/workspace/Build_JDK19_s390x_linux_Nightly/openj9/runtime/vm/jvminit.c:2421 +80
0x3ff9814e030: 0x000003ff306005b0 Obj - {java/lang/Class} for {java/io/Serializable}
0x3ff9814e038: 0x0000000000000000 Unknown!

So when we come into the isAllowedReadAccessModule we have a J9Class pointer instead J9Module pointer passed in as toModule.

Further checking down the stack from where we pass in this wrong object, I see that a call made from resolveStaticFieldRefInto at [3] to resolveClassRef [4] returns a corrupted class and we later ended up with corrupted J9Module. I was able to find out the arguments sent to that function which are as following.

vmStruct = 0x000003ff10002200 - VMThread pointer
ramCP = 0x000003ff9819a590 - CP - {java/lang/Thread}
cpIndex = 0x00003878
resolveFlags = 0x0000000000000008

On the surface, it looks like we have emitted a code in the AOT compiled code for which we do not have a relocation yet, but to confirm would need further investigations.

@dchopra001 Can you please take a look at this ? I did launch an internal grinder and seems like it is reproducible there as well.

[1]. https://github.com/eclipse/omr/blob/6568a4852f78c6b9dde438c4c0905627d6c9cdf7/util/hashtable/hashtable.c#L442
[2].

openj9/runtime/util/modularityHelper.c

Line 78 in b8752ec

targetPtr = hashTableFind(toModule->readAccessHashTable, &fromModule);

[3].

openj9/runtime/vm/resolvesupport.cpp

Line 710 in b8752ec

resolvedClass = resolveClassRef(vmStruct, ramCP, romFieldRef->classRefCPIndex, resolveFlags);

[4].

openj9/runtime/vm/resolvesupport.cpp

Line 257 in b8752ec

resolveClassRef(J9VMThread *vmStruct, J9ConstantPool *ramCP, UDATA cpIndex, UDATA resolveFlags)

[vij-singh]

Any new news on this one?

[r30shah]

@dchopra001 Can you post update of your investigations on this issue?

[dchopra001]

I haven't been able to reproduce this one locally. So instead I'm trying to reproduce this failure in the internal Jenkins grinder infrastructure.

Update on that front is that so far I've been able to reproduce the crash and some of the rtlogs generated during the AOT load are available. However I don't see the original AOT compilation trace logs anywhere in the failing jobs. I'll need those to progress further on this one. It's possible that the original JVM that does the AOT compilation doesn't receive the JVM options or that the logs are deleted for whatever reason. I'm experimenting with the options and will update here once I know more or am able to reproduce the failure with enough tracing info.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/14
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/14/functional_test_output.tar.gz

.Running  :testCacheCreationListingDestroying02  (Test suite : TestOptionsCacheDir)
#0: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa967ee) [0x3ff996167ee]
#1: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaa3e00) [0x3ff99623e00]
#2: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x174d04) [0x3ff98cf4d04]
#3: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x31c6e) [0x3ffa54b1c6e]
#4: [0x3ff7d6f9f58]
#5: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so(+0x84cd8) [0x3ffa5604cd8]
#6: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbe1252) [0x3ff99761252]
#7: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x5568fe) [0x3ff990d68fe]
#8: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x556af2) [0x3ff990d6af2]
#9: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x5531d4) [0x3ff990d31d4]
#10: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x552bc2) [0x3ff990d2bc2]
#11: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x561714) [0x3ff990e1714]
#12: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x561f88) [0x3ff990e1f88]
#13: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18bf88) [0x3ff98d0bf88]
#14: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18fa46) [0x3ff98d0fa46]
#15: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x1906a6) [0x3ff98d106a6]
#16: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19131a) [0x3ff98d1131a]
#17: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ba8) [0x3ffa54b2ba8]
#18: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e9e4) [0x3ff98d0e9e4]
#19: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18ef74) [0x3ff98d0ef74]
#20: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18d9e6) [0x3ff98d0d9e6]
#21: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18deec) [0x3ff98d0deec]
#22: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18df84) [0x3ff98d0df84]
#23: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ba8) [0x3ffa54b2ba8]
#24: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e3c6) [0x3ff98d0e3c6]
#25: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ffa5408a5e]
#26: /lib64/libpthread.so.0(+0x8312) [0x3ffa5e88312]
#27: /lib64/libc.so.6(+0x10e232) [0x3ffa5d0e232]
Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=000000a9 Signal_Code=00000001
Handler1=000003FFA55C44B0 Handler2=000003FFA54B1A50 InaccessibleAddress=0000000000000000
gpr0=0000000000000004 gpr1=000003FFA57F18BC gpr2=00000000000007B3 gpr3=00000000000B8C90
gpr4=000003FFA5604CD0 gpr5=0000000000000008 gpr6=0000000000000008 gpr7=0000000000080000
gpr8=00000000000B8C90 gpr9=000003FF414B95E0 gpr10=000000000006BD00 gpr11=00000000000007B3
gpr12=000003FF99A39000 gpr13=000003FFA57AE218 gpr14=000003FFA5604CD0 gpr15=000003FF7D6FA3E0
psw=000003FFA5604CD8 mask=0705200180000000 fpc=0008fe00 bea=000003FFA56041EC
fpr0 000003ff7d6fc220 (f: 2104476160.000000, d: 2.171841e-311)
fpr1 3fb93d4bb7e327a9 (f: 3085117440.000000, d: 9.859155e-02)
fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr3 3fd999f1abd2dada (f: 2882722560.000000, d: 4.000210e-01)
fpr4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr5 3fcc723e7dcde3fb (f: 2110645248.000000, d: 2.222365e-01)
fpr6 3fcb2a68ec9ad9b5 (f: 3969571328.000000, d: 2.122317e-01)
fpr7 3f83e83bb8b32a19 (f: 3098749440.000000, d: 9.720294e-03)
fpr8 000003ff8d714088 (f: 2373009664.000000, d: 2.171974e-311)
fpr9 000003ff60024d90 (f: 1610763648.000000, d: 2.171598e-311)
fpr10 000003ff8d7e4be0 (f: 2373864448.000000, d: 2.171975e-311)
fpr11 00000000065f90b0 (f: 106926256.000000, d: 5.282859e-316)
fpr12 0005ea53957f6f5a (f: 2508156672.000000, d: 8.226290e-309)
fpr13 0000000000000001 (f: 1.000000, d: 4.940656e-324)
fpr14 0000000000000005 (f: 5.000000, d: 2.470328e-323)
fpr15 00000000065f90a0 (f: 106926240.000000, d: 5.282858e-316)
Module=/home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
Module_base_address=000003FFA5580000

Method_being_compiled=java/lang/Thread.genThreadName()Ljava/lang/String;
Target=2_90_20221005_87 (Linux 3.10.0-1160.76.1.el7.s390x)
CPU=s390x (4 logical CPUs) (0x1ec1b1000 RAM)
----------- Stack Backtrace -----------
resolveStaticFieldRefInto+0x118 (0x000003FFA5604CD8 [libj9vm29.so+0x84cd8])
jitCTResolveStaticFieldRefWithMethod+0x62 (0x000003FF99761252 [libj9jit29.so+0xbe1252])
_ZN30TR_RelocationRecordDataAddress15findDataAddressEP20TR_RelocationRuntimeP19TR_RelocationTarget+0x106 (0x000003FF990D68FE [libj9jit29.so+0x5568fe])
_ZN30TR_RelocationRecordDataAddress15applyRelocationEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0x22 (0x000003FF990D6AF2 [libj9jit29.so+0x556af2])
_ZN19TR_RelocationRecord27applyRelocationAtAllOffsetsEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0x30c (0x000003FF990D31D4 [libj9jit29.so+0x5531d4])
_ZN24TR_RelocationRecordGroup16applyRelocationsEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0xf2 (0x000003FF990D2BC2 [libj9jit29.so+0x552bc2])
_ZN20TR_RelocationRuntime22relocateAOTCodeAndDataEPhS0_S0_S0_+0x2e4 (0x000003FF990E1714 [libj9jit29.so+0x561714])
_ZN20TR_RelocationRuntime29prepareRelocateAOTCodeAndDataEP10J9VMThreadP11TR_FrontEndPN2TR9CodeCacheEPK20J9JITDataCacheHeaderP8J9MethodbPNS4_7OptionsEPNS4_11CompilationEP17TR_ResolvedMethodPh+0x3d8 (0x000003FF990E1F88 [libj9jit29.so+0x561f88])
_ZN2TR28CompilationInfoPerThreadBase22installAotCachedMethodEP10J9VMThreadPKvP8J9MethodP11TR_FrontEndPNS_7OptionsEP17TR_ResolvedMethodP21TR_MethodToBeCompiledPNS_11CompilationE+0x98 (0x000003FF98D0BF88 [libj9jit29.so+0x18bf88])
_ZN2TR28CompilationInfoPerThreadBase14performAOTLoadEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodP11TR_J9VMBaseP8J9Method+0xb6 (0x000003FF98D0FA46 [libj9jit29.so+0x18fa46])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x916 (0x000003FF98D106A6 [libj9jit29.so+0x1906a6])
_ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x392 (0x000003FF98D1131A [libj9jit29.so+0x19131a])
omrsig_protect+0x3e8 (0x000003FFA54B2BA8 [libj9prt29.so+0x32ba8])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x3fc (0x000003FF98D0E9E4 [libj9jit29.so+0x18e9e4])
_ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x13c (0x000003FF98D0EF74 [libj9jit29.so+0x18ef74])
_ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF98D0D9E6 [libj9jit29.so+0x18d9e6])
_ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF98D0DEEC [libj9jit29.so+0x18deec])
_Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF98D0DF84 [libj9jit29.so+0x18df84])
omrsig_protect+0x3e8 (0x000003FFA54B2BA8 [libj9prt29.so+0x32ba8])
_Z21compilationThreadProcPv+0x196 (0x000003FF98D0E3C6 [libj9jit29.so+0x18e3c6])
thread_wrapper+0xf6 (0x000003FFA5408A5E [libj9thr29.so+0x8a5e])
start_thread+0xea (0x000003FFA5E88312 [libpthread.so.0+0x8312])
 (0x000003FFA5D0E232 [libc.so.6+0x10e232])
---------------------------------------

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/22
jdk_util_1
java/util/concurrent/FutureTask/NegativeTimeout.java

There is a core / javacore / Snap in .\aqa-tests\TKG\output_16650183664248\jdk_util_1\work\java\util\concurrent\FutureTask\ExplicitSet
Javacore shows gpf with VM flags:000000000005FFFF
1XHEXCPMODULE Compiling method: java/lang/Thread.genThreadName()Ljava/lang/String;

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/22/

22:57:54  ACTION: main -- Error. Agent communication error: java.net.SocketException: Broken pipe; check console log for any additional details
22:57:54  REASON: Assumed action based on file name: run main NegativeTimeout 
22:57:54  TIME:   0.002 seconds
22:57:54  messages:
22:57:54  command: main NegativeTimeout
22:57:54  reason: Assumed action based on file name: run main NegativeTimeout 
22:57:54  Mode: agentvm
22:57:54  Agent id: 4
22:57:54  elapsed time (seconds): 0.002
22:57:54  configuration:
22:57:54  Boot Layer
22:57:54    class path: /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/javatest.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/jtreg.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/junit.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/hamcrest.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/testng.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/jcommander.jar 
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/guice.jar 
22:57:54    patch:      java.base /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/patches/java.base
22:57:54  
22:57:54  Test Layer
22:57:54    class path: /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/classes/1/java/util/concurrent/FutureTask/NegativeTimeout.d
22:57:54                /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask
22:57:54  
22:57:54  rerun:
22:57:54  cd /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/scratch/0 && \
22:57:54  DISPLAY=:0 \
22:57:54  HOME=/home/jenkins \
22:57:54  LANG=en_US.UTF-8 \
22:57:54  PATH=/bin:/usr/bin:/usr/sbin \
22:57:54      /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image/bin/java \
22:57:54          -Dtest.vm.opts='-ea -esa -Xmx1540m -Xdump:system:none -Xdump:heap:none -Xdump:system:events=gpf+abort+traceassert+corruptcache -XX:-UseCompressedOops' \
22:57:54          -Dtest.tool.vm.opts='-J-ea -J-esa -J-Xmx1540m -J-Xdump:system:none -J-Xdump:heap:none -J-Xdump:system:events=gpf+abort+traceassert+corruptcache -J-XX:-UseCompressedOops' \
22:57:54          -Dtest.compiler.opts= \
22:57:54          -Dtest.java.opts= \
22:57:54          -Dtest.jdk=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
22:57:54          -Dcompile.jdk=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/j2sdk-image \
22:57:54          -Dtest.timeout.factor=8.0 \
22:57:54          -Dtest.nativepath=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/openjdkbinary/openjdk-test-image/jdk/jtreg/native \
22:57:54          -Dtest.root=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk \
22:57:54          -Dtest.name=java/util/concurrent/FutureTask/NegativeTimeout.java \
22:57:54          -Dtest.file=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask/NegativeTimeout.java \
22:57:54          -Dtest.src=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask \
22:57:54          -Dtest.src.path=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask \
22:57:54          -Dtest.classes=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/classes/1/java/util/concurrent/FutureTask/NegativeTimeout.d \
22:57:54          -Dtest.class.path=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/classes/1/java/util/concurrent/FutureTask/NegativeTimeout.d \
22:57:54          -Dtest.class.path.prefix=/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/classes/1/java/util/concurrent/FutureTask/NegativeTimeout.d:/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask \
22:57:54          -classpath /home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/TKG/output_16650183664248/jdk_util_1/work/classes/1/java/util/concurrent/FutureTask/NegativeTimeout.d:/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/aqa-tests/openjdk/openjdk-jdk/test/jdk/java/util/concurrent/FutureTask:/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/javatest.jar:/home/jenkins/workspace/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/jvmtest/openjdk/jtreg/lib/jtreg.jar \
22:57:54          NegativeTimeout
22:57:54  
22:57:54  TEST RESULT: Error. Agent communication error: java.net.SocketException: Broken pipe; check console log for any additional details
22:57:54  --------------------------------------------------

[pshipton]

I suspect there are number of dups of this issue (timeouts) that weren't identified as this issue.
Here is an older one
#15953

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/18
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/18/functional_test_output.tar.gz

.Running  :testCacheCreationListingDestroying02  (Test suite : TestOptionsCacheDir)
#0: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xa96ec6) [0x3ff97a16ec6]
#1: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xaa44d8) [0x3ff97a244d8]
#2: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x175084) [0x3ff970f5084]
#3: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x31c6e) [0x3ff9cd31c6e]
#4: [0x3ff7aef9f38]
#5: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so(+0x84cd8) [0x3ff9ce84cd8]
#6: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0xbe192a) [0x3ff97b6192a]
#7: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x556b76) [0x3ff974d6b76]
#8: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x556d6a) [0x3ff974d6d6a]
#9: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x55344c) [0x3ff974d344c]
#10: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x552e3a) [0x3ff974d2e3a]
#11: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x561cdc) [0x3ff974e1cdc]
#12: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x562550) [0x3ff974e2550]
#13: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18c308) [0x3ff9710c308]
#14: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18fdc6) [0x3ff9710fdc6]
#15: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x190a26) [0x3ff97110a26]
#16: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x19169a) [0x3ff9711169a]
#17: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ba8) [0x3ff9cd32ba8]
#18: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18ed64) [0x3ff9710ed64]
#19: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18f2f4) [0x3ff9710f2f4]
#20: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18dd66) [0x3ff9710dd66]
#21: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e26c) [0x3ff9710e26c]
#22: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e304) [0x3ff9710e304]
#23: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9prt29.so(+0x32ba8) [0x3ff9cd32ba8]
#24: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9jit29.so(+0x18e746) [0x3ff9710e746]
#25: /home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9thr29.so(+0x8a5e) [0x3ff9cc88a5e]
#26: /lib/s390x-linux-gnu/libpthread.so.0(+0x7e66) [0x3ff9d807e66]
#27: /lib/s390x-linux-gnu/libc.so.6(+0xfcbe6) [0x3ff9d67cbe6]
#28: [(nil)]
Unhandled exception
Type=Segmentation error vmState=0x0005ffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FF9CE444B0 Handler2=000003FF9CD31A50 InaccessibleAddress=0000000000018000
gpr0=0000000000000000 gpr1=000003FF9D0718BC gpr2=0000000000018000 gpr3=0000000000F5AC90
gpr4=000003FF9CE84CD0 gpr5=0000000000000008 gpr6=0000000000000008 gpr7=0000000000080000
gpr8=0000000000F5AC90 gpr9=000003FF414B95E0 gpr10=0000000000F0DD00 gpr11=0000000000018000
gpr12=000003FF97E3A000 gpr13=000003FF9D02E218 gpr14=000003FF9CE84CD0 gpr15=000003FF7AEFA3C0
psw=000003FF9CE84CD8 mask=0705200180000000 fpc=00080000 bea=000003FF9CE841EC
fpr0 000003ff74000c4c (f: 1946160256.000000, d: 2.171763e-311)
fpr1 000003ff97c0cebe (f: 2545995520.000000, d: 2.172060e-311)
fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr3 bd72c56463f0b1e0 (f: 1676718592.000000, d: -1.067011e-12)
fpr4 0000000000000007 (f: 7.000000, d: 3.458460e-323)
fpr5 bfcef2b8887c4b05 (f: 2289847040.000000, d: -2.417823e-01)
fpr6 000003ff740016c0 (f: 1946162944.000000, d: 2.171763e-311)
fpr7 3e1c4757a5e13871 (f: 2783000576.000000, d: 1.646036e-09)
fpr8 000003ff8af14088 (f: 2331066624.000000, d: 2.171953e-311)
fpr9 000002aa1f8bf8b0 (f: 529266880.000000, d: 1.447463e-311)
fpr10 000003ff7841d2c0 (f: 2017579776.000000, d: 2.171799e-311)
fpr11 000003ffc5677e10 (f: 3311894016.000000, d: 2.172438e-311)
fpr12 0005ead07f8460c6 (f: 2139382016.000000, d: 8.228940e-309)
fpr13 000002aa1dd22938 (f: 500312384.000000, d: 1.447448e-311)
fpr14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/home/jenkins/workspace/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9vm29.so
Module_base_address=000003FF9CE00000

Method_being_compiled=java/lang/Thread.genThreadName()Ljava/lang/String;
Target=2_90_20221011_91 (Linux 5.4.0-124-generic)
CPU=s390x (4 logical CPUs) (0x1f5906000 RAM)
----------- Stack Backtrace -----------
resolveStaticFieldRefInto+0x118 (0x000003FF9CE84CD8 [libj9vm29.so+0x84cd8])
jitCTResolveStaticFieldRefWithMethod+0x62 (0x000003FF97B6192A [libj9jit29.so+0xbe192a])
_ZN30TR_RelocationRecordDataAddress15findDataAddressEP20TR_RelocationRuntimeP19TR_RelocationTarget+0x106 (0x000003FF974D6B76 [libj9jit29.so+0x556b76])
_ZN30TR_RelocationRecordDataAddress15applyRelocationEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0x22 (0x000003FF974D6D6A [libj9jit29.so+0x556d6a])
_ZN19TR_RelocationRecord27applyRelocationAtAllOffsetsEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0x30c (0x000003FF974D344C [libj9jit29.so+0x55344c])
_ZN24TR_RelocationRecordGroup16applyRelocationsEP20TR_RelocationRuntimeP19TR_RelocationTargetPh+0xf2 (0x000003FF974D2E3A [libj9jit29.so+0x552e3a])
_ZN20TR_RelocationRuntime22relocateAOTCodeAndDataEPhS0_S0_S0_+0x2e4 (0x000003FF974E1CDC [libj9jit29.so+0x561cdc])
_ZN20TR_RelocationRuntime29prepareRelocateAOTCodeAndDataEP10J9VMThreadP11TR_FrontEndPN2TR9CodeCacheEPK20J9JITDataCacheHeaderP8J9MethodbPNS4_7OptionsEPNS4_11CompilationEP17TR_ResolvedMethodPh+0x3d8 (0x000003FF974E2550 [libj9jit29.so+0x562550])
_ZN2TR28CompilationInfoPerThreadBase22installAotCachedMethodEP10J9VMThreadPKvP8J9MethodP11TR_FrontEndPNS_7OptionsEP17TR_ResolvedMethodP21TR_MethodToBeCompiledPNS_11CompilationE+0x98 (0x000003FF9710C308 [libj9jit29.so+0x18c308])
_ZN2TR28CompilationInfoPerThreadBase14performAOTLoadEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodP11TR_J9VMBaseP8J9Method+0xb6 (0x000003FF9710FDC6 [libj9jit29.so+0x18fdc6])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x916 (0x000003FF97110A26 [libj9jit29.so+0x190a26])
_ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x392 (0x000003FF9711169A [libj9jit29.so+0x19169a])
omrsig_protect+0x3e8 (0x000003FF9CD32BA8 [libj9prt29.so+0x32ba8])
_ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x3fc (0x000003FF9710ED64 [libj9jit29.so+0x18ed64])
_ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x13c (0x000003FF9710F2F4 [libj9jit29.so+0x18f2f4])
_ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3ee (0x000003FF9710DD66 [libj9jit29.so+0x18dd66])
_ZN2TR24CompilationInfoPerThread3runEv+0xac (0x000003FF9710E26C [libj9jit29.so+0x18e26c])
_Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF9710E304 [libj9jit29.so+0x18e304])
omrsig_protect+0x3e8 (0x000003FF9CD32BA8 [libj9prt29.so+0x32ba8])
_Z21compilationThreadProcPv+0x196 (0x000003FF9710E746 [libj9jit29.so+0x18e746])
thread_wrapper+0xf6 (0x000003FF9CC88A5E [libj9thr29.so+0x8a5e])
start_thread+0xd6 (0x000003FF9D807E66 [libpthread.so.0+0x7e66])
 (0x000003FF9D67CBE6 [libc.so.6+0xfcbe6])
 (0x0000000000000000 [<unknown>+0x0])
---------------------------------------

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/26/
jdk_util_0
java/util/concurrent/FutureTask/Throw.java timed out

It's not actually running a test but doing cleanup before running the test.

core, javacore under .\aqa-tests\TKG\output_16655465638227\jdk_util_0\work\java\util\concurrent\FutureTask\ExplicitSet\ in https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/26/openjdk_test_output.tar.gz

4XENATIVESTACK               resolveStaticFieldRefInto+0x118 (0x000003FF87E04CD8 [libj9vm29.so+0x84cd8])
4XENATIVESTACK               jitCTResolveStaticFieldRefWithMethod+0x62 (0x000003FF874E192A [libj9jit29.so+0xbe192a])
4XENATIVESTACK               _ZN30TR_RelocationRecordDataAddress15findDataAddressEP20TR_RelocationRuntimeP19TR_RelocationTarget+0x106 (0x000003FF86E56B76 [libj9jit29.so+0x556b76])

3XMTHREADINFO      "main" J9VMThread:0x0000000000069700, omrthread_t:0x000003FF8800A0E0, java/lang/Thread:0x000000003404EE10, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x2, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x000000003404BB88)
3XMTHREADINFO1            (native thread ID:0x859F, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x000003FF8C936000, to:0x000003FF8C976000, size:0x40000)
3XMCPUTIME               CPU usage total: 1.432726781 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=1471408 (0x1673B0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Thread.getThreads(Native Method)
4XESTACKTRACE                at java/lang/Thread.getAllThreads(Thread.java:2701(Compiled Code))
4XESTACKTRACE                at java/lang/ThreadGroup.activeCount(ThreadGroup.java:377)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/MainActionHelper$AgentVMThreadGroup.liveThreads(MainActionHelper.java:422)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/MainActionHelper$AgentVMThreadGroup.cleanup(MainActionHelper.java:380)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/MainActionHelper$AgentVMThreadGroup.access$000(MainActionHelper.java:345)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/MainActionHelper.runClass(MainActionHelper.java:214)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/AgentServer.doMain(AgentServer.java:306)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/AgentServer.run(AgentServer.java:233)
4XESTACKTRACE                at com/sun/javatest/regtest/agent/AgentServer.main(AgentServer.java:70)

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/19
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/19/functional_test_output.tar.gz

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/23
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/23/functional_test_output.tar.gz

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/24
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/24/functional_test_output.tar.gz

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_2/32
SharedClasses.SCM01.MultiThreadMultiCL_1

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_2/32/system_test_output.tar.gz

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/33
java/lang/ClassLoader/Assert.java.Assert
java/lang/ProcessBuilder/Basic.java#id0.Basic_id0
java/lang/ProcessBuilder/Basic.java#id1.Basic_id1
java/lang/RuntimeTests/exec/ExecWithInput.java.ExecWithInput
java/lang/RuntimeTests/shutdown/Basic.java.Basic
java/lang/RuntimeTests/shutdown/ShutdownInterruptedMain.java.ShutdownInterruptedMain
java/lang/String/CompactString/Contains.java.Contains
java/lang/System/Logger/interface/LoggerInterfaceTest.java.LoggerInterfaceTest
java/lang/System/SecurityManagerWarnings.java.SecurityManagerWarnings
java/lang/ThreadLocal/MemoryLeak.java.MemoryLeak
java/lang/ThreadLocal/ThreadLocalSupplierTest.java.ThreadLocalSupplierTest
java/lang/Throwable/ChainedExceptions.java.ChainedExceptions
java/lang/invoke/VarHandles/VarHandleTestReflection.java.VarHandleTestReflection
jdk/internal/misc/Unsafe/CopyMemory.java.CopyMemory
jdk/modules/incubator/ImageModules.java.ImageModules
sun/misc/JarIndex/JarIndexMergeTest.java.JarIndexMergeTest

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/33/openjdk_test_output.tar.gz

[r30shah]

@dchopra001 If you are unable to narrow it down to the failing change set and getting the AOT compilation logs from the failure are the next unnecessary step, then let's see if there is a way to may be manually executing the steps for the job failure.

[dchopra001]

I did finally manage to get an AOT log for this. Due to the way the files are moved around post crash by the test harness the rtlog isn't there. I think I should still be able to get some information out of this though. I'm looking through the results right now. Hopefully I'll have an update soon.

[dchopra001]

This problem happens because of a countDigits transformation by idiomRecognition:

n564n     BBStart <block_65> (freq 9994)                                                      [     0x3ff008c7000] bci=[5,23,-] rc=0 vc=689 vn=- li=- udi=- nc=0
n1091n    istore  v1len<auto slot 4>[#516  Auto] [flags 0x3 0x0 ]                             [     0x3ff009714c0] bci=[5,34,-] rc=0 vc=0 vn=- li=- udi=- nc=1
n1090n      iadd                                                                              [     0x3ff00971470] bci=[5,34,-] rc=1 vc=0 vn=- li=- udi=- nc=2
n1086n        iload  v1len<auto slot 4>[#516  Auto] [flags 0x3 0x0 ] (cannotOverflow )        [     0x3ff00971330] bci=[5,34,-] rc=1 vc=689 vn=- li=- udi=81 nc=0 flg=0x1000
n1089n        countDigits                                                                     [     0x3ff00971420] bci=[5,37,-] rc=1 vc=0 vn=- li=- udi=- nc=2
n1087n          iload  i<auto slot 6>[#517  Auto] [flags 0x3 0x0 ] (cannotOverflow )          [     0x3ff00971380] bci=[5,23,-] rc=1 vc=689 vn=- li=- udi=82 nc=0 flg=0x1000
n1088n          loadaddr       0x3ff7d7db618[#572  Static] [flags 0x10307 0x0 ]               [     0x3ff009713d0] bci=[5,23,-] rc=1 vc=0 vn=- li=- udi=- nc=0
n1092n    goto --> block_64 BBStart at n513n                                                  [     0x3ff00971510] bci=[5,34,-] rc=0 vc=0 vn=- li=- udi=- nc=0
n989n     BBEnd </block_65> =====

The loadaddr at n1088n is a table that does not have a proper relocation record and is instead stored as a TR_DataAddress

Here's the table:

openj9/runtime/compiler/optimizer/IdiomTransformations.cpp

Lines 10452 to 10474 in 1af0a29

	static const int64_t digit10Table[] =
	{
	-10L, // 0
	-100L, // 1
	-1000L, // 2
	-10000L, // 3
	-100000L, // 4
	-1000000L, // 5
	-10000000L, // 6
	-100000000L, // 7
	-1000000000L // 8 (32-bit)
	#ifdef TR_TARGET_64BIT
	,-10000000000L, // 9
	-100000000000L, // 10
	-1000000000000L, // 11
	-10000000000000L, // 12
	-100000000000000L, // 13
	-1000000000000000L, // 14
	-10000000000000000L, // 15
	-100000000000000000L,// 16
	-1000000000000000000L// 17 (64-bit)
	#endif
	};

I'm not sure why we've only started seeing this in Java 19 as this code looks to have been around for a while. The fix will probably be either adding a new relo record or disabling this transformation for relocatable compiles (I'm not sure if the perf gains are meaningful here). I've launched a grinder to see if this fails when we disable the transformation.

I noticed that @hzongaro modified code in this area earlier this year. @hzongaro any chance this issue could be related to your PR?

[hzongaro]

Dhruv @dchopra001, I'll take a look

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/27
testSCCacheManagement_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/27/functional_test_output.tar.gz

[JasonFengJ9]

Similar segmentation error seen on RTC 148250

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/28
Running javac to compile the test material.

[pshipton]

None of the extended system testing got started.
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_0/35
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/35
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_1/35

[dchopra001]

@jdmpapin mentioned in an offline discussion that it should be reasonable okay to disable this transformations for relocatable compiles.

I was curious as to why this was only showing up now in Java 19. @hzongaro took at the logs for Java 17 and 19 earlier and it seems like it might just be luck that we're seeing this be so prevalent in Java 19 now.

[dchopra001]

I'm testing out a fix for this in the internal builds and will open a PR once the grinder passes.

[vij-singh]

Thank you Dhruv

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/29
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/30
https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_1/31

[dchopra001]

My test grinder with the fix failed. It seems to be a similar issue. I'm not sure why yet so have launched another grinder to collect trace logs again.

[dchopra001]

I wasn't able to get the logs when the failure occurred due to an infra failure:

java.io.IOException: Tried to load head FlowNodes for execution Owner

Trying again.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/43

[dchopra001]

I've opened PRs here:
#16273
eclipse/omr#6804

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_sanity.openjdk_s390x_linux_Nightly/45
java/util/concurrent/FutureTask/Throw.java

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.system_s390x_linux_Nightly_testList_0/44
SharedClasses.SCM01.MultiThreadMultiCL_0

[tajila]

@dchopra001 Do you think this will be resolved within 2 weeks?

[dchopra001]

Yes, the PRs should be approved+merged soon.

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_extended.functional_s390x_linux_Nightly_testList_0/41
testSCCacheManagement_0