"dump java core" is hanging process on RHEL 9.1/9.2 using Semeru OpenJDK 17

[mkarrahcl]

Java -version output

openjdk version "17.0.4.1" 2022-08-12
IBM Semeru Runtime Open Edition 17.0.4.1 (build 17.0.4.1+1)
Eclipse OpenJ9 VM 17.0.4.1 (build openj9-0.33.1, JRE 17 Windows 10 amd64-64-Bit Compressed References 20220812_237 (JIT enabled, AOT enabled)
OpenJ9 - 1d9d168
OMR - b58aa2708
JCL - 1f4d354e654 based on jdk-17.0.4.1+1)

Summary of problem

We are using Semeru OpenJDK 17 in our application and when we try to do "dump java core", the process is hanging on RHEL 9.1 or 9.2. The same Java works fine on CentOS/Windows/AIX. I am providing the call stack of the hung process.

Thread 1 (Thread 0x7f6c01fd5740 (LWP 3115552) "http"):
#0 0x00007f6c0282871f in poll () from /lib64/libc.so.6
#1 0x00007f6be578c9c0 in sem_timedwait_r () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#2 0x00007f6be578e2e9 in omrintrospect_threads_nextDo () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#3 0x00007f6be57901ef in omrsig_protect () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#4 0x00007f6be5609b1d in JavaCoreDumpWriter::writeThreadsWithNativeStacks() () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#5 0x00007f6be560a07d in protectedWriteThreadsWithNativeStacks () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#6 0x00007f6be57901ef in omrsig_protect () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#7 0x00007f6be56066b1 in JavaCoreDumpWriter::writeThreadSection() () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#8 0x00007f6be56017dd in protectedWriteSection () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#9 0x00007f6be57901ef in omrsig_protect () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#10 0x00007f6be5602bb5 in JavaCoreDumpWriter::JavaCoreDumpWriter(char const*, J9RASdumpContext*, J9RASdumpAgent*) () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#11 0x00007f6be560cb3c in runJavadump () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#12 0x00007f6be55f516f in doJavaDump () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#13 0x00007f6be55f47c5 in protectedDumpFunction () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#14 0x00007f6be57901ef in omrsig_protect () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9prt29.so
#15 0x00007f6be55f7f40 in runDumpFunction () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#16 0x00007f6be55f80bd in runDumpAgent () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#17 0x00007f6be55f8784 in createAndRunOneOffDumpAgent () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#18 0x00007f6be560ec85 in triggerOneOffDump () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9dmp29.so
#19 0x00007f6be44267b5 in Java_com_ibm_jvm_Dump_JavaDumpImpl () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libjclse29.so
#20 0x00007f6c0021196a in ffi_call_unix64 () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#21 0x00007f6c00210b02 in ffi_call_int () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#22 0x00007f6c0010ea18 in VM_BytecodeInterpreterCompressed::run(J9VMThread*) () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#23 0x00007f6c000fee7d in bytecodeLoopCompressed () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#24 0x00007f6c0019ffc2 in c_cInterpreter () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#25 0x00007f6c0008db87 in runCallInMethod () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#26 0x00007f6c000b122b in gpCheckCallin () from /opt/hcl/domino/notes/latest/linux/jvm/lib/default/libj9vm29.so
#27 0x00007f6c04c2860c in JNIEnv_::CallStaticVoidMethod(_jclass*, _jmethodID*, ...) () from /opt/hcl/domino/notes/latest/linux/libnotes.so
#28 0x00007f6c04c34947 in JavaDumpClass::Dump() () from /opt/hcl/domino/notes/latest/linux/libnotes.so
#29 0x00007f6c04c3499a in JavaDump () from /opt/hcl/domino/notes/latest/linux/libnotes.so
#30 0x00007f6c066dfedf in HTServerProcessConsoleCommand (cmdArgs=cmdArgs@entry=0x7ffcfdccd020, numArgs=3) at htstart.cpp:153
#31 0x00007f6c066c9192 in ProcessCommand (commandBuffer=commandBuffer@entry=0x7ffcfdccd980 "dump") at htmain.cpp:525
#32 0x00007f6c066c9961 in HTAddInMain (hModule=, argc=1, argv=0x1b7edb8) at htmain.cpp:337
#33 0x00000000004011df in AddInMain (hModule=, argc=, argv=) at htmainexe.cpp:26
#34 0x0000000000401222 in NotesMain ()
 #35 0x0000000000401329 in main ()

Has any one seen this issue? Any known workaround like RHEL9 configuration or JVM settings?
Any help/pointers is appreciated.

[pshipton]

@TobiAjila @keithc-ca fyi

[keithc-ca]

@mkarrahcl When you say "hang", was the process hung permanently, or did it resume after the 20 second timeout built into the dump code?

[mkarrahcl]

@keithc-ca We have upgraded to a later version of the Java 17 (i.e., 17.0.8.1+1) and this problem of hung thread is not happening anymore. I think we can close this now.

[keithc-ca]

The likely change that fixed this is eclipse/omr#6685.