JIT crash for java/lang/StringConcatHelper.stringOf(Ljava/lang/Object;)Ljava/lang/String;

[beikov]

Java -version output

openjdk version "17.0.10" 2024-01-16
IBM Semeru Runtime Open Edition 17.0.10.0 (build 17.0.10+7)
Eclipse OpenJ9 VM 17.0.10.0 (build openj9-0.43.0, JRE 17 Linux s390x-64-Bit Compressed References 20240116_630 (JIT enabled, AOT enabled)
OpenJ9 - 2c3d78b
OMR - ea8124dbc
JCL - 2aad089841f based on jdk-17.0.10+7)

Summary of problem

Running the Hibernate ORM testsuite triggers a JVM crash. Also see https://ci.hibernate.org/job/hibernate-orm-pipeline/job/wip%2F7.0/20/execution/node/66/log/

The problem can be reproduced by running ./gradlew check -PexcludeTests=**/KotlinProjectTests* -Plog-test-progress=true --stacktrace with the wip/7.0 branch of Hibernate ORM.

Diagnostic files

javacore.20240423.090841.2949679.0002.txt
jitdump.20240423.090841.2949679.0004.dmp
Snap.20240423.090841.2949679.0003.trc.txt

Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000003FFAA6C9410 Handler2=000003FFAA5B19D8 InaccessibleAddress=0000000010000000
gpr0=0000000000191C00 gpr1=0000000010000000 gpr2=0000000010000000 gpr3=000003FF89AB91FA
gpr4=000003FF883CBEF4 gpr5=00000000023A83B8 gpr6=0000000010000000 gpr7=00000000319EA0C0
gpr8=0000000010000000 gpr9=000003FF8992DE00 gpr10=000003FF8992DDF4 gpr11=0000000000318F00
gpr12=00000000EC5C95A8 gpr13=000000000006D200 gpr14=000003FF8992DE84 gpr15=000003FFAB2F1BA0
psw=000003FF89AB9226 mask=0705200180000000 fpc=00080000 bea=000003FF8992DE82
fpr0 3f40000000000000 (f: 0.000000, d: 4.882812e-04)
fpr1 000003ffaaf9f968 (f: 2868508928.000000, d: 2.172219e-311)
fpr2 00000000001d28d8 (f: 1911000.000000, d: 9.441594e-318)
fpr3 0000000000000007 (f: 7.000000, d: 3.458460e-323)
fpr4 000003ffa4000a78 (f: 2751465984.000000, d: 2.172161e-311)
fpr5 000003ffa4000f70 (f: 2751467264.000000, d: 2.172161e-311)
fpr6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr7 9ae13b28b86404a3 (f: 3093562624.000000, d: -3.322044e-179)
fpr8 0000000030678b80 (f: 812092288.000000, d: 4.012269e-315)
fpr9 0000000002f39458 (f: 49517656.000000, d: 2.446497e-316)
fpr10 00000000306783a8 (f: 812090304.000000, d: 4.012259e-315)
fpr11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
fpr12 000616c33b5900b9 (f: 995688640.000000, d: 8.467681e-309)
fpr13 000003fecc016a08 (f: 3422644736.000000, d: 2.170371e-311)
fpr14 0000000000040000 (f: 262144.000000, d: 1.295163e-318)
fpr15 000003fecc031588 (f: 3422754304.000000, d: 2.170371e-311)

Compiled_method=java/lang/StringConcatHelper.stringOf(Ljava/lang/Object;)Ljava/lang/String;
Target=2_90_20240116_630 (Linux 4.18.0-425.3.1.el8.s390x)
CPU=s390x (8 logical CPUs) (0x3ddbc6000 RAM)
----------- Stack Backtrace -----------
 (0x000003FF89AB9226 [<unknown>+0x0])
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2024/04/23 09:08:41 - please wait.
JVMDUMP032I JVM requested System dump using '/home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/core.20240423.090841.2949679.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.2950261.  Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.

JVMPORT049I The core file created by child process with pid = 2950261 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" to find where the core file is written and ensure that program does not truncate it.

JVMDUMP012E Error in System dump: /home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/core.20240423.090841.2949679.0001.dmp
JVMDUMP032I JVM requested Java dump using '/home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/javacore.20240423.090841.2949679.0002.txt' in response to an event
JVMDUMP010I Java dump written to /home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/javacore.20240423.090841.2949679.0002.txt
JVMDUMP032I JVM requested Snap dump using '/home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/Snap.20240423.090841.2949679.0003.trc' in response to an event
JVMDUMP010I Snap dump written to /home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/Snap.20240423.090841.2949679.0003.trc
JVMDUMP032I JVM requested JIT dump using '/home/linux1/workspace/hibernate-orm-pipeline_wip_7.0/hibernate-core/target/OOM-dump/jitdump.20240423.090841.2949679.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'Test worker' thread 0x000000000006D200
JVMDUMP053I JIT dump is recompiling java/lang/StringConcatHelper.stringOf(Ljava/lang/Object;)Ljava/lang/String;
JVMDUMP013I Processed dump event "gpf", detail "".

[babsingh]

fyi @r30shah

[r30shah]

@beikov Looking at the segmentation fault, I see that the failure is happening while executing a JIT compiled method. From the output, it seems like it has generated diagnostic files (core dump, javacore, jitdump, etc). If the test has archived core dump or it is still available, would it be possible for us to checkout that ? It would help us with diagnosing problem. NVM, saw that it failed with producing core-dump

[pshipton]

Pls give the 17.0.10 Milestone 2 build a try to see if the problem still occurs.

https://github.com/ibmruntimes/semeru17-binaries/releases/tag/jdk-17.0.11%2B7_openj9-0.44.0-m2

[beikov]

No more JIT error with that milestone, but now we have a stackoverflow that makes no sense. The JIT seems to produce wrong code: https://ci.hibernate.org/job/hibernate-orm-pipeline/job/wip%2F7.0/21/testReport/junit/org.hibernate.orm.test.bootstrap.jpa/PersistenceUnitOverridesTests/Build___s390x_h2___Test___testCfgXmlBaseline/

It reports this SO which makes no sense:

	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:453)
	at org.hibernate.boot.model.naming.Identifier.render(Identifier.java:222)
	at org.hibernate.boot.model.naming.Identifier.toString(Identifier.java:232)
	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:453)
	at org.hibernate.boot.model.naming.Identifier.render(Identifier.java:222)
	at org.hibernate.boot.model.naming.Identifier.toString(Identifier.java:232)
	at java.base/java.lang.String.valueOf(String.java:4988)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
	at org.hibernate.boot.model.relational.QualifiedNameParser$NameParts.<init>(QualifiedNameParser.java:50)

The methods in Identifier do not call this.toString(). The same thing works fine on x86 with the Hotspot JVM.

	public String render() {
		return isQuoted
				? '`' + getText() + '`'
				: getText();
	}

	@Override
	public String toString() {
		return render();
	}

[r30shah]

It may be the same issue so upgrading the build did not work. Looking at the method StringConcatHelper.toString, I see that it will call Object.toString() method, that is what Identifier.toString() call is.
Given that you have a fair success in getting this to fail, I will try to reproduce this and see if I can get the core-dump.

[r30shah]

I can reproduce this on my personal VM and now I do have a core-dump to analyze.

Following is the failing stack,

{java/lang/StringConcatHelper.stringOf} JIT
{java/lang/invoke/LambdaForm$DMH/0x00000000841cc320.invokeStatic} JIT 
{java/lang/invoke/LambdaForm$MH/0x00000000c00b4410.invoke} JIT
{java/lang/invoke/LambdaForm$MH/0x000000008479d7e0.linkToTargetMethod} JIT 
{org/hibernate/persister/entity/AbstractEntityPersister.internalInitSubclassPropertyAliasesMap} INT
{org/hibernate/persister/entity/AbstractEntityPersister.internalInitSubclassPropertyAliasesMap} JIT

Looking at the disassembly of StringConcatHelper.stringOf, we end up in this method with corrupted object (Instead of the Object, it points to an address of another JIT compiled method. Chasing down the parameter, something messed up in the last / second last method as core-dump complained about corrupt stack and I can not get proper object values to check. Looking at the code for internalInitSubclassPropertyAliasesMap , it seems to be failing while calling a code for String concatenation at [1]. I will see if this only appears to be seen on Z or I can reproduce it on X as well. Will post update on this issue.

[1]. https://github.com/hibernate/hibernate-orm/blob/2dddf6c3e24c1c9d480fadf8e055f4cac5f26766/hibernate-core/src/main/java/org/hibernate/persister/entity/AbstractEntityPersister.java#L6796C1-L6796C2

[beikov]

Any updates on this? I see you released a new version recently, but since you didn't respond here, I wanted to know if that version is supposed to fix this problem.

[r30shah]

@beikov This is still under investigation, will post more detailed analysis on the investigation soon and will share which release would contain the fix, but to answer your question, the new version (0.44) that was released does not contain the fix - That would be more or less same driver that you tried in #19369 (comment).