Transient fatal error when building spring-graalvm-native with GraalVM 20.2.0

[sdeleuze]

We see sometimes on our CI such kind of transient error on various samples with with GraalVM 20.2.0-dev:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f992032fafb, pid=1073, tid=0x00007f98521fc700
#
# JRE version: OpenJDK Runtime Environment (8.0_262-b10) (build 1.8.0_262-b10)
# Java VM: OpenJDK 64-Bit Server VM GraalVM 20.2.0-dev (25.262-b10-jvmci-20.2-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x6d0afb]  JfrEventClassTransformer::on_klass_creation(InstanceKlass*&, ClassFileParser&, Thread*)+0x92b
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /tmp/build/80754af9/spring-graalvm-native/spring-graalvm-native-samples/webflux-netty/target/native-image/hs_err_pid1073.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/oracle/graal/issues
#
Error: Image build request failed with exit status 134
com.oracle.svm.driver.NativeImage$NativeImageError: Image build request failed with exit status 134
	at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1558)
	at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1308)
	at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1269)
	at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1228)
real	0m2.883s
user	0m7.934s
sys	0m0.527s
FAILURE: an error occurred when compiling the native-image

It happens with Java 8 (https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java8-key-samples-build/builds/476 or https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java8-key-samples-build/builds/478) and also on Java 11 (https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java11-key-samples-build/builds/420).

We use up to date builds of release/graal-vm/20.2 with openjdk-8u262+10-jvmci-20.2-b03 and labsjdk-ce-11.0.8+10-jvmci-20.2-b03.

Despite the message I don't see any hs_err_pid1073.log error file on our CI machine.
I have enabled core dumps on the CI in order to try to provide more information.

Please tag this issue with the spring label.

[eginez]

Thanks for the report @sdeleuze.
The builds above are using:
Graal commit: release/graal-vm/20.2
with openjdk-8u262+10-jvmci-20.2-b03
and labsjdk-ce-11.0.8+10-jvmci-20.2-b03

[eginez]

@dougxc do you think this is possibly to the problem observed in #2732 ?

[dougxc]

Unlikely - the deadlock in #2732 should never result in a crash.
This looks related to https://bugs.openjdk.java.net/browse/JDK-8245283 and/or https://bugs.openjdk.java.net/browse/JDK-8232997.
Either way, it's hard to diagnose further without the hs-err file.

[dougxc]

Despite the description above, I think the crashes are only happening in Java 8. The crash in https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java11-key-samples-build/builds/420 is:

# JRE version: OpenJDK Runtime Environment (8.0_262-b10) (build 1.8.0_262-b10)
# Java VM: OpenJDK 64-Bit Server VM GraalVM 20.2.0-dev (25.262-b10-jvmci-20.2-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x6cda34]  position_stream_after_cp(ClassFileStream*)+0x44

This is the first OpenJDK 8 update release to include a backport of JFR. It's possible that the JFR bug fixes mentioned above were not included in this backport (@adinn @zakkak ).

[adinn]

@dougxc JDK-8245283 has not been back-ported to jdk11u so that might cause a failure on jdk11u. However, it's not going to be the cause of the problem on to jdk8u. The patch for that bug skips over a JVM_Constant_Dynamic bytecode but that was only added to the byecode spec from release 11.

This may perhaps be the same error as was seen in JDK-8232997. The error was resolved by changes made for a related issue, JDK-8230400 (for which DK-8232997 was marked as a duplicate). The latter had a two part fix.

The first part avoids a potential error when comparing old and new methods in the case where the method arrays may not have the same size. The change was backported to jdk11u but the method comparison still present in jdk8u. This may well be what is causing the SEGV. However, in order for this to be the cause of the crash I think the user has to have installed their own agent in the runtime. Is that the case here?

The second part of the fix was pushed as a fix to JDK-8233111. It does not appear to have been backported to either jdk11u or jdk8u. I'm not sure whether it could be the cause of the SEGV but I suppose it is possible. It certainly seems to relate to running a Java JIT in the JVM.

[dougxc]

There is an SVM agent in play although I don't know what it does (@peter-hofer ?):
https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java8-key-samples-build/builds/476#L5ec26892:688

[adinn]

Ah yes. That agent is responsible for adding code to manage class and instance init, tracking init at build time and maybe deferring it to runtime. It can add a new <clinit> method. That might be enough to cause the problem.

[peter-hofer]

There is an SVM agent in play although I don't know what it does (@peter-hofer ?):
https://ci.spring.io/teams/spring-graalvm-native/pipelines/spring-graalvm-native/jobs/java8-key-samples-build/builds/476#L5ec26892:688

The agent that I think you're referring to (-javaagent:/opt/java/jre/lib/svm/builder/svm.jar) is NativeImageBytecodeInstrumentationAgent, perhaps @vjovanov can comment on this.

[vjovanov]

As @adinn said the agent is there to add the <clinit> method and instrumentation. First, I would not use that in CI as this is only for debugging by the users. Second, we are narrowing down -H:+TraceClassInitialization to use a regular expression. When we do that, only a few classes will be instrumented and the impact should be minimal.

I would advise removing -H:+TraceClassInitialization from the CI (it will anyhow not be possible soon) and see if the issue prevails.

[sdeleuze]

@vjovanov I reproduced the issue locally (without -H:+TraceClassInitialization BTW) and sent to you the hs_err_pid1073.log file.

[vjovanov]

hs_err_pid26455.log