Dependencies checking failed when jvmci installing code when running with openjdk master.

[XiaohongGong]

BootstrapWithSystemAssertionsRegisterPressure failed when running with openjdk master with the following reason:

Thread[JVMCI CompilerThread24,9,system]: Compilation of java.lang.StringLatin1.toUpperCase(String, byte[], Locale) @ -1 failed:`

java.lang.AssertionError: dependencies invalid Code installation failed: dependencies invalid
Failed dependency of type abstract_with_unique_concrete_subtype
context = *java.lang.CharacterData
class   = java.lang.CharacterDataLatin1
witness = java.lang.CharacterData00`



    at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(HotSpotCodeCacheProvider.java:148)
    at jdk.internal.vm.compiler/org.graalvm.compiler.core.target.Backend.createInstalledCode(Backend.java:194)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.aarch64.AArch64HotSpotBackend.createInstalledCode(AArch64HotSpotBackend.java:155)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.installMethod(CompilationTask.java:363)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.access$400(CompilationTask.java:66)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:177)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:84)
    at jdk.internal.vm.compiler/org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:178)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.runCompilation(CompilationTask.java:334)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:144)
    at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:109)
    at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.compileMethod(HotSpotJVMCIRuntime.java:696)`

Here is the running command:

/home/ent-user/build-fastdebug/images/jdk/bin/java -server -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI --module-path=/home/ent-user/ci-scripts/Graal/sdk/mxbuild/dists/jdk14/graal-sdk.jar:/home/ent-user/ci-scripts/Graal/truffle/mxbuild/dists/jdk14/truffle-api.jar --upgrade-module-path=/home/ent-user/ci-scripts/Graal/compiler/mxbuild/dists/jdk13/graal.jar:/home/ent-user/ci-scripts/Graal/compiler/mxbuild/dists/jdk13/graal-management.jar -Dgraal.PrintGraph=Network -Dgraal.CompilationWatchDogStartDelay=60.0D -esa -Dgraal.DetailedAsserts=true -Dgraal.RegisterPressure=r0,r1,r2,r3,r4,v0,v1,v2,v3 -Dgraal.CompilationFailureAction=ExitVM -XX:+UseJVMCICompiler -XX:-TieredCompilation -XX:+BootstrapJVMCI -version

[XiaohongGong]

Hi @sanzinger , this is an openjdk issue which has been fixed by:

commit 6ab00a2a9765b97b595f43c193fd4defa13acdcd Author: redestad <unknown> Date: Fri Aug 9 13:58:48 2019 +0200 8229283: StringLatin1 should consistently use CharacterDataLatin1.instance when applicable Reviewed-by: jlaskey

Thanks!

[XiaohongGong]

Although the above bootstrap issue is resolved by 8229283 , the dependency issue is still there. This jdk patch 8229283 just did some modifications to the StringLatin1.java, which will make the bootstrap issue be walked around.
The dependency issue is caused by another jdk patch: 8222446, which will change the result of code installation from dependency_failed to dependency_invalid when validate_dependencies in hotspot fails. The difference between the two kinds of result is that graal will throw BailoutException for dependency_failed while throw AssertionError for dependency_invalid. And
BailoutException is not treated as failure by default. So there will be some invalid dependency issue when running with graal.
Currently another dependency issue is when running dacapo with graal + jdk14master. Here is the command and failing log:

gate: 12 Aug 2019 09:51:08(+1:26:24) BEGIN: DaCapo_pmd:BatchMode
Migrating cache file of DACAPO from /home/ent-user/.mx/cache/DACAPO_e39957904b7e79caf4fa54f30e8e4ee74d4e9e37.jar to /home/ent-user/.mx/cache/DACAPO_e39957904b7e79caf4fa54f30e8e4ee74d4e9e37/dacapo.jar
env MX_SUBPROCESS_COMMAND_FILE=/tmp/mx_subprocess_command.wRDLyA MX_HOME=/opt/mx MX_PRIMARY_SUITE_PATH=/home/ent-user/ci-scripts/Graal/compiler MX__SUITEMODEL=sibling \
/home/ent-user/build-fastdebug/images/jdk/bin/java -server -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI --module-path=/home/ent-user/ci-scripts/Graal/sdk/mxbuild/dists/jdk14/graal-sdk.jar:/home/ent-user/ci-scripts/Graal/truffle/mxbuild/dists/jdk14/truffle-api.jar --upgrade-module-path=/home/ent-user/ci-scripts/Graal/compiler/mxbuild/dists/jdk13/graal.jar:/home/ent-user/ci-scripts/Graal/compiler/mxbuild/dists/jdk13/graal-management.jar -Dgraal.PrintGraph=Network -XX:+UseSerialGC -Xms2g -XX:-UseCompressedOops -Djava.net.preferIPv4Stack=true -Dgraal.CompilationFailureAction=ExitVM -XX:+UseJVMCICompiler -Xbatch -jar /home/ent-user/.mx/cache/DACAPO_e39957904b7e79caf4fa54f30e8e4ee74d4e9e37/dacapo.jar pmd -n 1

===== DaCapo 9.12 pmd starting =====
[Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
Thread[JVMCI CompilerThread1,9,system]: Compilation of net.sourceforge.pmd.ast.JavaParserTokenManager.jjMoveStringLiteralDfa1_0(long, long) @ -1 failed:
 java.lang.AssertionError: dependencies invalid Code installation failed: dependencies invalid
 Failed dependency of type leaf_type
 context = net.sourceforge.pmd.ast.JavaCharStream
 witness = net.sourceforge.pmd.jsp.ast.JspCharStream

 at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(HotSpotCodeCacheProvider.java:148)
 at jdk.internal.vm.compiler/org.graalvm.compiler.core.target.Backend.createInstalledCode(Backend.java:194)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.aarch64.AArch64HotSpotBackend.createInstalledCode(AArch64HotSpotBackend.java:155)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.installMethod(CompilationTask.java:363)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.access$400(CompilationTask.java:66)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:177)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(CompilationTask.java:84)
 at jdk.internal.vm.compiler/org.graalvm.compiler.core.CompilationWrapper.run(CompilationWrapper.java:178)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.CompilationTask.runCompilation(CompilationTask.java:334)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:144)
 at jdk.internal.vm.compiler/org.graalvm.compiler.hotspot.HotSpotGraalCompiler.compileMethod(HotSpotGraalCompiler.java:109)
 at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotJVMCIRuntime.compileMethod(HotSpotJVMCIRuntime.java:696)
To disable compilation failure notifications, set CompilationFailureAction to Silent (e.g., -Dgraal.CompilationFailureAction=Silent).
To print a message for a compilation failure without retrying the compilation, set CompilationFailureAction to Print (e.g., -Dgraal.CompilationFailureAction=Print).
Retrying compilation of net.sourceforge.pmd.ast.JavaParserTokenManager.jjMoveStringLiteralDfa1_0(long, long) @ -1
Could not connect to the IGV on 127.0.0.1:4445. Dumping IGV graphs in /home/ent-user/ci-scripts/graal_dumps/2019.08.12.09.51.44.558/graal_diagnostics_32172/net.sourceforge.pmd.ast.JavaParserTokenManager.jjMoveStringLiteralDfa1_0(long,_long)_@_-1
Exiting VM after retry compilation of net.sourceforge.pmd.ast.JavaParserTokenManager.jjMoveStringLiteralDfa1_0(long, long) @ -1`

[XiaohongGong]

This issue also exists on X86_64 platform.

[dougxc]

This looks a missing change from JDK-8222446:

diff -r 22fa46d5dc2e src/hotspot/share/jvmci/jvmciRuntime.cpp
--- a/src/hotspot/share/jvmci/jvmciRuntime.cpp  Thu Aug 15 15:39:43 2019 +0800
+++ b/src/hotspot/share/jvmci/jvmciRuntime.cpp  Thu Aug 15 12:11:12 2019 +0200
@@ -1325,21 +1325,13 @@
     return JVMCI::dependencies_failed;
   }

-  // Dependencies must be checked when the system dictionary changes
-  // or if we don't know whether it has changed (i.e., compile_state == NULL).
   CompileTask* task = compile_state == NULL ? NULL : compile_state->task();
   Dependencies::DepType result = dependencies->validate_dependencies(task, failure_detail);
   if (result == Dependencies::end_marker) {
     return JVMCI::ok;
   }

-  if (!Dependencies::is_klass_type(result) || compile_state == NULL) {
-    return JVMCI::dependencies_failed;
-  }
-  // The dependencies were invalid at the time of installation
-  // without any intervening modification of the system
-  // dictionary.  That means they were invalidly constructed.
-  return JVMCI::dependencies_invalid;
+  return JVMCI::dependencies_failed;
 }

 // Reports a pending exception and exits the VM.

Can you please test that locally and then open a OpenJDK bug for it to be fixed.

[dougxc]

This is actually an opportunity to clean up the no longer used JVMCI::dependencies_invalid value. Here's the complete cleanup patch:

diff -r 22fa46d5dc2e src/hotspot/share/jvmci/jvmci.hpp
--- a/src/hotspot/share/jvmci/jvmci.hpp	Thu Aug 15 15:39:43 2019 +0800
+++ b/src/hotspot/share/jvmci/jvmci.hpp	Thu Aug 15 12:22:57 2019 +0200
@@ -65,7 +65,6 @@
   enum CodeInstallResult {
      ok,
      dependencies_failed,
-     dependencies_invalid,
      cache_full,
      code_too_large
   };
diff -r 22fa46d5dc2e src/hotspot/share/jvmci/jvmciRuntime.cpp
--- a/src/hotspot/share/jvmci/jvmciRuntime.cpp	Thu Aug 15 15:39:43 2019 +0800
+++ b/src/hotspot/share/jvmci/jvmciRuntime.cpp	Thu Aug 15 12:22:57 2019 +0200
@@ -1325,21 +1325,13 @@
     return JVMCI::dependencies_failed;
   }
 
-  // Dependencies must be checked when the system dictionary changes
-  // or if we don't know whether it has changed (i.e., compile_state == NULL).
   CompileTask* task = compile_state == NULL ? NULL : compile_state->task();
   Dependencies::DepType result = dependencies->validate_dependencies(task, failure_detail);
   if (result == Dependencies::end_marker) {
     return JVMCI::ok;
   }
 
-  if (!Dependencies::is_klass_type(result) || compile_state == NULL) {
-    return JVMCI::dependencies_failed;
-  }
-  // The dependencies were invalid at the time of installation
-  // without any intervening modification of the system
-  // dictionary.  That means they were invalidly constructed.
-  return JVMCI::dependencies_invalid;
+  return JVMCI::dependencies_failed;
 }
 
 // Reports a pending exception and exits the VM.
diff -r 22fa46d5dc2e src/hotspot/share/jvmci/vmStructs_jvmci.cpp
--- a/src/hotspot/share/jvmci/vmStructs_jvmci.cpp	Thu Aug 15 15:39:43 2019 +0800
+++ b/src/hotspot/share/jvmci/vmStructs_jvmci.cpp	Thu Aug 15 12:22:57 2019 +0200
@@ -542,11 +542,10 @@
   declare_constant(JumpData::taken_off_set)                               \
   declare_constant(JumpData::displacement_off_set)                        \
                                                                           \
-  declare_preprocessor_constant("JVMCIEnv::ok",                   JVMCI::ok)                      \
-  declare_preprocessor_constant("JVMCIEnv::dependencies_failed",  JVMCI::dependencies_failed)     \
-  declare_preprocessor_constant("JVMCIEnv::dependencies_invalid", JVMCI::dependencies_invalid)    \
-  declare_preprocessor_constant("JVMCIEnv::cache_full",           JVMCI::cache_full)              \
-  declare_preprocessor_constant("JVMCIEnv::code_too_large",       JVMCI::code_too_large)          \
+  declare_preprocessor_constant("JVMCI::ok",                   JVMCI::ok)                      \
+  declare_preprocessor_constant("JVMCI::dependencies_failed",  JVMCI::dependencies_failed)     \
+  declare_preprocessor_constant("JVMCI::cache_full",           JVMCI::cache_full)              \
+  declare_preprocessor_constant("JVMCI::code_too_large",       JVMCI::code_too_large)          \
   declare_constant(JVMCIRuntime::none)                                    \
   declare_constant(JVMCIRuntime::by_holder)                               \
   declare_constant(JVMCIRuntime::by_full_signature)                       \
diff -r 22fa46d5dc2e src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotCodeCacheProvider.java
--- a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotCodeCacheProvider.java	Thu Aug 15 15:39:43 2019 +0800
+++ b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotCodeCacheProvider.java	Thu Aug 15 12:22:57 2019 +0200
@@ -144,9 +144,6 @@
                 } else {
                     msg = String.format("Code installation failed: %s", resultDesc);
                 }
-                if (result == config.codeInstallResultDependenciesInvalid) {
-                    throw new AssertionError(resultDesc + " " + msg);
-                }
                 throw new BailoutException(result != config.codeInstallResultDependenciesFailed, msg);
             } else {
                 throw new BailoutException("Error installing %s: %s", ((HotSpotCompiledCode) compiledCode).getName(), resultDesc);
diff -r 22fa46d5dc2e src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotVMConfig.java
--- a/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotVMConfig.java	Thu Aug 15 15:39:43 2019 +0800
+++ b/src/jdk.internal.vm.ci/share/classes/jdk.vm.ci.hotspot/src/jdk/vm/ci/hotspot/HotSpotVMConfig.java	Thu Aug 15 12:22:57 2019 +0200
@@ -351,11 +351,10 @@
     final int vmIntrinsicLinkToSpecial = getConstant("vmIntrinsics::_linkToSpecial", Integer.class);
     final int vmIntrinsicLinkToInterface = getConstant("vmIntrinsics::_linkToInterface", Integer.class);
 
-    final int codeInstallResultOk = getConstant("JVMCIEnv::ok", Integer.class);
-    final int codeInstallResultDependenciesFailed = getConstant("JVMCIEnv::dependencies_failed", Integer.class);
-    final int codeInstallResultDependenciesInvalid = getConstant("JVMCIEnv::dependencies_invalid", Integer.class);
-    final int codeInstallResultCacheFull = getConstant("JVMCIEnv::cache_full", Integer.class);
-    final int codeInstallResultCodeTooLarge = getConstant("JVMCIEnv::code_too_large", Integer.class);
+    final int codeInstallResultOk = getConstant("JVMCI::ok", Integer.class);
+    final int codeInstallResultDependenciesFailed = getConstant("JVMCI::dependencies_failed", Integer.class);
+    final int codeInstallResultCacheFull = getConstant("JVMCI::cache_full", Integer.class);
+    final int codeInstallResultCodeTooLarge = getConstant("JVMCI::code_too_large", Integer.class);
 
     String getCodeInstallResultDescription(int codeInstallResult) {
         if (codeInstallResult == codeInstallResultOk) {
@@ -364,9 +363,6 @@
         if (codeInstallResult == codeInstallResultDependenciesFailed) {
             return "dependencies failed";
         }
-        if (codeInstallResult == codeInstallResultDependenciesInvalid) {
-            return "dependencies invalid";
-        }
         if (codeInstallResult == codeInstallResultCacheFull) {
             return "code cache is full";
         }

[XiaohongGong]

This looks a missing change from JDK-8222446:

diff -r 22fa46d5dc2e src/hotspot/share/jvmci/jvmciRuntime.cpp
--- a/src/hotspot/share/jvmci/jvmciRuntime.cpp  Thu Aug 15 15:39:43 2019 +0800
+++ b/src/hotspot/share/jvmci/jvmciRuntime.cpp  Thu Aug 15 12:11:12 2019 +0200
@@ -1325,21 +1325,13 @@
     return JVMCI::dependencies_failed;
   }

-  // Dependencies must be checked when the system dictionary changes
-  // or if we don't know whether it has changed (i.e., compile_state == NULL).
   CompileTask* task = compile_state == NULL ? NULL : compile_state->task();
   Dependencies::DepType result = dependencies->validate_dependencies(task, failure_detail);
   if (result == Dependencies::end_marker) {
     return JVMCI::ok;
   }

-  if (!Dependencies::is_klass_type(result) || compile_state == NULL) {
-    return JVMCI::dependencies_failed;
-  }
-  // The dependencies were invalid at the time of installation
-  // without any intervening modification of the system
-  // dictionary.  That means they were invalidly constructed.
-  return JVMCI::dependencies_invalid;
+  return JVMCI::dependencies_failed;
 }

 // Reports a pending exception and exits the VM.

Can you please test that locally and then open a OpenJDK bug for it to be fixed.

Yes, I think so. I think changing the result to JVMCI::dependencies_failed is more reasonable. I will open a OpenJDK bug and do more tests about it. Thank you!

[XiaohongGong]

Hi @dougxc , here is the JBS: https://bugs.openjdk.java.net/browse/JDK-8229797

[dougxc]

Thanks!