AArch64 CompileTheWorld: (assembler_aarch64.hpp:218) guarantee(chk == -1 || chk == 0) failed: Field too big for insn

[sanzinger]

When running mx gate -t CTW:hosted on aarch64 the VM crashes:

$ mx gate -t CTW:hosted
gate: 26 Feb 2018 05:43:38(+00:00) BEGIN: Gate
gate: 26 Feb 2018 05:43:38(+00:00) BEGIN: CTW:hosted
WARNING: Using incubator modules: jdk.incubator.httpclient
CompileTheWorld : iteration 0
Feb 26, 2018 5:43:53 AM sun.awt.X11.XBaseWindow init
WARNING: Exception during peer initialization
java.awt.AWTError: Local GraphicsEnvironment must not be null
        at java.desktop/sun.awt.X11.XToolkit.getDisplay(XToolkit.java:309)
        at java.desktop/sun.awt.X11.XToolkit.getDefaultRootWindow(XToolkit.java:317)
        at java.desktop/sun.awt.X11.XBaseWindow.checkParams(XBaseWindow.java:280)
        at java.desktop/sun.awt.X11.XBaseWindow.create(XBaseWindow.java:323)
        at java.desktop/sun.awt.X11.XBaseWindow.init(XBaseWindow.java:183)
        at java.desktop/sun.awt.X11.XRootWindow$LazyHolder.<clinit>(XRootWindow.java:41)
        at java.desktop/sun.awt.X11.XRootWindow.getInstance(XRootWindow.java:49)
        at java.desktop/sun.awt.X11.XBaseWindow.getXAWTRootWindow(XBaseWindow.java:426)
        at java.desktop/sun.awt.X11.MotifDnDDragSourceProtocol.<init>(MotifDnDDragSourceProtocol.java:52)
        at java.desktop/sun.awt.X11.MotifDnDDragSourceProtocol.createInstance(MotifDnDDragSourceProtocol.java:61)
        at java.desktop/sun.awt.X11.XDragAndDropProtocols.<clinit>(XDragAndDropProtocols.java:58)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
        at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1042)
        at jdk.internal.vm.ci/jdk.vm.ci.hotspot.HotSpotConstantPool.loadReferencedType(HotSpotConstantPool.java:715)
        at org.graalvm.compiler.hotspot.test.CompileTheWorld.compile(CompileTheWorld.java:574)
        at org.graalvm.compiler.hotspot.test.CompileTheWorld.compile(CompileTheWorld.java:256)
        at org.graalvm.compiler.hotspot.test.CompileTheWorld.main(CompileTheWorld.java:849)

Monitor starting
CompileTheWorld : Waiting for 236047 compiles
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (assembler_aarch64.hpp:218), pid=12665, tid=12776
#  guarantee(chk == -1 || chk == 0) failed: Field too big for insn
#
# JRE version: OpenJDK Runtime Environment (10.0) (build 10-internal+0-adhoc.sanzinger.jdk10)
# Java VM: OpenJDK 64-Bit Server VM (10-internal+0-adhoc.sanzinger.jdk10, mixed mode, tiered, jvmci, compressed oops, g1 gc, linux-aarch64)
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/sanzinger/graal/graal/compiler/hs_err_pid12665.log
[thread 12756 also had an error]
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

gate: 26 Feb 2018 05:44:09(+00:30) END:   CTW:hosted [0:00:30.690080] [disk (free/total): 3.5TB/3.6TB]

Stacktrace:

Command Line: -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCI --module-path=/home/sanzinger/graal/graal/sdk/mxbuild/modules/org.graalvm.graal_sdk.jar:/home/sanzinger/graal/graal/truffle/mxbuild/modules/com.oracle.truffle.truffle_api.jar:/home/sanzinger/graal/graal/compiler/mxbuild/modules/jdk.internal.vm.compiler.jar -Djava.awt.headless=true -esa -XX:-UseJVMCICompiler -XX:+EnableJVMCI -DCompileTheWorld.MultiThreaded=true -Dgraal.InlineDuringParsing=false -DCompileTheWorld.Verbose=false -XX:ReservedCodeCacheSize=300m -DCompileTheWorld.Config=Inline=false#CompilationFailureAction=ExitVM -DCompileTheWorld.ExcludeMethodFilter=sun.awt.X11.*.* --add-modules=jdk.pack,java.transaction,jdk.rmic,java.corba,jdk.internal.vm.ci,jdk.jcmd,java.xml.bind,java.xml.ws,jdk.xml.ws,jdk.scripting.nashorn.shell,java.xml.ws.annotation,jdk.xml.bind,java.activation,java.se.ee,jdk.hotspot.agent,jdk.incubator.httpclient --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot=ALL-UNNAMED --add-exports=jdk.internal.vm.ci/jdk.vm.ci.meta=ALL-UNNAMED --add-exports=jdk.internal.vm.ci/jdk.vm.ci.services=ALL-UNNAMED --add-exports=jdk.internal.vm.ci/jdk.vm.ci.runtime=ALL-UNNAMED org.graalvm.compiler.hotspot.test.CompileTheWorld

Host: AArch64, 32 cores, 251G, Oracle Linux Server release 7.4
Time: Mon Feb 26 05:44:09 2018 PST elapsed time: 28 seconds (0d 0h 0m 28s)

---------------  T H R E A D  ---------------

Current thread (0x0000ffff94979000):  JavaThread "CompileTheWorld-53" daemon [_thread_in_vm, id=12776, stack(0x0000fffdb5c00000,0x0000fffdb5e00000)]

Stack: [0x0000fffdb5c00000,0x0000fffdb5e00000],  sp=0x0000fffdb5dfcb60,  free space=2034k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xa3a4e0]  VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x14c
V  [libjvm.so+0xa3b078]  VMError::report_and_die(Thread*, char const*, int, char const*, char const*, std::__va_list)+0x50
V  [libjvm.so+0x4c1d48]  report_vm_error(char const*, int, char const*, char const*, ...)+0xc0
V  [libjvm.so+0x7d0f38]  MacroAssembler::pd_patch_instruction_size(unsigned char*, unsigned char*)+0x210
V  [libjvm.so+0x454090]  CodeBuffer::relocate_code_to(CodeBuffer*) const+0x334
V  [libjvm.so+0x454c4c]  CodeBuffer::copy_code_to(CodeBlob*)+0x248
V  [libjvm.so+0x868004]  nmethod::nmethod(Method*, CompilerType, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, _jobject*, _jobject*)+0x350
V  [libjvm.so+0x8689b4]  nmethod::new_nmethod(methodHandle const&, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, _jobject*, _jobject*)+0x1c4
V  [libjvm.so+0x6d0304]  JVMCIEnv::register_method(methodHandle const&, nmethod*&, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, AbstractCompiler*, DebugInformationRecorder*, Dependencies*, JVMCIEnv*, int, bool, bool, Handle, Handle, Handle)+0x2b4
V  [libjvm.so+0x69b4ec]  CodeInstaller::install(JVMCICompiler*, Handle, Handle, CodeBlob*&, Handle, Handle, Thread*)+0x45c
V  [libjvm.so+0x6cd5e0]  c2v_installCode(JNIEnv_*, _jobject*, _jobject*, _jobject*, _jobject*, _jobject*)+0x370
j  jdk.vm.ci.hotspot.CompilerToVM.installCode(Ljdk/vm/ci/code/TargetDescription;Ljdk/vm/ci/hotspot/HotSpotCompiledCode;Ljdk/vm/ci/code/InstalledCode;Ljdk/vm/ci/hotspot/HotSpotSpeculationLog;)I+0 jdk.internal.vm.ci@10-internal
j  jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljdk/vm/ci/code/CompiledCode;Ljdk/vm/ci/code/InstalledCode;Ljdk/vm/ci/meta/SpeculationLog;Z)Ljdk/vm/ci/code/InstalledCode;+102 jdk.internal.vm.ci@10-internal
j  org.graalvm.compiler.core.target.Backend.createInstalledCode(Lorg/graalvm/compiler/debug/DebugContext;Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljdk/vm/ci/code/CompilationRequest;Lorg/graalvm/compiler/code/CompilationResult;Ljdk/vm/ci/meta/SpeculationLog;Ljdk/vm/ci/code/InstalledCode;Z[Ljava/lang/Object;)Ljdk/vm/ci/code/InstalledCode;+164 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.installMethod(Lorg/graalvm/compiler/debug/DebugContext;Lorg/graalvm/compiler/code/CompilationResult;)V+116 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.access$500(Lorg/graalvm/compiler/hotspot/CompilationTask;Lorg/graalvm/compiler/debug/DebugContext;Lorg/graalvm/compiler/code/CompilationResult;)V+3 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+243 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.CompilationWrapper.run(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+157 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation()Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+36 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.compileMethod(Ljdk/vm/ci/hotspot/HotSpotResolvedJavaMethod;I)V+60
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.access$400(Lorg/graalvm/compiler/hotspot/test/CompileTheWorld;Ljdk/vm/ci/hotspot/HotSpotResolvedJavaMethod;I)V+3
j  org.graalvm.compiler.hotspot.test.CompileTheWorld$1.run()V+52
j  java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 java.base@10-internal
j  java.util.concurrent.FutureTask.run()V+39 java.base@10-internal
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@10-internal
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@10-internal
j  java.lang.Thread.run()V+11 java.base@10-internal
j  org.graalvm.compiler.core.CompilerThread.run()V+12 jdk.internal.vm.compiler
v  ~StubRoutines::call_stub
V  [libjvm.so+0x63ebd4]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x38c
V  [libjvm.so+0x8959f8]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*), JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x20
V  [libjvm.so+0x63d6f8]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x168
V  [libjvm.so+0x682c20]  thread_entry(JavaThread*, Thread*)+0x94
V  [libjvm.so+0x9e7774]  JavaThread::thread_main_inner()+0xfc
V  [libjvm.so+0x8978a0]  thread_native_entry(Thread*)+0x104
C  [libpthread.so.0+0x7bb0]  start_thread+0xb4
C  [libc.so.6+0xdb630]  thread_start+0x30

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  jdk.vm.ci.hotspot.CompilerToVM.installCode(Ljdk/vm/ci/code/TargetDescription;Ljdk/vm/ci/hotspot/HotSpotCompiledCode;Ljdk/vm/ci/code/InstalledCode;Ljdk/vm/ci/hotspot/HotSpotSpeculationLog;)I+0 jdk.internal.vm.ci@10-internal
j  jdk.vm.ci.hotspot.HotSpotCodeCacheProvider.installCode(Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljdk/vm/ci/code/CompiledCode;Ljdk/vm/ci/code/InstalledCode;Ljdk/vm/ci/meta/SpeculationLog;Z)Ljdk/vm/ci/code/InstalledCode;+102 jdk.internal.vm.ci@10-internal
j  org.graalvm.compiler.core.target.Backend.createInstalledCode(Lorg/graalvm/compiler/debug/DebugContext;Ljdk/vm/ci/meta/ResolvedJavaMethod;Ljdk/vm/ci/code/CompilationRequest;Lorg/graalvm/compiler/code/CompilationResult;Ljdk/vm/ci/meta/SpeculationLog;Ljdk/vm/ci/code/InstalledCode;Z[Ljava/lang/Object;)Ljdk/vm/ci/code/InstalledCode;+164 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.installMethod(Lorg/graalvm/compiler/debug/DebugContext;Lorg/graalvm/compiler/code/CompilationResult;)V+116 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.access$500(Lorg/graalvm/compiler/hotspot/CompilationTask;Lorg/graalvm/compiler/debug/DebugContext;Lorg/graalvm/compiler/code/CompilationResult;)V+3 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+243 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask$HotSpotCompilationWrapper.performCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.core.CompilationWrapper.run(Lorg/graalvm/compiler/debug/DebugContext;)Ljava/lang/Object;+2 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation(Lorg/graalvm/compiler/debug/DebugContext;)Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+157 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.CompilationTask.runCompilation()Ljdk/vm/ci/hotspot/HotSpotCompilationRequestResult;+36 jdk.internal.vm.compiler
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.compileMethod(Ljdk/vm/ci/hotspot/HotSpotResolvedJavaMethod;I)V+60
j  org.graalvm.compiler.hotspot.test.CompileTheWorld.access$400(Lorg/graalvm/compiler/hotspot/test/CompileTheWorld;Ljdk/vm/ci/hotspot/HotSpotResolvedJavaMethod;I)V+3
j  org.graalvm.compiler.hotspot.test.CompileTheWorld$1.run()V+52
j  java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 java.base@10-internal
j  java.util.concurrent.FutureTask.run()V+39 java.base@10-internal
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@10-internal
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@10-internal
j  java.lang.Thread.run()V+11 java.base@10-internal
j  org.graalvm.compiler.core.CompilerThread.run()V+12 jdk.internal.vm.compiler
v  ~StubRoutines::call_stub

JDK 10 Version:

jdk10$ hg log -l 1
changeset:   48546:ed9ff662b1e0
tag:         tip
user:        lana
date:        Tue Feb 20 17:01:18 2018 +0000
summary:     Added tag jdk-10+44 for changeset 663f20fc5109

/cc: @adinn

[adinn]

Hi Stefan,

It appears that this is happening because Graal is not allowing for the possibility that certain branches between generated routines may require offsets greater than 26 bits. That can happen when you use a large enough code cache (the CTW tests passes -XX:ReservedCodeCacheSize=300m).

This specific error is happening when a method gets relocated. The unverified entry for the method ends with a branch to the ic_miss_stub. This is generated in Graal by calling AArch64_Call.directJmp which uses the macroassembler method jmp to plant an unconditional PC-relative branch instruction (b 0xNNNNNNNN). The range of the offset is 26 signed bits. For this specific relocation the relocation buffer is far enough from the generated ic_miss_stub that the branch offset will not into the offset slot, hence the assert.

The corresponding code generation in AArch64 can be seen by looking at file aarch64.ad (for the C2 compiler). If you search for ic_miss_stub you will see that the code which emits the unverified entry call macroassembler method far_jump. That method checks the cache size and if it might result in offsets which may not fit into 26 signed bits uses an adrp instruction to load the address and then an unconditional branch via register to do the jump.

So, what is needed is either (short-term) to limit the cache size to avoid this problem or (long-term) to modify the Graal code generation strategy so that far jumps (also far calls) are used where necessary. Is that a task that has already been assigned to anyone?

[adinn]

I just noticed that Andrew Haley provided a PR which fixes the offset problem on January 31:

#292

It would be a very good idea if someone accepted that pull request :-)

[sanzinger]

Hi Andrew,

thank you very much for the explanation. I have not looked in the PRs so far. Sorry about that. So let me check how I can get in the changes.