Trivial call site not inlining with invokedynamic #6280
Comments
|
Note that I need to run via |
|
Thanks for filing this! As we discussed on Matrix, I did play with your other example a bit and was able to reproduce it. When the target script is immediately compiled, everything inlines well. But when not immediately compiling that script and instead allowing normal JIT thresholds to kick in, we get this peculiar "unloaded signature classes" message from e.g. PrintInlining output. I only looked into it briefly, but turning off tiered compilation seems to help; the message and deopt still appears, but it's followed shortly after by a new compilation that appears to inline and emit the expected native code. So tiering could indeed be to blame. The other theory I have is that our one-method-one-classloader mechanism (done so that abandoned JIT classes can get thrown out by the JVM) may be confounding something in the method handle inlining logic. FWIW I do not believe this has always been here, but I couldn't point at any particular change that might have regressed it. There's also the possibility that this has been intrinsic to the always-on tiered compilation (starting some time in Java 8) and we just never saw it because we usually test against a fully-compiled target script and/or a JVM in which tiering has been turned off (to avoid the extra noise caused by early tiers. In any case, this is a great find, and could mean that a whole lot of invokedynamic optimizations are simply not being realized in JRuby today. |
|
Some things we found out:
|
|
So this does appear to be a bug in tiered compilation going back at least as far as OpenJDK 8 and still present in OpenJDK 14. WIth tiered compilation disabled (-XX:-TieredCompilation) the "bar" method appears to inline properly, and the resulting native code is what we would expect to see. This means that using C2 alone works fine. Given the complexities of tiered compilation in OpenJDK, I'm not sure it's a good use of my time to try to dig deeper, so I will raise a red flag on the Hotspot compiler dev list. |
|
What it looks like when it's working 0x000000010e3cd540: mov %eax,-0x14000(%rsp)
0x000000010e3cd547: push %rbp
0x000000010e3cd548: sub $0x20,%rsp ;*synchronization entry
; - Users.chrisseaton.src.github_dot_com.jruby.jruby.inline::RUBY$method$foo$0@-1
0x000000010e3cd54c: mov %rsi,%r10
0x000000010e3cd54f: mov 0x8(%r8),%r9d ; implicit exception: dispatches to 0x000000010e3cd5e0
0x000000010e3cd553: cmp $0x201b37,%r9d ; {metadata('org/jruby/gen/RubyObject1')}
0x000000010e3cd55a: jne 0x000000010e3cd5b4
0x000000010e3cd55c: mov %r8,%r11 ;*checkcast {reexecute=0 rethrow=0 return_oop=0}
; - org.jruby.RubyBasicObject::getMetaClass@1 (line 525)
; - org.jruby.ir.targets.indy.Bootstrap::testType@2 (line 1001)
; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
; - java.lang.invoke.LambdaForm$MH/0x0000000801063440::invoke@25
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@37
; - java.lang.invoke.DelegatingMethodHandle$Holder::delegate@17
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@78
; - java.lang.invoke.Invokers$Holder::linkToCallSite@17
; - Users.chrisseaton.src.github_dot_com.jruby.jruby.inline::RUBY$method$foo$0@9 (line 8)
0x000000010e3cd55f: mov 0x14(%r11),%ebp ;*getfield metaClass {reexecute=0 rethrow=0 return_oop=0}
; - org.jruby.RubyBasicObject::getMetaClass@4 (line 525)
; - org.jruby.ir.targets.indy.Bootstrap::testType@2 (line 1001)
; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
; - java.lang.invoke.LambdaForm$MH/0x0000000801063440::invoke@25
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@37
; - java.lang.invoke.DelegatingMethodHandle$Holder::delegate@17
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@78
; - java.lang.invoke.Invokers$Holder::linkToCallSite@17
; - Users.chrisseaton.src.github_dot_com.jruby.jruby.inline::RUBY$method$foo$0@9 (line 8)
0x000000010e3cd563: cmp $0xc030001c,%ebp ; {oop(a 'org/jruby/RubyClass'{0x00000006018000e0})}
0x000000010e3cd569: jne 0x000000010e3cd585 ;*if_acmpne {reexecute=0 rethrow=0 return_oop=0}
; - org.jruby.ir.targets.indy.Bootstrap::testType@5 (line 1001)
; - java.lang.invoke.DirectMethodHandle$Holder::invokeStatic@11
; - java.lang.invoke.LambdaForm$MH/0x0000000801063440::invoke@25
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@37
; - java.lang.invoke.DelegatingMethodHandle$Holder::delegate@17
; - java.lang.invoke.LambdaForm$MH/0x0000000801064040::guard@78
; - java.lang.invoke.Invokers$Holder::linkToCallSite@17
; - Users.chrisseaton.src.github_dot_com.jruby.jruby.inline::RUBY$method$foo$0@9 (line 8)
0x000000010e3cd56b: movabs $0x6004309d8,%rax ; {oop(a 'org/jruby/RubyFixnum'{0x00000006004309d8})}
0x000000010e3cd575: add $0x20,%rsp
0x000000010e3cd579: pop %rbp
0x000000010e3cd57a: mov 0x110(%r15),%r10
0x000000010e3cd581: test %eax,(%r10) ; {poll_return}
0x000000010e3cd584: retq |
|
And here's the PrintInlining output for the broken setup: With tiered compilation disabled, the ...bar... method inlines correctly. |
|
I have confirmed this fails on the most recent JRuby release, 9.2.11.1, and works properly there with tiered compilation disabled. Here's the PrintInlining output for the working setup with tiering off: |
|
Hey guys! I had some time this morning so I decided to dig into this a bit while writing some doc :) I was able to reproduce the issue on my Mac but it failed maybe 1 out of 10 times for me so I started playing with command line options to make it more reproducible. When I run with the following options I see the failure to inline 100% of the time: The last line in the PrintInlining is always: Starting at the error message Out of time for today most likely but I might find some more time later this evening to dig further into this. |
In #6280 we found that JIT-generated code for Ruby methods was not inlining, due to a reported error of "unloaded signature classes". This error means that not all classes in the target method's signature were fully loaded and resolved at the point of inlining, and since the Hotspot JIT cannot directly load classes it would reject these methods. Specifically for this case, at least the String argument (called method name) and possibly the RubyModule argument (called-as class) were flagged as "unloaded", likely because the simple code body in #6280 never actually accessed those arguments. This patch forces the generated code to forcibly load and resolve all classes used in the jitted method signatures, which should guarantee they do not appear "unloaded" at native JIT inlining time. Ideally this will be a fairly cheap operation, performed only once at classloading time, since the classes in question are clearly already loaded in the parent classloaders.
|
Updating from the hotspot-compiler-dev list discussion... Vladimir Ivanov was able to reproduce the "unloaded signature class" problem and let us know that the issue was those classes not being fully loaded and resolved in the JIT method's classloader. As @charliegracie reported above, the example seems to fail with one of two "unloaded" classes, either RubyModule or String. These two classes are provided to method bodies so they can "super" with a proper method name and lookup class, but this trivial example does not use either of those arguments. As a result, not all classes in the method signature are fully loaded and resolved and the JIT rejects it for inlining. I have committed a workaround that forcibly loads all classes used by our JIT method signatures, and this appears to fix the issue. It is not clear at this time whether we're doing anything "wrong". The method is called (possibly hundreds of times) before becoming eligible for inlining, but without accessing those arguments nothing forces those classes to resolve. I do not like the workaround, and it seems like this problem could be affecting lots of other generated code, but at least we have a path forward. The output below shows the inlined assembly for "foo" plus "bar" and it looks about like we expect it to look. |
In #6280 we found that JIT-generated code for Ruby methods was not inlining, due to a reported error of "unloaded signature classes". This error means that not all classes in the target method's signature were fully loaded and resolved at the point of inlining, and since the Hotspot JIT cannot directly load classes it would reject these methods. Specifically for this case, at least the String argument (called method name) and possibly the RubyModule argument (called-as class) were flagged as "unloaded", likely because the simple code body in #6280 never actually accessed those arguments. This patch forces the generated code to forcibly load and resolve all classes used in the jitted method signatures, which should guarantee they do not appear "unloaded" at native JIT inlining time. Ideally this will be a fairly cheap operation, performed only once at classloading time, since the classes in question are clearly already loaded in the parent classloaders. For JRuby 9.2, this is only done for the "JVM7" logic, since the "JVM6" logic typically does not make methods inlinable anyway.
|
Better fix will be to make sure any classes used in generated signatures are added to the static initializer, rather than having a hardcoded list. |
|
I'm going to mark this fixed for 9.2.14 and 9.3 and open a separate issue for improving this logic. |
|
Noting for the record that we ran into this JIT corner case in a JMH benchmark for measuring the performance of Scala's generated lampepfl/dotty-feature-requests#196 In that case, the nested classloader was just part of the usual the JDK 9+ classloader layers (not a custom one like in this ticket). The funny part was that although the benchmark instantiated the test data with lots of |
|
Thank you for posting @retronym! Good to hear that you determined the cause and an appropriate solution. I did ask the JIT folks if there's any way to force signature classes to initialize when they become eligible for JIT or inlining but the answer gets complicated due to the asynchronous nature of the JIT (it can't just call back into Java to boot those classes). Perhaps a classloader-level solution might be workable, like an eager-loading classloader that scans all signatures and makes sure the classes get properly initialized. I still have that open issue above to find a more general fix. |
Environment Information
Provide at least:
jruby 9.3.0.0-SNAPSHOT (2.6.5) 2020-06-12 ccecbe8014 OpenJDK 64-Bit Server VM 14.0.1+7 on 14.0.1+7 +jit [darwin-x86_64](master)Darwin Chris-Seatons-MacBook-Pro.local 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64Expected Behavior
main.rb
inline.rb
I'd expect the
invokedynamiccall fromfootobarto be fully inlined in C2.Actual Behavior
Not inlined.
The inlining log here is confusing - sometimes says it's inlined, sometimes says not due toThe inlining log says 'unloaded signature classes'. Possibly worth looking for a deoptimisation and recompilation? Could also be related to tiering.But whatever the logs say, bottom line seems to be there is a
callqwhere that does not seem reasonable. (But I'm not an expert oninvokedynamic.)As discussed on Matrix.
The text was updated successfully, but these errors were encountered: