8360557: CTW: Inline cold methods to reach more code

[shipilev]

We use CTW testing for making sure compilers behave well. But we compile the code that is not executed at all, and since our inlining heuristics often looks back at profiles, we end up not actually inlining all too much! This means CTW testing likely misses lots of bugs that normal code is exposed to, especially e.g. in loop optimizations.

There is an intrinsic tradeoff with accepting more inilned methods in CTW: the compilation time gets significantly worse. With just accepting the cold methods we have reasonable CTW times, eating the improvements we have committed in mainline recently. And it still finds bugs. See the RFE for sample data.

After this lands and CTW starts to compile cold methods, one can greatly expand the scope of the CTW testing by overriding the static inlining limits. Doing e.g. TEST_VM_OPTS="-XX:MaxInlineSize=70 -XX:C1MaxInlineSize=70" finds even more bugs. Unfortunately, the compilation times suffer so much, they are impractical to run in standard configurations, see data in RFE. We will enable some of that testing in special testing pipelines.

Pre-empting the question: "Well, why not use -Xcomp then, and make sure it inlines well?" The answer is in RFE as well: Xcomp causes a lot of stray compilations for JDK and CTW infra itself. For small JARs in large corpus this eats precious testing time that we would instead like to spend on deeper inlining in the actual JAR code. This also does not force us to look into how CTW works in Xcomp at all; I expect some surprises there. Feather-touching the inlining heuristic paths to just accept methods without looking at profiles looks better.

Tobias had an idea to implement the stress randomized inlining that would expand the scope of inlining. This improvement stacks well with it. This improvement provides the base case of inlining most reasonable methods, and then allow stress infra to inline some more on top of that.

Additional testing:

• GHA
• Linux x86_64 server fastdebug, applications/ctw/modules
• Linux x86_64 server fastdebug, large CTW corpus (now failing in interesting ways)

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8360557: CTW: Inline cold methods to reach more code (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26068/head:pull/26068
$ git checkout pull/26068

Update a local copy of the PR:
$ git checkout pull/26068
$ git pull https://git.openjdk.org/jdk.git pull/26068/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26068

View PR using the GUI difftool:
$ git pr show -t 26068

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26068.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@shipilev The following label will be automatically applied to this pull request:

• hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (dedbcfed)
• 00: Full (b16cbabb)

[shipilev]

We are on par for CTW testing time, comparing to the state a week back:

# Before CTW perf improvements
real	5m0.528s
user	79m5.193s
sys	14m16.678s

# Current mainline
real	3m59.274s
user	68m9.663s
sys	5m19.026s

# This PR
real	4m56.248s
user	89m48.364s
sys	5m24.091s

[vnkozlov]

This has to be tested by us to make sure we clean up all issues this change find.

[shipilev]

This has to be tested by us to make sure we clean up all issues this change find.

Sure thing. There is a chicken-and-egg kind of problem that some bugs reproduce only with this PR, and maybe with extra inline tuning :) I am following up on failures that we are seeing.

[TobiHartmann]

I submitted some testing to make sure that CTW is clean in our CI.

[TobiHartmann]

I submitted some testing to make sure that CTW is clean in our CI.

I see the following crashes that would need to be fixed before this is integrated:

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/workspace/open/src/hotspot/share/opto/phaseX.cpp:2790), pid=3196445, tid=3196462
#  assert(!failure) failed: PhaseCCP not at fixpoint: analysis result may be unsound.
#
# JRE version: Java(TM) SE Runtime Environment (26.0) (fastdebug build 26-internal-2025-07-02-0711056.tobias.hartmann.jdk4)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 26-internal-2025-07-02-0711056.tobias.hartmann.jdk4, mixed mode, sharing, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x180cff8]  PhaseCCP::verify_analyze(Unique_Node_List&) [clone .part.0]+0x28

Current CompileTask:
C2:13166 2238    b        com.ibm.icu.impl.LocaleUtility::fallback (78 bytes)

Stack: [0x00007f20eca0c000,0x00007f20ecb0c000],  sp=0x00007f20ecb07050,  free space=1004k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x180cff8]  PhaseCCP::verify_analyze(Unique_Node_List&) [clone .part.0]+0x28  (phaseX.cpp:2790)
V  [libjvm.so+0x181e8aa]  PhaseCCP::analyze()+0x7ca  (phaseX.cpp:2790)
V  [libjvm.so+0xb44c94]  Compile::Optimize()+0x964  (compile.cpp:2479)
V  [libjvm.so+0xb480d3]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1ec3  (compile.cpp:858)
V  [libjvm.so+0x96d157]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x467  (c2compiler.cpp:141)
V  [libjvm.so+0xb574f8]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xb58  (compileBroker.cpp:2323)
V  [libjvm.so+0xb586c8]  CompileBroker::compiler_thread_loop()+0x578  (compileBroker.cpp:1967)
V  [libjvm.so+0x10abd0b]  JavaThread::thread_main_inner()+0x13b  (javaThread.cpp:773)
V  [libjvm.so+0x1b11f26]  Thread::call_run()+0xb6  (thread.cpp:243)
V  [libjvm.so+0x178c718]  thread_native_entry(Thread*)+0x128  (os_linux.cpp:868)

# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/workspace/open/src/hotspot/share/opto/phaseX.cpp:784), pid=2175071, tid=2175089
#  assert(no_dead_loop) failed: dead loop detected
#
# JRE version: Java(TM) SE Runtime Environment (26.0) (fastdebug build 26-internal-2025-07-02-0711056.tobias.hartmann.jdk4)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 26-internal-2025-07-02-0711056.tobias.hartmann.jdk4, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x180d285]  PhaseGVN::dead_loop_check(Node*) [clone .part.0]+0x1d5

Current CompileTask:
C2:4914 2051   !b  4       com.sun.beans.introspect.MethodInfo::get (273 bytes)

Stack: [0x00007fe603f00000,0x00007fe604000000],  sp=0x00007fe603ffaef0,  free space=1003k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x180d285]  PhaseGVN::dead_loop_check(Node*) [clone .part.0]+0x1d5  (phaseX.cpp:784)
V  [libjvm.so+0x181c309]  PhaseIterGVN::transform_old(Node*)+0x529  (phaseX.cpp:767)
V  [libjvm.so+0x1820505]  PhaseIterGVN::optimize()+0xc5  (phaseX.cpp:1054)
V  [libjvm.so+0xb414ba]  Compile::inline_incrementally_cleanup(PhaseIterGVN&)+0x2ca  (compile.cpp:2151)
V  [libjvm.so+0xb41ed6]  Compile::inline_incrementally(PhaseIterGVN&)+0x416  (compile.cpp:2201)
V  [libjvm.so+0xb447ae]  Compile::Optimize()+0x47e  (compile.cpp:2329)
V  [libjvm.so+0xb480d3]  Compile::Compile(ciEnv*, ciMethod*, int, Options, DirectiveSet*)+0x1ec3  (compile.cpp:858)
V  [libjvm.so+0x96d157]  C2Compiler::compile_method(ciEnv*, ciMethod*, int, bool, DirectiveSet*)+0x467  (c2compiler.cpp:141)
V  [libjvm.so+0xb574f8]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0xb58  (compileBroker.cpp:2323)
V  [libjvm.so+0xb586c8]  CompileBroker::compiler_thread_loop()+0x578  (compileBroker.cpp:1967)
V  [libjvm.so+0x10abd0b]  JavaThread::thread_main_inner()+0x13b  (javaThread.cpp:773)
V  [libjvm.so+0x1b11f26]  Thread::call_run()+0xb6  (thread.cpp:243)
V  [libjvm.so+0x178c718]  thread_native_entry(Thread*)+0x128  (os_linux.cpp:868)