8370315: [IR-Framework] Allow scenarios to be run in parallel by dafedafe · Pull Request #28065 · openjdk/jdk

dafedafe · 2025-10-30T16:27:11Z

Issue

Today, the only practical ways to run IR Framework scenarios in parallel seems to be:

spawning threads manually in a single test, or
letting jtreg treat each scenario as a separate test (the only way to potentially distribute across hosts).

This makes it a bit cumbersome to use host CPU cores efficiently when running multiple scenarios within the same test.

Change

This change introduces a method TestFramework::startParallel to execute multiple scenarios concurrently. The implementation:

launches one task per scenario and runs them in parallel (by default, the maximum concurrency should match the host’s available cores)
captures each task’s System.out into a dedicated buffer and flushes it when the task completes to avoid interleaved output between scenarios (Note: only call paths within the compile.lib.ir_framework package are modified to per-task output streams. ProcessTools methods still write directly to stdout, so their output may interleave).
adds an option -DForceSequentialScenarios=true to force all scenarios to be run sequentially.

Testing

Tier 1-3+
explicit ir_framework.tests runs
- added IR-Framework test TestDForceSequentialScenarios.java to test forcing sequential testing (checkin the output order) and added a parallel run to TestScenatios.java (as well as adding ForceSequentialScenarios flag to TestDFlags.java)

As reference: a comparison of the execution time between sequential and parallel of all IR-Framework tests using scenarios on our machines (linux x64/aarch64, macosx x64/aarch64, windows x64 with different number of cores, so the results for a single test might not be relevant) gave me an average speedup of 1.9.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8370315: [IR-Framework] Allow scenarios to be run in parallel (Enhancement - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28065/head:pull/28065
$ git checkout pull/28065

Update a local copy of the PR:
$ git checkout pull/28065
$ git pull https://git.openjdk.org/jdk.git pull/28065/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28065

View PR using the GUI difftool:
$ git pr show -t 28065

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28065.diff

Using Webrev

Link to Webrev Comment

bridgekeeper · 2025-10-30T16:28:51Z

👋 Welcome back dfenacci! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2025-10-30T16:29:41Z

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

openjdk · 2025-10-30T16:30:56Z

@dafedafe The following label will be automatically applied to this pull request:

hotspot-compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

dafedafe · 2025-10-31T16:31:04Z

The full list of tests using IR-Framework scenarios:

compiler/c2/irTests/gc/ReferenceClearTests.java
compiler/c2/irTests/gc/ReferenceRefersToTests.java
compiler/c2/irTests/igvn/TestCleanMemPhi.java
compiler/c2/irTests/scalarReplacement/AllocationMergesTests.java
compiler/c2/irTests/TestFloat16ScalarOperations.java
compiler/c2/irTests/TestPostParseCallDevirtualization.java
compiler/c2/irTests/TestScheduleSmallMethod.java
compiler/c2/irTests/TestVectorizationMismatchedAccess.java🎲
compiler/gcbarriers/TestG1BarrierGeneration.java
compiler/loopopts/superword/ProdRed_Double.java
compiler/loopopts/superword/ProdRed_Float.java
compiler/loopopts/superword/ProdRed_Int.java
compiler/loopopts/superword/RedTest_int.java
compiler/loopopts/superword/RedTest_long.java
compiler/loopopts/superword/SumRed_Double.java
compiler/loopopts/superword/SumRed_Float.java
compiler/loopopts/superword/SumRed_Int.java
compiler/loopopts/superword/SumRed_Long.java
compiler/loopopts/superword/SumRedAbsNeg_Double.java
compiler/loopopts/superword/SumRedAbsNeg_Float.java
compiler/loopopts/superword/SumRedSqrt_Double.java
compiler/loopopts/superword/TestMemorySegment_8359688.java
compiler/loopopts/superword/TestMemorySegment_ReassociateInvariants1.java
compiler/loopopts/superword/TestMemorySegment_ReassociateInvariants2.java
compiler/loopopts/superword/TestMemorySegmentByteSizeLongLoopLimit.java
compiler/loopopts/superword/TestMemorySegmentField.java
compiler/loopopts/superword/TestMemorySegmentFilterSummands.java
compiler/loopopts/TestArrayFillIntrinsic.java
compiler/vectorization/TestFloatConversionsVector.java

Maybe we could run some of them concurrently (e.g. the ones that show more than a certain speedup to avoid parallelizing short tests that might waste most of the time spawning threads).

mlbridge · 2025-10-31T16:39:54Z

Webrevs

chhagedorn

Overall looks good, thanks for improving this! I left a few suggestions.

Now the only question remaining is which tests would already benefit from using the parallel version. I guess we can investigate that separately.

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

chhagedorn · 2025-11-03T07:51:19Z

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

+                synchronized (printLock) {
+                    String output = baos.toString();
+                    if (!output.isEmpty()) {
+                        System.out.println(output);
+                    }
+                }


This will probably also not be sorted by scenario index? Could we also just gather it here and then dump it after the stream? Maybe we can put output into Outcome as well as the exceptions by using a ConcurrentSkipListMap<Scenario, Outcome> map in the parallel case or a normal TreeMap in the non-parallel case.

The idea was to print the output as soon as one process finishes, so that we can follow the progress a bit better (and if there is an TestFormatException we have it printed up to the exception, although this could be done later as well). Of course then it is not sorted...
That said, the output would be cleaner and more readable if we printed it in order as you suggest. Maybe we could even interleave output and exceptions for the same scenario. What do you think? (though I just noticed we throw another TestRunException after printing the exceptions)

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestScenarios.java

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

dafedafe · 2025-11-05T15:59:36Z

Now the only question remaining is which tests would already benefit from using the parallel version. I guess we can investigate that separately.

Let me file an RFE for that.

bridgekeeper · 2025-12-04T00:12:24Z

@dafedafe This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

chhagedorn

Sorry, I dropped the ball on this - thanks for the updates! Some more comments but then I think it looks good!

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestScenarios.java

chhagedorn · 2025-12-11T15:20:58Z

test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

+                    String output = baos.toString();
+                    if (!output.isEmpty()) {
+                        System.out.println(output);
+                    }


We probably also need to do a similar trick as for the exceptions in order to have ordered stdouts for the scenarios?

That would be a good idea. Adding it...

@chhagedorn I guess we can drop the printing of exception at the end then. Is that part of what you were suggesting?

I might have spoken too soon: JTReg seems to collect stdout and stderr and print them out at once at the end of each (JTReg) test. In this case it doesn't make much sense to print out the output of each test as soon as it finishes (it would be better to collect them and print them in order at the end). @chhagedorn, is there possibly a way to make JTReg print the output "on-the-fly" that you are aware of?

Yes, I agree. Let's just collect everything, stdout and exceptions, and then print them in scenario index order at the end. That probably also simplifies the logic.

is there possibly a way to make JTReg print the output "on-the-fly" that you are aware of?

I'm not aware of such an option but thought it would be useful in the past when having a long running test and I'm actually only interested in some printed messages at the very start.

test/hotspot/jtreg/compiler/lib/ir_framework/shared/TestFormat.java

test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestBadFormat.java

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

…enarios.java Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

….java Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

bridgekeeper · 2026-01-14T09:23:50Z

@dafedafe This pull request has been inactive for more than 4 weeks and will be automatically closed if another 4 weeks passes without any activity. To avoid this, simply issue a /touch or /keepalive command to the pull request. Feel free to ask for assistance if you need help with progressing this pull request towards integration!

dafedafe added 5 commits October 30, 2025 10:18

JDK-8370315: [IR-Framework] Allow scenarios to be run in parallel

aa2f058

JDK-8370315: fix output processing

9add841

JDK-8370315: revert removal of lastTestVMOutput

17b63f5

JDK-8370315: move getLastTestVMOutput

82da8ad

JDK-8370315: revert a few line changes

ca3336f

openjdk bot changed the title ~~JDK-8370315: [IR-Framework] Allow scenarios to be run in parallel~~ 8370315: [IR-Framework] Allow scenarios to be run in parallel Oct 30, 2025

openjdk bot added the hotspot-compiler hotspot-compiler-dev@openjdk.org label Oct 30, 2025

dafedafe added 3 commits October 31, 2025 15:39

JDK-8370315: add flag and tests

7c1a85e

JDK-8370315: remove garbage

7a75682

JDK-8370315: move declaration

6795083

dafedafe marked this pull request as ready for review October 31, 2025 16:34

openjdk bot added the rfr Pull request is ready for review label Oct 31, 2025

chhagedorn reviewed Nov 3, 2025

View reviewed changes

dafedafe and others added 7 commits November 3, 2025 16:52

JDK-8370315: move flag check to startParallel

97a21c4

JDK-8370315: don't process all scenarios if TestFormatException

d807a2f

JDK-8370315: make failures thread local

8fa0e8c

JDK-8370315: add test with scenarios to format exception tests

c929597

JDK-8370315: add exceptions as they arise

4c72531

Apply suggestion from @chhagedorn

2b8de57

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

JDK-8370315: use check function

4e9cc0a

JDK-8370315: fix typo

7b64383

chhagedorn reviewed Dec 11, 2025

View reviewed changes

Update test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

396bb8b

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

dafedafe and others added 3 commits December 12, 2025 18:40

Update test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

e735b43

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

Update test/hotspot/jtreg/compiler/lib/ir_framework/TestFramework.java

0b0e7df

Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

Update test/hotspot/jtreg/testlibrary_tests/ir_framework/tests/TestSc…

f85f341

…enarios.java Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

openjdk bot removed the rfr Pull request is ready for review label Dec 12, 2025

dafedafe and others added 3 commits December 12, 2025 18:44

Update test/hotspot/jtreg/compiler/lib/ir_framework/shared/TestFormat…

203c32d

….java Co-authored-by: Christian Hagedorn <christian.hagedorn@oracle.com>

JDK-8370315: syntax

74b51fd

JDK-8370315: Remove whitespace

f3e51d5

openjdk bot added the rfr Pull request is ready for review label Dec 12, 2025

Conversation

dafedafe commented Oct 30, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Change

Testing

Progress

Issue

Reviewing

Uh oh!

bridgekeeper bot commented Oct 30, 2025

Uh oh!

openjdk bot commented Oct 30, 2025

Uh oh!

openjdk bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dafedafe commented Oct 31, 2025

Uh oh!

mlbridge bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

chhagedorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

chhagedorn Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

dafedafe Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dafedafe commented Nov 5, 2025

Uh oh!

bridgekeeper bot commented Dec 4, 2025

Uh oh!

chhagedorn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chhagedorn Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

dafedafe Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

dafedafe Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

dafedafe Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

chhagedorn Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bridgekeeper bot commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

dafedafe commented Oct 30, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Oct 30, 2025 •

edited

Loading

mlbridge bot commented Oct 31, 2025 •

edited

Loading