ForkJoinPool compensation-thread regression (JDK-8300995) causes test flakiness on JDK 21+

[He-Pin]

Summary

The Pekko nightly CI has observed intermittent test failures on JDK 21 and JDK 25 that trace to a ForkJoinPool behavioral regression in JDK 21+ (JDK-8300995, JDK-8321335).

This issue documents the root cause and proposed mitigations for Pekko's stream tests. Related: #2573.

---

Root Cause

Pekko's default fork-join-executor uses asyncMode=true (FIFO, via task-peeking-mode = "FIFO"). In JDK 21+, this mode interacts poorly with actor round-trip workloads:

1. When actor A sends a message to actor B and waits for a reply, the reply task is placed at the back of the FIFO work queue.
2. With a fixed-size pool (e.g. 8 threads) under load, the reply waits behind all pending tasks before being executed.
3. The compensation-thread mechanism (which should unblock this) was regressed in JDK 21 for asyncMode=FIFO pools — compensation threads are not reliably created when needed.
4. This cascades for workloads with many actor hand-offs (e.g. MergeHub buffer=1: one full round-trip per element).

JDK 25 exacerbates the issue because work-stealing heuristics under high CI load diverge further from expected throughput.

---

Evidence

• ForkJoinPoolStarvationSpec has been marked pending on JDK 17+ acknowledging this class of issue
• stream-typed-tests/MapAsyncPartitionedSpec.scala:110-116 already has a comment referencing nightly tests (CI): java 25 runs have a lot of stream test failures #2573: "JDK 25 makes 1000-sample property checks noticeably more expensive"
• Nightly CI for JDK 25 was failing for 30+ consecutive days before the fix in PR fix: stabilise stream tests on JDK 25 nightly (timeout scaling, element counts) #2869

---

Mitigations Applied (PR #2869)

1. Stream test dispatcher: Changed task-peeking-mode from FIFO to LIFO in stream-testkit/src/test/resources/reference.conf. With LIFO, response tasks go to the front of each worker's deque and are processed immediately. Actor mailbox ordering is unaffected (UnboundedMailbox is always FIFO).

2. minimum-runnable now configurable: PekkoForkJoinPool hardcoded minimumRunnable=1 in the JDK 9+ ForkJoinPool constructor. This is now exposed as minimum-runnable in the fork-join-executor config block (default: 1, preserving current behaviour). Users experiencing actor starvation on JDK 21+ can increase this to e.g. parallelism/2.

3. HubSpec / AggregateWithBoundarySpec / TCK timeouts / FlowMapAsyncPartitionedSpec: test-level fixes documented in fix: stabilise stream tests on JDK 25 nightly (timeout scaling, element counts) #2869.

---

Future Work

• Investigate whether virtualize = on (JDK 21+, Pekko v1.2.0) should be the recommended mode for actor dispatchers on JDK 21+. Virtual threads avoid the FJP compensation-thread regression entirely because the scheduler is cooperative rather than preemptive.
• Consider increasing the default minimum-runnable on JDK 21+ from 1 to something like parallelism/2 to proactively create compensation threads before complete starvation.

[He-Pin]

🔬 Virtual Thread Experiment — Key Finding

We tested enabling virtualize = on on the stream test dispatcher as part of the nightly-failure investigation.

Result

HubSpec (48 tests)	Default FJP	virtualize = on
Time	2+ min, 3 failures	~20 seconds, 48/48 pass

This confirms the root cause: the FJP FIFO compensation-thread starvation is responsible for the failures, and virtual threads bypass it entirely (blocked virtual threads unmount their carrier, so no compensation threads are ever needed for actor-reply waits).

Why it's not the default (yet)

Enabling virtual threads silently in the test dispatcher would change the runtime model for all stream tests — carrier pinning, blocking detection, and FJP scheduling assumptions would differ. This could produce false positives or mask real issues.

Recommended opt-in path: add virtualize = on to your own test dispatcher config if you want to run on virtual threads. The infrastructure has been in Pekko since v1.2.0. A dedicated VirtualThreadDispatcherSpec would be the right way to validate this path explicitly.

Configuration tuning (without virtual threads)

If you cannot use JDK 21+ virtual threads, the options in order of effectiveness:

Option	Config	Effect
Higher timefactor	-Dpekko.test.timefactor=4	Wider timeout budget (already in nightly CI)
Configurable minimum-runnable	minimum-runnable = N in fork-join-executor	Prompts faster compensation-thread creation
Larger maximum-pool-size	maximum-pool-size = N	Allows more compensation threads
Reduce element counts	Test-level fix	Less work per test = less FJP pressure

[He-Pin]

@pjfanning Should we make the virtual thread executor the default in tests or nightly build?