PreparedModeAuthenticationFunctionalTest hangs intermittently on V2/V2, silently consuming the job timeout

[jbachorik]

Summary

PreparedModeAuthenticationFunctionalTest intermittently hangs on its V2/V2 parameterisations,
consuming the full 30-minute job budget and producing no failing assertion, no stack trace, and
several orphaned JVMs. Seen on test (17.0.19-tem) in CI; not reproduced locally in one attempt on
17.0.14.

Evidence

Same PR, three runs of the same tier:

Run	Result
30381203688 job 90349867449	hung, cancelled at 30m
30381203688 job 90359031612 (re-run)	hung, cancelled at 30m
30391315946	passed in 4m47s

develop passes the same tier in 4m16s, so ~4–5 minutes is the normal duration and 30 minutes is a
stall rather than slowness.

It is not a fixed case. The first run stalled on client v2, server v2, auto=false, supplied token=true; the re-run passed that one and stalled on a later V2/V2 parameterisation, on a
different executor thread.

Last output before the silence, both times:

PreparedModeAuthenticationFunctionalTest > ... > client v2, server v2, ...
    [pool-6-thread-1] INFO io.btrace.client.Client - Successfully started BTrace probe: OnTimerArgTest.java
                                   ← ~25 minutes of nothing
##[error]The operation was canceled.

Job cleanup then reports:

Terminate orphan process: pid (3339) (java)      ... seven of these

So authentication, negotiation and probe submission all completed. The stall is after that.

Why this is worse than a normal flake

The failure is silent. There is no assertion failure and no thread dump, so localising it required
reading the raw job log and comparing timestamps. Every occurrence costs the whole job budget.

It also reaches the release path: release-smoke runs after the operator has published to Maven
Central and finalize-tag depends on it, so a stall there leaves artifacts on Central with no tag,
no GitHub release and no SDKMAN update — the same shape as a failure, but with no diagnostic.

Unverified hypothesis

Offered as a starting point, not a conclusion — it has not been reproduced.

Every wait inside assertProbeRoundTrip is bounded (started.get(20s), submission.get(10s)), so
a 25-minute silence cannot be the test awaiting an assertion. The probe submission runs on
Executors.newSingleThreadExecutor(), whose threads are non-daemon, and executor.shutdownNow()
interrupts — but a thread blocked in socket I/O does not respond to interrupt. A surviving
non-daemon thread would keep the test JVM alive, so the Gradle worker never exits and the build
stalls with nothing to report. The orphaned JVMs at cleanup are consistent with this.

Suggested changes

Independent of root cause:

1. Per-test timeout in the integration harness, so a stall becomes a fast, attributable failure
   with a stack dump naming the blocked thread instead of a silent 30-minute job.
2. Daemon threads for the test executors, so a thread stuck in socket I/O cannot hold the test
   JVM open.
3. Consider a thread dump on timeout, which would have made this a one-line diagnosis.

[jbachorik]

Local reproduction attempt: 14 clean passes, no stall

Ran the class in a watchdogged loop on macOS: 8 iterations unloaded, 4 with the box deliberately
oversubscribed (20 spinners on 12 cores, to emulate a 2-core runner), plus 2 further passes.
14 runs, 6/6 tests passing every time, zero stalls. Durations 71–252 s. No dumps were captured
because nothing stalled.

Correction: this is not a "JDK 17" problem

TEST_JAVA_HOME selects only the traced application JVM (RuntimeTest.resolveTestJavaHome).
The test worker running the btrace client is pinned separately to Java 24 by setup-java. So
the job named test (17.0.19-tem) runs the client on Java 24 and only the target app on 17.0.19 —
the matrix name is about the tracee, not the code that hangs. The earlier framing in this issue was
misleading on that point.

The hypothesis in the description is refuted as the stall point

Its premise is correct — a healthy thread dump confirms pool-N-thread-1 carries no daemon
marker, so such a thread would keep the worker JVM alive. But the dump also shows pools 1–5 had
already terminated while pool 6 ran, so in the healthy path client.close() + shutdownNow() do
reap them reliably; they do not accumulate.

More decisively, the CI log shape rules it out. Gradle emits a PASSED line as each test
completes. If a lingering non-daemon thread were the problem, the parameterisation would be
reported PASSED first and the stall would follow the class. The CI log shows silence immediately
after Successfully started BTrace probe with no PASSED line, so the test method never
returned. The hang is inside the method.

The bounded waits are also excluded: submission.get(10, TimeUnit.SECONDS) catches only
ExecutionException, so a TimeoutException would propagate and fail loudly.

That leaves the finally block.

Better candidates

1. client.close() deadlocking against the blocked reader. After negotiation,
Client.createV2Protocol restores the socket's previous timeout in a finally — and the default
is 0, i.e. infinite. So commandLoop → BinaryWireProtocol.read → BinaryWireIO.read blocks
with no timeout, which is exactly why only the V2/V2 cases are affected. BinaryWireProtocol.close()
then closes the streams from the main test thread; NioSocketImpl.close() marks the socket closing,
signals the reader, and waits on stateLock until that reader exits. If the wake-up does not land,
the two threads wait on each other inside one JVM.

2. TestApp.stop()'s unbounded process.waitFor(). If the traced app does not exit, this never
returns — and unlike a single lingering worker thread, it accounts for the seven orphaned
java processes seen at CI cleanup.

Both sit inside the test method, so both match the log. A single thread dump taken during a real
stall would separate them immediately.

Suggested next step for reproduction

The untested axis is platform. NioSocketImpl's wake-up of a blocked reader uses different
native mechanics on Linux and macOS, and that is precisely the code path implicated above. Further
macOS iterations look unpromising; a Linux container is the next thing worth trying.

The harness changes suggested in the description are unaffected by any of this, and a thread dump
on timeout would have made the whole question a one-liner.