ci: the 18 integration_tests_sv2 targets compile but never run — they need process infrastructure and a triage pass

[defenwycke]

Split out of #580, which is closed: every other integration target now runs and passes in CI. This is the
one set that does not.

Current state

integration_tests_sv2 has 18 targets. CI compiles them and does not run them:

- name: Compile SV2 integration targets (no run)
  run: cargo test -p integration_tests_sv2 --no-run

That gate is worth keeping regardless of what happens here — it catches the rot that actually bit.
tests/integration stopped compiling when f1c14cdb9 added a field to ShareConvergenceResponse and
nobody noticed for weeks; four PoolConfig initializers went the same way during the ghost-registry
deletion. Measured at 3m11s with no external services.

Why they cannot simply be un-excluded

191 call sites across the 18 targets spin up real processes — start_pool,
start_template_provider, start_sniffer — and only one of the 18 is #[ignore]d. Turning them on
as-is would make main red and flaky, which is strictly worse than not running them: a flaky blocking job
trains everyone to ignore it, and then it stops catching anything at all.

What would need to exist

1. Process lifecycle that is reliable in CI. Each target needs a pool, a template provider and
   sometimes a sniffer, on ports that do not collide when targets run concurrently. Today they assume
   they can bind and that nothing else is using the port.
2. A per-test time budget. These wait on real handshakes. The integration-tests job already sits
   inside a 45-minute timeout and was cancelled once for exceeding it when merged into test.
3. A triage pass, target by target. CI never runs 13 of 14 integration test targets — payout_ledger_e2e has been red on main unnoticed #580's lesson is that switching a job to blocking before
   triaging produces a red main and a job people disable. Expect some of the 18 to be stale in the same
   way payout_ledger_e2e was — a fixture that no longer matches a rule, rather than a live defect.
4. A decision on what runs where. Plausibly: the cheap ones blocking on every push, the
   process-heavy ones on a schedule or on demand. Not everything has to gate every commit to be worth
   running.

Why it matters

These are the only tests that exercise the real SV2 path end to end — the path every miner on the pool
actually uses. pool_sv2 not binding :34255 until its TDP handshake completes, the extranonce2_size
change, the PoW job-target-vs-channel-target bug (#583): all of that lives in territory only these
targets cover, and all of it was found in production rather than in CI.

Not urgent for v1 — the compile gate removes the silent-rot failure mode, which was the sharp edge. This
is the remaining depth.

[defenwycke]

Triage done — the first measurement of what these 18 targets actually do

Ran each target individually with a 180s cap. 17 of 18 complete
(translator_integration still running at time of writing).

verdict	count	targets
genuinely passes	2	mining_device_fast_hasher_equivalence (1 test), template_provider_integration (3 tests)
passes VACUOUSLY	2	bitcoin_core_ipc_integration (0 run, 3 ignored), jds_block_propagation (0 run, 1 ignored)
fails	3	jd_provide_missing_transaction, jdc_receives_submit_shares_success, sv1
hangs to 180s	10	extensions, jd_integration, jd_tproxy_integration, jdc_block_propagation, jdc_cached_shares, monitoring_integration, pool_integration, pool_solo_mining, sniffer_integration, sv2_mining_device

Three corrections to the issue body

1. "only ONE of the 18 is #[ignore]d" — it is 4 attributes across 2 files.
bitcoin_core_ipc_integration.rs has 3, which is every test in it. That target reports
test result: ok while running nothing, so a summary line counts it as healthy. It is disabled.
jds_block_propagation.rs has 1 and is in the same state.

⚠ So a naive cargo test would report 4 passing targets. Two of them execute zero tests.

2. The dominant failure mode is HANGING, not failing. Ten of seventeen hit the cap. That changes
the shape of the work: a per-test time budget is not a refinement, it is the difference between a
red job and a job that never finishes. A 45-minute CI timeout with ten hanging targets is a
cancelled run, which #685 already showed reports as cancelled, not failure.

3. Not all 18 need process infrastructure. mining_device_fast_hasher_equivalence passes in
0.01s with no processes at all, and template_provider_integration passes in 6.8s. The
191-call-site framing is accurate in aggregate and hides that a couple are already CI-ready today.

One real bug found and fixed — PR #762

tests/sv1.rs was not failing for infrastructure reasons. Sv1Sniffer::start bound its port
directly instead of claiming the reservation that get_available_address() holds, so both its tests
died on Os { code: 98, kind: AddrInUse }. #612 fixed exactly this in the SV2 Sniffer::start
and missed the SV1 one.

After the fix AddrInUse is gone and the target fails one layer deeper, on
pool_sv2::template_receiver::sv2_tp: Noise handshake failed error=HandshakeTimeout — the pool
cannot reach a template provider. That is the infrastructure gap this issue describes, and it
was masked by a bug that had nothing to do with it.

⚠ The fix does NOT make the target pass. It removes something that was hiding the real problem.

What this suggests for sequencing

The issue proposes triage as step 3. On this evidence it belongs first, because it changes the
estimate: two targets are ready now, two are disabled and should either be deleted or un-ignored
with a reason, and the ten that hang share one likely cause — no template provider — which may be
one piece of infrastructure rather than ten separate problems.

Raw log: 180s cap, one target at a time, no external services beyond what each target starts itself.

[defenwycke]

Addendum: the triage harness itself hung, which is the same hazard

translator_integration — the eighteenth target — never produced a result. Not because the test
hung: because the harness did.

The triage script wrapped each target as out=$(timeout 180 cargo test -p integration_tests_sv2 --test "$t" 2>&1). On the last target that shell sat for two hours against a 180-second cap,
and when inspected it had no children at all — no cargo, no pool_sv2, no template provider,
nothing reparented to init. A childless bash with its stdout on a plain file, waiting on nothing
observable.

So timeout did its job on the process it could see, and the wrapper still did not return.

That matters for this issue's plan more than the missing data point does. The proposal is to give
these targets "a per-test time budget" — and this is direct evidence that a naive timeout around
cargo test is not one. In CI that shape does not produce a failed job, it produces a job that runs
until the workflow's own 45-minute limit and reports as cancelled, not failure (#685).

Whatever budget these get needs to be enforced somewhere that cannot be outlived — --test-threads
plus a harness-level timeout inside the test process, or a job-level step timeout, not a shell
wrapper around the whole invocation.

Revised triage totals: 17 of 18 measured, 1 unmeasured. The verdict table in the previous
comment stands; translator_integration should be read as unknown rather than absent.

[defenwycke]

Ran pool_integration — one of the ten — against today's main, and it does not hang. It aborts the process, and the causal chain is worth writing down because it is structural rather than specific to this target.

translator_sv2::upstream: Connected to upstream at 127.0.0.1:40261
  ... 5.015s later ...
stratum_apps::network_helpers::noise_stream: ERROR Handshake failed with upstream: HandshakeTimeout
panicked at tests/integration-sv2/lib/sniffer.rs:150: Failed to create upstream
panicked at crates/stratum-apps/src/fallback_coordinator.rs:110: FallbackHandler dropped without calling done()
<raw backtrace, process gone>

1. The trigger is a 5-second Noise handshake timeout. Connected at :29.376, timed out at :34.391. Not a TCP failure — the socket connects and the handshake never completes.

⚠ The template provider is NOT the missing piece here. The earlier triage suspected "no template provider" as the shared cause. In this run the TP is demonstrably up: pool_sv2::template_receiver::sv2_tp: Connecting to template provider followed by Sending SetupConnection message to the Template Provider, and both bitcoind (bitcoin-30.2) and sv2-tp-1.0.3 are present and launched. The handshake that fails is the sniffer → pool one, further down the chain.

2. The failure cannot fail the test, because it happens in a detached task. Sniffer::start does its whole lifecycle inside tokio::spawn, and the upstream failure is an .expect() in there (sniffer.rs:150). A panic in a spawned task does not propagate to the test — the JoinHandle is dropped and the panic is merely printed. The test then waits for a message that can never arrive.

That is the hang mechanism, and every sniffer-based target inherits it. It explains why the dominant failure mode is hanging rather than failing: the harness is structurally incapable of reporting this class of error as a test failure.

3. A panicking Drop turns it into a process abort. Unwinding that spawned task drops a FallbackHandler, whose Drop impl panics if done() was not called (fallback_coordinator.rs:107-112). Panicking inside Drop during an unwind is a double panic, which aborts. So the binary dies with no test result line at all — no pass, no fail, and the other 8 tests in the target never run.

⚠ This means a per-test time budget is necessary but NOT sufficient. A timeout cannot rescue a process that aborts, and an abort with no test result is exactly the shape that reads as a cancelled job rather than a failure (#685).

⚠ Panicking in Drop is worth looking at on its own merits, independent of these tests. It converts any unwind on that path into an abort, in production code, not just here.

What this changes about the plan

The issue's step 3 (triage, target by target) is likely to be much cheaper than 10 separate investigations, but only after two structural fixes, in this order:

1. Make the sniffer's failure reachable by the test — surface it through a channel or a shared error slot instead of .expect() in a detached task. Ten hangs become ten fast, legible failures.
2. Stop FallbackHandler::drop panicking, so a failure produces a test failure rather than an abort.

Only then is "why does target X fail" a question worth asking per target — and some of the ten may well turn out to share the single handshake-timeout cause.

⚠ Note the triage above recorded these as "hangs to 180s" on 2026-08-23 while this run aborted at 66s. #762 (the SV1 sniffer port-claim fix) landed in between, and machine load differs, so I would not assume all ten behave identically today. The structural points hold regardless of which of the two shapes a given target takes.

Raw log kept: 396 lines, RUST_LOG=info, one target, --test-threads=1.

[defenwycke]

Root cause found, and it explains the whole cluster

The template provider never binds its sv2 port. The test harness is holding it.

Proof, taken live while sniffer_integration was running:

$ pgrep -a sv2-tp
135927 sv2-tp                      <- the process IS running

$ ss -ltnp | grep -E "sv2-tp|sniffer_integ"
LISTEN 0 128 127.0.0.1:40213  users:(("sniffer_integra",pid=135875,fd=11))
LISTEN 1 128 127.0.0.1:45077  users:(("sniffer_integra",pid=135875,fd=9))

sv2-tp owns no listening socket at all. Both ports belong to the test binary. And note Recv-Q = 1 on 45077 — a completed connection sitting in the backlog that nobody will ever accept.

Why

get_available_address() binds a real TcpListener and keeps it in the RESERVED registry (lib/utils.rs:56). That is deliberate and correct for Rust components: they later call claim_listener() to take ownership of the reserved socket, which is what #612 introduced to close the probe-and-release race from #408.

But sv2-tp is an external process. It cannot be handed a Rust TcpListener, and start_template_provider never releases the reservation — template_provider.rs contains no claim_listener and no release. So the port stays owned by an idle listener in the test binary for the life of the test, and sv2-tp -sv2port=P fails to bind it.

A dial then succeeds at TCP level, because the reservation listener is listening with a backlog, and hangs forever at the protocol level because nothing accepts or reads. That surfaces as HandshakeTimeout after exactly NOISE_HANDSHAKE_TIMEOUT (10s).

Why it stayed invisible

Two things hide it, and both are worth fixing regardless:

• sv2-tp's output is discarded — .stdout(Stdio::null()).stderr(Stdio::null()) (template_provider.rs:362-363). Whatever it says about failing to bind goes nowhere.
• Readiness is a blind std::thread::sleep(3) with the comment "Wait for sv2-tp to start and connect to Bitcoin Core". Nothing ever checks that it is listening, so a TP that never bound is indistinguishable from one that did.

This explains the pass/fail split exactly

target	sniffers	dials the TP's sv2 port?	result
template_provider_integration	0	no — let (tp, _) = start_template_provider(...), address discarded	passes
mining_device_fast_hasher_equivalence	0	no processes at all	passes
the other 16	1–18	yes, directly or through a sniffer	fail / hang

The two targets that pass are precisely the two that never dial that port. Every target that does — whether the pool dials it directly or through a sniffer — meets a socket nobody is serving.

⚠ So the sniffer is not the culprit, despite being the perfect correlate of the failures. It is simply the component that dials the dead port most often. My earlier comment framed the sniffer's ordering as the open question; it is not — the sniffer behaves correctly, it just never gets an answer.

Fix shape

1. Release the reservation before spawning an external process. Rust components claim the listener; external ones need it dropped. The window this reopens is closed by (2), not by holding the port.
2. Replace the sleep(3) with a real readiness poll — dial the sv2 port until it answers, with a deadline. That both closes the race and makes a TP that failed to start a loud, immediate failure instead of a 10-second handshake timeout ten layers away.
3. Stop discarding sv2-tp's stderr. A process whose diagnostics go to /dev/null cannot tell you why it did not start.

That is one piece of infrastructure, as the earlier triage suspected — just not the one it guessed.

[defenwycke]

Fixed on main in 46b2f434f, with the measured effect and one correction to my own analysis.

What changed

• The reservation is released before sv2-tp is spawned, so it can bind the port it is told to use.
• The blind sleep(3) is now a readiness poll that fails loudly, naming the port and pointing at the stderr log.
• sv2-tp's stderr is captured to sv2-tp.stderr.log in the node datadir instead of /dev/null.
• get_available_address no longer reissues a port within a process, since callers derive per-port state from it (BitcoinCore's datadir is .bitcoin-{port}).

⚠ Releasing reopens the window claim_listener exists to close. What closes it now is observing the child actually accepting — holding the port never could, because holding it is what broke the child in the first place. Both mechanisms are needed and the comments say which hazard each addresses.

Measured effect, clean state each run

target	before (08-23 triage)	after
sniffer_integration	hangs to 180s	ok. 2 passed, 10.3s
jdc_cached_shares	hangs	FAILED in 12s — diagnosable
jdc_block_propagation	hangs	FAILED in 69s — diagnosable
template_provider_integration	2 passed, 1 failed	2 passed, 1 failed — unchanged
8 others	hangs	still exceed a 240s cap

A correction

I initially reported that this change regressed template_provider_integration from 3 passing to 2, and gave a confident cause (port reuse colliding on the .bitcoin-{port} datadir). That was wrong.

Three checks settled it: adding the port-reuse guard did not help; wiping all 33 stale .bitcoin-{port} datadirs did not help; and stashing every change and rebuilding the baseline produced the identical FAILED. 2 passed; 1 failed. tp_high_diff fails on this machine with or without the change — it is pre-existing and environment-dependent, most likely the cached high_diff_chain / signet datadir, since the 08-23 triage recorded 3 passing.

The port-reuse guard is kept, but re-justified honestly in the code: it guards a hazard that releasing creates and which has not been observed failing.

Why eight targets still exceed the cap

Not deadlock, arithmetic. wait_for_message_type panics at DEFAULT_TIMEOUT = 60s in the test's own thread — a proper failure. A target with nine failing tests therefore needs ~9 minutes, so any 240s cap kills it regardless of whether each test now fails cleanly. That is why the original triage's 180s cap reported "hangs": they were timing out legitimately, three tests in.

⚠ So the remaining work is no longer "why do ten targets hang". It is per-target triage against a working template provider, with a cap that reflects tests × 60s. Whether the tests pass once they can reach a live TP is a genuinely open question — sniffer_integration did, and the two jdc_* targets still fail, now for reasons that are visible.

[defenwycke]

A fast reproduction of the remaining hang, and where I stopped

The three infrastructure fixes are in (ce2075b52, 46b2f434f, 8afbade6c). One hang survives them, and this narrows it a long way.

It is not the components, and not a bare panic

I bisected by starting one more component at a time and returning immediately. All exit cleanly:

case	components	result
1	TP only	ok, 4s
2	+ sniffer + pool	ok, 5s
3	+ second sniffer	ok, 5s
4	+ translator	ok, 8s
5	+ minerd (full set)	ok, 8s
6	full set, then panic!("deliberate")	ok, 7.7s

So neither starting/dropping the whole stack nor unwinding through it is enough to hang.

It reproduces in ~40s from inside wait_for_message_type

#[tokio::test]
#[should_panic]
async fn repro() {
    let (_tp, tp_addr) = start_template_provider(None, DifficultyLevel::Low);
    let (sniffer, sniffer_addr) = start_sniffer("tp", tp_addr, false, vec![], Some(5));
    let (_pool, pool_addr, _) = start_pool(sv2_tp_config(sniffer_addr), vec![], vec![], false).await;
    let (_s2, s2_addr) = start_sniffer("pool", pool_addr, false, vec![], Some(5));
    let (_tr, tproxy, _) = start_sv2_translator(&[s2_addr], false, vec![], vec![], None, false).await;
    let (_minerd, _) = start_minerd(tproxy, None, None, false).await;
    // 0xFF is not a real message type, so this can only end in the timeout panic.
    sniffer.wait_for_message_type(MessageDirection::ToDownstream, 0xFF).await;
}

With a 5-second sniffer timeout this is still alive at 42s, thread pinned in R, ~64% CPU, /proc/<tid>/syscall empty. That is a ~40s reproduction of something that otherwise costs 10+ minutes per target to observe.

⛔ Do not commit that test. It hangs, so it would wedge CI. It is a debugging recipe, not a test.

What distinguishes the hanging case

The only differences from case 6 are that the panic originates inside wait_for_message_type — which takes a blocking std::sync::Mutex across a spawn_blocking — and that minerd has had time to connect and start pushing shares.

Under strace -f the tail is a continuous flood of successful recvfrom(fd, …) = 1388 with the buffer shrinking then resetting: that is read_frame relaying live minerd traffic. So after the test panics, the components have not quiesced — real I/O is still flowing while another thread spins in userspace.

That suggests the hang needs in-flight traffic at the moment of the panic, not merely a panic. Two candidates worth checking next, in order:

1. A spawn_blocking closure holding the aggregator's std::sync::Mutex when the panic unwinds — poisoning it, or leaving a blocking-pool thread that runtime shutdown then waits on.
2. The sniffer's relay select!, whose two recv arms never complete while their channels stay open and traffic keeps arriving, so the task can never finish for shutdown.

Why I stopped here

The spin makes no syscalls, so strace cannot point at it, and ptrace_scope=1 with no passwordless sudo means no debugger on an already-running process — gdb, lldb and perf are all absent. Localising it further wants either a debugger or targeted instrumentation, and guessing further would be exactly the "confident cause" mistake I already made once in this thread.

⚠ 8afbade6c guards a different, provable infinite loop in read_frame (writable_len() == 0 can never make progress). It is worth having on its own merits and it ships in production code, but it did not fix this hang and its commit message says so.

[defenwycke]

The hang is an unbounded spin inside Drop, in vendored SV2 code

Captured from the spinning thread itself. Since the spin makes no syscalls (strace blind) and ptrace_scope=1 blocks attaching a debugger, I installed a SIGUSR1 handler that dumps Backtrace::force_capture(), then used tgkill(2) to deliver the signal to the spinning TID specifically — kill(pid) lands on the main harness thread, which is merely waiting for the test to finish.

 2: core::core_arch::x86::sse2::_mm_pause
 3: core::hint::spin_loop
 4: <buffer_sv2::buffer_pool::BufferPool<...> as Drop>::drop
        at vendor/stratum/sv2/buffer-sv2/src/buffer_pool/mod.rs:926:13
 5: drop_glue::<BufferPool<...>>
 6: drop_glue::<codec_sv2::decoder::WithNoise<BufferPool<...>>>
 7: drop_glue::<NoiseTcpReadHalf<AnyMessage>>
 8: drop_glue::<Connection::spawn_reader<...>>
16: tokio::runtime::task::harness::cancel_task::<...>

The code:

#[cfg(not(test))]
impl<T: Buffer> Drop for BufferPool<T> {
    // Waits until all slices are released before dropping the `BufferPool`.
    fn drop(&mut self) {
        while self.shared_state.load(Ordering::Relaxed) != 0 {
            core::hint::spin_loop();
        }
    }
}

A Drop that can block forever, with no bound, no yield and no escape. When a reader task is cancelled during runtime shutdown, its NoiseTcpReadHalf — and with it the decoder's BufferPool — is dropped while a slice is still outstanding. shared_state never reaches zero, and the thread spins at 100% CPU making no syscalls, which is exactly why every observation tool came back empty.

⚠ #[cfg(not(test))] means this impl is not even compiled for that crate's own tests, so its test suite is structurally incapable of catching it.

Why every earlier theory missed

The spin is in a Drop reached through cancel_task, so it is invisible to reasoning about the test's own control flow. Four theories died on measurement before this: the sniffer's ordering, the panic site, a writable_len() == 0 decoder stall, and a discarded relay JoinHandle. Each was plausible; none survived.

Not test-only

pool_sv2 and translator_sv2 build noise connections on these buffer pools. A production node that drops a connection with a frame still outstanding spins a core forever, silently — no log line, no syscall, nothing for strace. Worth treating as a production hazard, not a test-harness quirk.

⚠ The obvious fix is unsafe

Bounding the spin and giving up would risk a use-after-free: the pool owns the memory those outstanding slices point into, which is why it waits. A safe fix is one of:

1. Leak instead of hang — after a bounded wait, forget the backing allocation rather than free it. Trades a bounded leak for a guaranteed-terminating drop.
2. Fix the ordering — ensure no slice outlives the pool: drain or drop the frame channels before the connection halves. That is in our noise_connection, not vendored code, and is the more principled fix.
3. Report upstream — a Drop that can spin forever is a defect in buffer-sv2 regardless of how we work around it here.

I am not changing vendored memory management on a money-path codebase without more care than the end of a session allows, so I am stopping at the diagnosis. Option 2 is where I would start.

[defenwycke]

Step 1 answered: the slice is held by a frame queued in the connection's own inbound channel

Three probes, each disabling one candidate holder and asking only "does it still hang":

probe	disabled	result
A	MessagesAggregator retains nothing	still hangs
B	sniffer forwards no frames across the relay	still hangs
C	no frame escapes the connection at all	TERMINATES

Probe C is the positive: with frames dropped instead of sent into sender_incoming, the target completes (FAILED. 0 passed — expected, since dropping every frame cannot pass — but it exits, which is the signal).

The mechanism

Connection::spawn_reader (crates/stratum-apps/src/network_helpers/noise_connection.rs:67) owns read_half, and with it the decoder and its BufferPool, while pushing decoded frames out through sender_incoming:

task::spawn(async move {
    loop { tokio::select! {
        res = read_half.read_frame() => match res {
            Ok(frame) => { if sender_incoming.send(frame).await.is_err() { ... } }

On cancellation the task's future is dropped, which drops read_half and frees the pool. Any frame still queued in that channel holds a Slice pointing into the freed memory — Slice.owned is None in pool mode, so the slice does not own its bytes. shared_state never reaches zero and BufferPool::drop spins forever.

⚠ Two hypotheses died here, both of which I would have shipped a fix for on reasoning alone. The aggregator stores AnyMessage<'static> and I expected that 'static to be hiding a pool-backed borrow — it is not. And frames crossing the relay between two connections looked like the obvious culprit given every hanging target uses a sniffer — also not.

What this means for the fix

It is in our code, not the vendored crate. The plan's Step 2 applies and Step 3 (changing BufferPool::drop) is no longer required to resolve this, though it remains worth doing as defence in depth, since a Drop that can spin forever will find another ordering bug eventually.

The invariant to restore: the read half must outlive every frame it produced. Options, roughly in order of preference:

1. Do not let the cancellable task own the pool. Hold read_half in the ConnectionState so cancelling the reader does not free the pool; the pool then drops with the connection, after the channels. Ordering fix, no copy.
2. Graceful shutdown instead of cancellation — signal the reader to stop, drain the channel, then drop. Correct but needs every call site to stop relying on abrupt cancellation.
3. Copy frames into owned buffers at the channel boundary. Guarantees no slice escapes, but adds a copy per SV2 message on the hot path — for a mining pool that is a real cost and I would not choose it without measuring.

⚠ This is production code in pool_sv2 and translator_sv2, so whichever option is taken wants review rather than a quick commit. Not before the payout gate.