CI never runs 13 of 14 integration test targets — payout_ledger_e2e has been red on main unnoticed

[defenwycke]

13 of 14 integration test targets never run in CI — and two are red on main right now

Found while verifying #578. bins/ghost-pool/tests/payout_ledger_e2e.rs has 2 failing tests on
unmodified main (9b7e356d6), and CI is green, because CI never runs that file.

What CI runs

.github/workflows/ci.yml:

- name: Run unit tests
  run: cargo test $WORKSPACE_EXCLUDES --lib --bins     # excludes tests/ targets entirely

- name: Run integration tests
  run: cargo test --test integration                   # ONE named target

--lib --bins does not build tests/ integration targets, and --test integration names a single
one. Everything else is invisible.

What exists

bins/ghost-pool/tests/empty_template_e2e.rs
bins/ghost-pool/tests/fleet_rehearsal.rs
bins/ghost-pool/tests/payout_ledger_e2e.rs          <- 2 failing on main
bins/wraith-coordinator/tests/bond_e2e.rs
bins/wraith-coordinator/tests/gossip.rs
bins/wraith-coordinator/tests/router.rs
crates/ghost-consensus/tests/mesh_onboarding_fanout.rs
crates/ghost-consensus/tests/mpc_mesh_dispatch.rs
crates/ghost-mpc/tests/mpc_lifecycle.rs
crates/ghost-reaper/tests/backtest_computational_only.rs
crates/ghost-reaper/tests/backtest_mainnet.rs
crates/ghost-reaper/tests/backtest_simulator_delta.rs
crates/ghost-verification/tests/identity_tls_handshake.rs
crates/mpc-xproc-harness/tests/xproc.rs

These are the highest-value tests in the repo — end-to-end payout ratification, mesh onboarding,
MPC lifecycle, reaper backtests, TLS identity — and they are exactly the ones not gating anything.

This is the same class as #530 (feature-gated modules compiled but untested), which the CI file
itself documents. The fix there named the missing features; the integration targets were missed.

The two failures (triaged — NOT a production bug)

block_win_is_ratified_and_pays_every_miner_the_ledger_owes
share_arriving_after_the_cutoff_does_not_break_ratification

GHOST-02: 0/8 nodes approved, quorum needs 6 — the fleet would reject its own
payout and the coinbase would fall back to paying pool_payout_address

Alarming on its face — this is the coinbase-split approval path that runs when the pool wins a
block, and per #556 the fleet has not won one since 2026-06-02, so production has not exercised it
in two months.

It is a stale fixture, not a live defect:

node rejected the payout: pre-gate proposal timestamp 1785476925 not within 1800s of now 1785480528

The test builds its cutoff at Utc::now() - 3_600 (line 113) while
PRE_GATE_FRESHNESS_SECS = 1800 (payout.rs:432) rejects anything older than 30 minutes. The
fixture predates that rule. Production proposals are stamped fresh, so the live path is unaffected.

I also verified it is not gate-related: still fails with BLOCK_HEIGHT raised above
PAYOUT_TOLERANCE_V2_HEIGHT (960_106).

Suggested fix

1. Run all integration targets: cargo test $WORKSPACE_EXCLUDES --tests (or --all-targets),
   rather than one named target.
2. Fix the payout_ledger_e2e fixture — cutoff inside the 1800 s freshness window.
3. Expect others in that list to be red too; they have not gated anything in a long time. Worth
   triaging before switching the job to blocking, so it can go green and stay green.

The general rule this keeps violating: a check that never runs cannot fail. Same root as #555,
#563, #564 and #530.

[defenwycke]

Landed in #581 — partially. SV2 triage still open, so leaving this open.

Merged as 29a1988ab. CI now runs the integration targets in a dedicated parallel job.

What is covered now

job	duration	contents
Integration tests	23m18s	12 targets — payout ratification, mesh onboarding, TLS identity, coordinator gossip/router, bond_e2e, the reaper #[ignore]d no-ops
MPC integration	13m49s	mpc_lifecycle + xproc under --features mpc-test-cap

payout_ledger_e2e is fixed and green — the fixture stamped its proposal at now - 3_600 against a
PRE_GATE_FRESHNESS_SECS = 1800 rule added later. Not a production defect; verified separately that
it is not gate-related.

A mistake worth recording

The first attempt added the integration run as a step on the existing test job. That pushed it
from ~28 min past its 45-minute timeout and GitHub cancelled it mid-run — CI red for a budget
reason, not a test reason. Moving it to a separate parallel job fixed it: Test (ubuntu-latest) is
back to 25m12s, unchanged from before.

Worth remembering: adding coverage to an existing job spends that job's time budget. A parallel
job costs nothing on the critical path.

Also worth recording

My first triage enumerated integration targets by globbing bins/*/tests and crates/*/tests and
found 14. cargo metadata --no-deps reports 47 — three packages sit outside those two trees.
Had the blanket --tests shipped on that assumption it would have pulled in
integration_tests_sv2, which spawns real pool processes. Ask the build system, not the
filesystem.

Still open — why this issue stays open

integration_tests_sv2: 18 targets, never run in CI, untriaged.

bitcoin_core_ipc_integration   jd_integration                 jdc_cached_shares
extensions                     jd_provide_missing_transaction jdc_receives_submit_shares_success
jd_tproxy_integration          jdc_block_propagation          jds_block_propagation
mining_device_fast_hasher_equivalence  monitoring_integration  pool_integration
pool_solo_mining               sniffer_integration            sv1
sv2_mining_device              template_provider_integration  translator_integration

They spin up real SRI pool / template-provider processes — 58 start_pool call sites, only one
#[ignore]d — so they need external services and a per-test time budget. Given how much of the SV2
path is live in production (translator, JD, sniffer, monitoring), these are worth having, but wiring
them needs its own design: what to stand up, how long to allow, and whether they gate or report.

Unknown until triaged whether any are currently red.

[defenwycke]

Done and verified by outcome on main (9a70982b5, CI run 30888590602 — all 11 jobs green).

1. All integration targets now run

A dedicated integration-tests job runs cargo test $WORKSPACE_EXCLUDES --tests. Extracted from that
job's own log, 28 integration targets ran, 0 failures (67 test result: ok):

backtest_computational_only  backtest_mainnet     backtest_simulator_delta  bond_e2e
chain_scan_utxos             check_for_update     cluster_chaos             empty_template_e2e
fleet_rehearsal              gossip               gsp_mock                  identity_tls_handshake
integration                  l2_transfer_e2e      load_tests                lock_recovery_e2e
mainnet_guards               mesh_onboarding_fanout  mpc_mesh_dispatch      payout_ledger_e2e
peer_rotation                prepare_lock_e2e     regtest_settlement_rehearsal  router
wallet_lifecycle             watch_payments       watch_payments_daemon     wraith_e2e

The two targets excluded from that job (mpc_lifecycle, xproc) need a per-package feature a
workspace-wide --tests cannot set, so they run in the MPC integration job instead — also green, both
present in its log. Nothing from the list in this issue is uncovered.

It is a separate job rather than a step on test deliberately: adding it there pushed that job past its
45-minute timeout and it was cancelled mid-run. In parallel it cannot slow the critical path.

2. payout_ledger_e2e is green

Locally, all five including the two this issue reported red:

test ghost02_round_scoped_recompute_is_rejected ... ok
test share_arriving_after_the_cutoff_does_not_break_ratification ... ok
test block_win_is_ratified_and_pays_every_miner_the_ledger_owes ... ok
test mined_block_is_accepted_by_ghostd_and_pays_the_miners ... ok
test tip_change_arms_the_coinbase_and_leaves_the_ledger_owed ... ok
test result: ok. 5 passed; 0 failed

The triage in this issue was right: a stale fixture (cutoff built outside PRE_GATE_FRESHNESS_SECS),
not a live payout defect.

3. The rest were triaged, not just switched on

They pass, which is why the job could be made blocking rather than advisory. That was the condition this
issue set — "worth triaging before switching the job to blocking, so it can go green and stay green".

What is NOT done, and why it is its own issue

The 18 integration_tests_sv2 targets are compile-gated only — cargo test -p integration_tests_sv2 --no-run, measured at 3m11s with no external services. They cannot be run here: 191
start_pool/start_template_provider/start_sniffer call sites spin up real SRI processes and only one
of the 18 is #[ignore]d. Un-excluding them would make main red and flaky, which is worse than not
running them.

The compile gate is worth having on its own — it catches the rot that actually bit. tests/integration
stopped compiling when f1c14cdb9 added a field to ShareConvergenceResponse and nobody noticed for
weeks, and four PoolConfig initializers went the same way during the ghost-registry deletion.

Running them for real needs CI infrastructure for the SRI processes, which is a different piece of work
with a different shape. Tracked separately so this issue closes on what it actually asked for.

Closing. The general rule it was filed against — a check that never runs cannot fail — now holds for
every integration target except the sv2 set, which at least cannot silently stop compiling.