Skip to content

test(jobs): fix DB-subset pg pool exhaustion in jobs_stream SSE tests - #196

Merged
TaprootFreak merged 1 commit into
stagingfrom
fix/db-subset-jobs-stream-pool
Jun 4, 2026
Merged

test(jobs): fix DB-subset pg pool exhaustion in jobs_stream SSE tests#196
TaprootFreak merged 1 commit into
stagingfrom
fix/db-subset-jobs-stream-pool

Conversation

@TaprootFreak

Copy link
Copy Markdown
Contributor

Problem

The DB Subset Tests (M3 Ultra) CI job (ci:db label, narrow nextest selection, --test-threads 8) intermittently fails the five SSE tests with create: PoolTimedOut, each running >100 s before timing out:

  • router::tests::jobs_endpoint_tests::jobs_stream_404_for_unknown_id
  • router::tests::jobs_endpoint_tests::jobs_stream_closes_immediately_for_terminal_job
  • router::tests::jobs_endpoint_tests::jobs_stream_emits_initial_phase_for_non_terminal_job
  • router::tests::jobs_endpoint_tests::jobs_stream_failed_terminal_closes_with_complete_and_error
  • router::tests::jobs_endpoint_tests::jobs_stream_forwards_dispatcher_phase_transition

The full coverage gate (same --test-threads 8, same SHA) keeps these tests green, so the tests themselves are sound.

Root cause

Migration-replay contention, not raw connection exhaustion. Every test that calls crate::test_db::setup_pool() creates a fresh per-test schema and replays the full migration suite (16 DDL files: tables, triggers, views) into the single shared postgres:17 container. Postgres serialises concurrent DDL on its system catalogs, so when the DB subset packs the migration-replaying tests together and eight run at once, each setup_pool() stretches from <1 s to tens of seconds.

The jobs_stream_* tests additionally hold their pool across deliberate sleep/timeout windows, so under that contention their connection acquisition exceeds the pool's 60 s acquire_timeout and surfaces as PoolTimedOut. The full coverage gate stays green because the same heavy tests are interleaved across the whole ~440-test suite rather than clustered in the narrow subset.

Local instrumentation confirmed peak server connections stay ~14/100 throughout — the bottleneck is DDL catalog locking, not the connection ceiling.

Fix

A workspace-root .config/nextest.toml test-group that caps the router::tests::jobs_endpoint_tests module at 2 concurrent threads. This bounds simultaneous migration replays for the heaviest module so connection acquisition stays well under the 60 s timeout, while keeping useful parallelism for the rest of the suite.

  • Honoured by both cargo nextest run (the subset gates) and cargo llvm-cov nextest (the coverage gate).
  • Carries no coverage semantics — the 100 % line/function gate is unaffected.
  • Touches no test pool: the deliberately-narrow error-path dead_pool (max_connections(1) / 50 ms acquire_timeout) keeps exercising its PoolTimedOut arms verbatim.

How verified (local)

  • Reproduced the contention: with the exact DB-subset selection from ci.yaml under --test-threads 8, the jobs_endpoint_tests ran ~34 s each (the slow path that tips into PoolTimedOut on the slower/noisier CI box).
  • Subset green with fix, twice: 239/239 passed both runs, no PoolTimedOut; the jobs_stream_* tests dropped from ~34 s to ~15 s each.
  • Coverage unchanged: the full gate (cargo llvm-cov nextest ... --fail-under-lines 100 --fail-under-functions 100 --test-threads 8) passes 442/442 at 100.00 % lines + 100.00 % functions.

Note: a hard PoolTimedOut failure was not reproduced on the local box (it passed at 8-wide, just slowly) — the local machine stays under the 60 s acquire_timeout that the busier CI runner crosses. The fix is validated by the green + much-faster subset runs and the unchanged coverage gate.

… subset parallelism

The "DB Subset Tests" CI job runs a narrow nextest selection
(`db::tests` + `job_store::tests` + `router::tests::jobs_*` + ...)
under `--test-threads 8`. In that job the five SSE tests
`router::tests::jobs_endpoint_tests::jobs_stream_*` intermittently
fail with `create: PoolTimedOut` after running >100 s, while the full
coverage gate (same `--test-threads 8`, on the same SHA) keeps them
green.

Root cause is migration-replay contention, not raw connection
exhaustion. Every test that calls `crate::test_db::setup_pool()`
CREATEs a fresh per-test schema and replays the full migration suite
(16 DDL files: tables, triggers, views) into the single shared
`postgres:17` container. Postgres serialises concurrent DDL on its
system catalogs, so when the DB subset packs the migration-replaying
tests together and eight run at once, each `setup_pool()` stretches
from <1 s to tens of seconds. The `jobs_stream_*` tests additionally
hold their pool across deliberate sleep/timeout windows, so under that
contention their connection acquisition exceeds the pool's 60 s
`acquire_timeout` and surfaces as `PoolTimedOut`. The full gate stays
green because the same heavy tests are interleaved across the entire
suite rather than clustered. Peak server connections stay ~14/100
throughout, confirming the bottleneck is DDL catalog locking.

Fix: add a workspace-root `.config/nextest.toml` test-group that caps
the `router::tests::jobs_endpoint_tests` module at 2 concurrent
threads. This bounds simultaneous migration replays for the heaviest
module so connection acquisition stays well under the 60 s timeout,
while keeping useful parallelism for the rest of the suite. The config
is honoured by both `cargo nextest run` (the subset gates) and
`cargo llvm-cov nextest` (the coverage gate), carries no coverage
semantics, and touches no test pool — the deliberately-narrow
error-path `dead_pool` (`max_connections(1)` / 50 ms timeout) keeps
exercising its `PoolTimedOut` arms verbatim.

Verified locally: the exact DB-subset selection now passes 239/239
twice with no `PoolTimedOut` (the `jobs_stream_*` tests drop from ~34 s
to ~15 s each), and the 100% line + function coverage gate is
unchanged.
@TaprootFreak
TaprootFreak marked this pull request as ready for review June 3, 2026 16:43
@TaprootFreak TaprootFreak added the ci:db DB Subset Tests on M3 Ultra (~15 min). Exclusive with ci:full. label Jun 3, 2026
@TaprootFreak
TaprootFreak merged commit 974cd5d into staging Jun 4, 2026
15 checks passed
TaprootFreak added a commit that referenced this pull request Jun 4, 2026
…205)

* ci: collapse test gating to 2 tiers (lint&build default, ci:full = full gate)

Replace the 3-tier test-gating model with a clean 2-tier model:

- Tier 1 `Lint & Build` (GitHub-hosted) — the default; runs on every
  non-draft PR and every push, no label required.
- Tier 2 `Tests + Coverage Gate (M3 Ultra)` — opt-in via the `ci:full`
  label; the full node + shared nextest suite under llvm-cov including
  the Postgres db_tests, the Plonky2 prover flows, and the 100% line +
  function coverage gate, in one job.

Changes:
- Remove the `DB Subset Tests` and `Prover Subset Tests` jobs, the
  `ci:db` / `ci:prover` labels, and the `!contains(... 'ci:full')`
  mutual-exclusion clauses entirely. The heavy gate is a strict
  superset of both subsets, so they only added filter-drift
  maintenance burden without extending coverage.
- Rewrite the ci.yaml header / inline comments to the 2-tier model.
- Keep `.config/nextest.toml` (jobs-endpoint max-threads=2 from #196);
  update its comment to note the cap now guards the full gate generally
  rather than the removed DB Subset job.
- Auto-promote: label the staging -> develop Promote PR with `ci:full`
  automatically (mirrors the develop -> main Release PR), so every
  promotion is validated against the full gate. Adds `issues: write`
  for `gh label create`.
- CONTRIBUTING.md: rewrite all 3-tier / subset references to 2-tier.

* ci: scrub internal runner hostname from ci.yaml comments

The Tier-2 comment block named the internal CI host (dfx01) and its
agent/core topology. This repo is public; replace with neutral
'shared self-hosted M3 Ultra runner pool' wording. No workflow logic
changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:db DB Subset Tests on M3 Ultra (~15 min). Exclusive with ci:full.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant