test(jobs): fix DB-subset pg pool exhaustion in jobs_stream SSE tests - #196
Merged
Conversation
… subset parallelism The "DB Subset Tests" CI job runs a narrow nextest selection (`db::tests` + `job_store::tests` + `router::tests::jobs_*` + ...) under `--test-threads 8`. In that job the five SSE tests `router::tests::jobs_endpoint_tests::jobs_stream_*` intermittently fail with `create: PoolTimedOut` after running >100 s, while the full coverage gate (same `--test-threads 8`, on the same SHA) keeps them green. Root cause is migration-replay contention, not raw connection exhaustion. Every test that calls `crate::test_db::setup_pool()` CREATEs a fresh per-test schema and replays the full migration suite (16 DDL files: tables, triggers, views) into the single shared `postgres:17` container. Postgres serialises concurrent DDL on its system catalogs, so when the DB subset packs the migration-replaying tests together and eight run at once, each `setup_pool()` stretches from <1 s to tens of seconds. The `jobs_stream_*` tests additionally hold their pool across deliberate sleep/timeout windows, so under that contention their connection acquisition exceeds the pool's 60 s `acquire_timeout` and surfaces as `PoolTimedOut`. The full gate stays green because the same heavy tests are interleaved across the entire suite rather than clustered. Peak server connections stay ~14/100 throughout, confirming the bottleneck is DDL catalog locking. Fix: add a workspace-root `.config/nextest.toml` test-group that caps the `router::tests::jobs_endpoint_tests` module at 2 concurrent threads. This bounds simultaneous migration replays for the heaviest module so connection acquisition stays well under the 60 s timeout, while keeping useful parallelism for the rest of the suite. The config is honoured by both `cargo nextest run` (the subset gates) and `cargo llvm-cov nextest` (the coverage gate), carries no coverage semantics, and touches no test pool — the deliberately-narrow error-path `dead_pool` (`max_connections(1)` / 50 ms timeout) keeps exercising its `PoolTimedOut` arms verbatim. Verified locally: the exact DB-subset selection now passes 239/239 twice with no `PoolTimedOut` (the `jobs_stream_*` tests drop from ~34 s to ~15 s each), and the 100% line + function coverage gate is unchanged.
TaprootFreak
marked this pull request as ready for review
June 3, 2026 16:43
This was referenced Jun 4, 2026
Closed
TaprootFreak
added a commit
that referenced
this pull request
Jun 4, 2026
…205) * ci: collapse test gating to 2 tiers (lint&build default, ci:full = full gate) Replace the 3-tier test-gating model with a clean 2-tier model: - Tier 1 `Lint & Build` (GitHub-hosted) — the default; runs on every non-draft PR and every push, no label required. - Tier 2 `Tests + Coverage Gate (M3 Ultra)` — opt-in via the `ci:full` label; the full node + shared nextest suite under llvm-cov including the Postgres db_tests, the Plonky2 prover flows, and the 100% line + function coverage gate, in one job. Changes: - Remove the `DB Subset Tests` and `Prover Subset Tests` jobs, the `ci:db` / `ci:prover` labels, and the `!contains(... 'ci:full')` mutual-exclusion clauses entirely. The heavy gate is a strict superset of both subsets, so they only added filter-drift maintenance burden without extending coverage. - Rewrite the ci.yaml header / inline comments to the 2-tier model. - Keep `.config/nextest.toml` (jobs-endpoint max-threads=2 from #196); update its comment to note the cap now guards the full gate generally rather than the removed DB Subset job. - Auto-promote: label the staging -> develop Promote PR with `ci:full` automatically (mirrors the develop -> main Release PR), so every promotion is validated against the full gate. Adds `issues: write` for `gh label create`. - CONTRIBUTING.md: rewrite all 3-tier / subset references to 2-tier. * ci: scrub internal runner hostname from ci.yaml comments The Tier-2 comment block named the internal CI host (dfx01) and its agent/core topology. This repo is public; replace with neutral 'shared self-hosted M3 Ultra runner pool' wording. No workflow logic changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The DB Subset Tests (M3 Ultra) CI job (
ci:dblabel, narrow nextest selection,--test-threads 8) intermittently fails the five SSE tests withcreate: PoolTimedOut, each running >100 s before timing out:router::tests::jobs_endpoint_tests::jobs_stream_404_for_unknown_idrouter::tests::jobs_endpoint_tests::jobs_stream_closes_immediately_for_terminal_jobrouter::tests::jobs_endpoint_tests::jobs_stream_emits_initial_phase_for_non_terminal_jobrouter::tests::jobs_endpoint_tests::jobs_stream_failed_terminal_closes_with_complete_and_errorrouter::tests::jobs_endpoint_tests::jobs_stream_forwards_dispatcher_phase_transitionThe full coverage gate (same
--test-threads 8, same SHA) keeps these tests green, so the tests themselves are sound.Root cause
Migration-replay contention, not raw connection exhaustion. Every test that calls
crate::test_db::setup_pool()creates a fresh per-test schema and replays the full migration suite (16 DDL files: tables, triggers, views) into the single sharedpostgres:17container. Postgres serialises concurrent DDL on its system catalogs, so when the DB subset packs the migration-replaying tests together and eight run at once, eachsetup_pool()stretches from <1 s to tens of seconds.The
jobs_stream_*tests additionally hold their pool across deliberatesleep/timeoutwindows, so under that contention their connection acquisition exceeds the pool's 60 sacquire_timeoutand surfaces asPoolTimedOut. The full coverage gate stays green because the same heavy tests are interleaved across the whole ~440-test suite rather than clustered in the narrow subset.Local instrumentation confirmed peak server connections stay ~14/100 throughout — the bottleneck is DDL catalog locking, not the connection ceiling.
Fix
A workspace-root
.config/nextest.tomltest-group that caps therouter::tests::jobs_endpoint_testsmodule at 2 concurrent threads. This bounds simultaneous migration replays for the heaviest module so connection acquisition stays well under the 60 s timeout, while keeping useful parallelism for the rest of the suite.cargo nextest run(the subset gates) andcargo llvm-cov nextest(the coverage gate).dead_pool(max_connections(1)/ 50 msacquire_timeout) keeps exercising itsPoolTimedOutarms verbatim.How verified (local)
ci.yamlunder--test-threads 8, thejobs_endpoint_testsran ~34 s each (the slow path that tips intoPoolTimedOuton the slower/noisier CI box).PoolTimedOut; thejobs_stream_*tests dropped from ~34 s to ~15 s each.cargo llvm-cov nextest ... --fail-under-lines 100 --fail-under-functions 100 --test-threads 8) passes 442/442 at 100.00 % lines + 100.00 % functions.