Summary
The cargo test --workspace job intermittently fails on the Postgres path while passing on a re-run of the same commit — main has shown both a passing and a failing run on one SHA.
Evidence
.github/workflows/ci.yml runs the whole suite against a single shared Postgres database (SANDBOXWICH_TEST_POSTGRES_URL). The TestServer harness (crates/sandboxwich-api/tests/http_contract.rs) spawns a fresh API process per test, each running migrations and mutating shared tables against that one database. Observed failures during the #85–#91 merge train:
server did not become healthy panics (helper ~line 3278) in several auth/tenant tests — consistent with concurrent migrations/contention on the shared DB.assertion failed: cleanup.expired... in the Postgres lifecycle test — timing against the background expiry sweeper.
These are load/timing dependent, not deterministic logic bugs (the same code passes on re-run).
Why it matters
Flaky required-path CI erodes trust and blocks clean merges; it already forced re-runs during this batch.
Suggested fix
Isolate Postgres state per test: give each TestServer its own database or schema (e.g. CREATE SCHEMA test_<uuid> + search_path, or a fresh database per test), and/or serialize the Postgres-backed tests. Poll for readiness/expiry with generous bounds on slow CI runners.
Summary
The
cargo test --workspacejob intermittently fails on the Postgres path while passing on a re-run of the same commit — main has shown both a passing and a failing run on one SHA.Evidence
.github/workflows/ci.ymlruns the whole suite against a single shared Postgres database (SANDBOXWICH_TEST_POSTGRES_URL). TheTestServerharness (crates/sandboxwich-api/tests/http_contract.rs) spawns a fresh API process per test, each running migrations and mutating shared tables against that one database. Observed failures during the #85–#91 merge train:server did not become healthypanics (helper ~line 3278) in several auth/tenant tests — consistent with concurrent migrations/contention on the shared DB.assertion failed: cleanup.expired...in the Postgres lifecycle test — timing against the background expiry sweeper.These are load/timing dependent, not deterministic logic bugs (the same code passes on re-run).
Why it matters
Flaky required-path CI erodes trust and blocks clean merges; it already forced re-runs during this batch.
Suggested fix
Isolate Postgres state per test: give each
TestServerits own database or schema (e.g.CREATE SCHEMA test_<uuid>+search_path, or a fresh database per test), and/or serialize the Postgres-backed tests. Poll for readiness/expiry with generous bounds on slow CI runners.