Summary
A small family of timing-sensitive tests fails intermittently in the sync-service CI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.
Evidence
Electric.ShapeCacheTest — "await_snapshot_start/4 should wait for consumer to come up"
Electric.ShapeCacheTest — "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"
Electric.Plug.TraceContextPlugTest — "electric member is found among other tracestate members"
ShapeLogCollectorTest — FlushTracker timing out in Repatch.cleanup
Common signature
- ~5s
Task.await timeouts on tests that race consumer/snapshot startup
- Postgres connection churn in the surrounding logs (
tcp connect timeout, econnrefused, tcp recv (idle): closed) suggesting the shared PG instance is briefly saturated by concurrent async tests
- Not reproducible locally on fast machines; version-hopping across the PG matrix
Suggested directions
- Widen or parametrize the tight awaits in the affected tests (they're asserting eventual startup, not 5-second startup)
- Or address the underlying PG saturation (connection pool sizing / readiness in the test env)
- At minimum, tag the known-flaky set so authors and reviewers can distinguish them from real regressions without re-deriving this analysis
Filed while triaging CI on #4711, where these flakes twice muddied the signal.
Summary
A small family of timing-sensitive tests fails intermittently in the
sync-serviceCI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.Evidence
Electric.ShapeCacheTest— "await_snapshot_start/4 should wait for consumer to come up"Task.await(..., 5300)timeoutasync: falseintegration tests run after the async phase this test runs in — no load coupling)Electric.ShapeCacheTest— "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"Electric.Plug.TraceContextPlugTest— "electric member is found among other tracestate members"ShapeLogCollectorTest— FlushTracker timing out inRepatch.cleanupCommon signature
Task.awaittimeouts on tests that race consumer/snapshot startuptcp connect timeout,econnrefused,tcp recv (idle): closed) suggesting the shared PG instance is briefly saturated by concurrent async testsSuggested directions
Filed while triaging CI on #4711, where these flakes twice muddied the signal.