Skip to content

Flaky timing-sensitive sync-service tests: ShapeCacheTest await/startup timeouts under CI load #4712

Description

@robacourt

Summary

A small family of timing-sensitive tests fails intermittently in the sync-service CI workflow — hopping between Postgres matrix versions, passing on rerun with no code change, and burying real signal for PR authors. Two recent PRs (#4711 and its review) each spent effort separating these from genuine failures.

Evidence

Electric.ShapeCacheTest — "await_snapshot_start/4 should wait for consumer to come up"

Electric.ShapeCacheTest — "get_or_create_shape_handle/2 against real db crashes when initial snapshot query fails to return data quickly enough"

Electric.Plug.TraceContextPlugTest — "electric member is found among other tracestate members"

ShapeLogCollectorTest — FlushTracker timing out in Repatch.cleanup

Common signature

  • ~5s Task.await timeouts on tests that race consumer/snapshot startup
  • Postgres connection churn in the surrounding logs (tcp connect timeout, econnrefused, tcp recv (idle): closed) suggesting the shared PG instance is briefly saturated by concurrent async tests
  • Not reproducible locally on fast machines; version-hopping across the PG matrix

Suggested directions

  • Widen or parametrize the tight awaits in the affected tests (they're asserting eventual startup, not 5-second startup)
  • Or address the underlying PG saturation (connection pool sizing / readiness in the test env)
  • At minimum, tag the known-flaky set so authors and reviewers can distinguish them from real regressions without re-deriving this analysis

Filed while triaging CI on #4711, where these flakes twice muddied the signal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions