Skip to content

test(db): wait for real Postgres readiness, not just the mapped port - #64

Merged
jasoet merged 1 commit into
nextfrom
fix/postgres-testcontainer-readiness
Aug 17, 2026
Merged

test(db): wait for real Postgres readiness, not just the mapped port#64
jasoet merged 1 commit into
nextfrom
fix/postgres-testcontainer-readiness

Conversation

@jasoet

@jasoet jasoet commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Symptom

The Release pipeline failed twice in a row on the db integration suite — a different test each time, same signature:

--- FAIL: TestOTelCallbacksTableAndRowsAffected (10.77s)      # run 1
--- FAIL: TestPostgresMigrationsInvalidPath   (10.52s)        # run 2 (re-run)

failed to ping database at localhost:33451/testdb:
  [::1]:33451  failed to receive message: unexpected EOF
  127.0.0.1:33451  read: connection reset by peer

Both ~10s in — well inside the 60s deadline, so not a timeout. Both pass locally. The triggering commit was docs-only.

Root cause

The wait strategy, not the tests.

The postgres image runs initdb against a temporary server, stops it, then starts the real one. All eight Postgres containers in this package waited only on the mapped port:

wait.ForListeningPort("5432/tcp").WithStartupTimeout(60*time.Second)

So the container could be declared ready while the server was on its way down, and the connection that followed got reset.

This is documented in testcontainers-go itself — postgres.BasicWaitStrategies():

First, we wait for the container to log readiness twice. This is because it will restart itself after the first startup.
[…] For non-linux OSes like Mac […] Docker will have to start a separate proxy. Without this, the tests will be flaky on those OSes!

The db tests used only the second half of that pair. The runner is a Mac.

That also explains the intermittency and why a docs commit "caused" it: the window is invisible on an idle machine and opens under CI load. MySQL and MSSQL in the same file were never affected — they already wait on a real query via wait.ForSQL, with a comment saying so.

Fix

Add the occurrence-2 log check alongside the port check, behind one shared helper (postgresReady()) so the strategy can't drift back one call site at a time, with a deadline sized for a loaded runner. 8 call sites across 3 files now share it.

Confirmed no other package is affected: every postgres.Run in the repo is in db/, and the remaining ForListeningPort uses (Temporal, sshd) are services that don't restart themselves during init.

Verification

condition result
db integration, idle ok — 80.3s
db integration, 10-way CPU contention ok — 102.7s
task ci:check pass, 0 lint issues
go vet -tags='example integration argo' ./... clean
golangci-lint --build-tags=integration ./db/... 0 issues

The loaded run is the condition CI was failing under.

The Release pipeline failed twice in a row on the db integration suite, with a
different test each time (TestOTelCallbacksTableAndRowsAffected, then
TestPostgresMigrationsInvalidPath) and the same signature: "failed to ping
database ... connection reset by peer / unexpected EOF", roughly 10s in — well
inside the 60s deadline, so not a timeout.

Root cause is the wait strategy. The postgres image runs initdb against a
temporary server, stops it, then starts the real one. All eight Postgres
containers in this package waited only on the mapped port, so the container
could be declared ready while the server was on its way down; the connection
that followed was reset. testcontainers-go documents this in
postgres.BasicWaitStrategies, which waits for the readiness log line twice
"because it will restart itself after the first startup" and warns the tests
"will be flaky" on macOS otherwise — this runner is a Mac.

That also explains the intermittency: the window is invisible on an idle
machine and opens under CI load, which is why it reproduced on the runner and
never locally. MySQL and MSSQL in the same package were unaffected because they
already wait on a real query via wait.ForSQL.

Add the occurrence-2 log check alongside the port check behind a single shared
helper, so the strategy cannot drift back one call site at a time, and give it
a deadline with headroom for a loaded runner.

Verified: db integration suite passes idle (80s) and under 10-way CPU
contention (103s), where the loaded case is what CI was failing.
@jasoet
jasoet merged commit b9b19e4 into next Aug 17, 2026
1 check passed
@jasoet
jasoet deleted the fix/postgres-testcontainer-readiness branch August 17, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant