Skip to content

fix(ci): retry docker compose up on transient registry pull failures#24546

Merged
spalladino merged 2 commits into
merge-train/spartan-v5from
cb/fix-flake-e2e-ha-full-docker-pull
Jul 6, 2026
Merged

fix(ci): retry docker compose up on transient registry pull failures#24546
spalladino merged 2 commits into
merge-train/spartan-v5from
cb/fix-flake-e2e-ha-full-docker-pull

Conversation

@AztecBot

@AztecBot AztecBot commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

e2e_ha_full.parallel.test.ts ("should produce blocks with HA coordination and attestations") failed in CI on an unrelated PR (#24492) before the test even started: Docker Hub returned a transient 502 while docker compose up pulled postgres:16-alpine for the HA compose stack.

CI log: http://ci.aztec-labs.com/cac25b9251c99496

postgres Error unknown: failed to resolve reference "docker.io/library/postgres:16-alpine":
unexpected status from HEAD request to https://registry-1.docker.io/v2/library/postgres/manifests/16-alpine: 502 Bad Gateway

This can hit any compose-based test (run_compose_test is shared by the e2e compose/web3signer/ha suites and docs examples) whenever a required image is not in the local Docker cache and the registry blips.

Fix

Wrap the docker compose up -d --force-recreate in ci3/run_compose_test with the existing ci3/retry -p <regex> helper, retrying only failures that match transient registry/network errors (5xx, HEAD-request failures, TLS/connection timeouts, rate limiting). This reuses the same pattern-gated retry convention already used for network flakes in l1-contracts/bootstrap.sh, barretenberg/sol/bootstrap.sh, and noir-projects/aztec-nr/bootstrap.sh, so genuine failures (e.g. an image tag that doesn't exist) still fail immediately without retrying.

Retrying up -d --force-recreate is idempotent: a retry recreates any containers from a partially failed attempt.

Verified the exact error message from the CI log matches the retry pattern (retried, then succeeds), while a manifest not found failure exits immediately without retries.

No tracked issue exists for this flake.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 6, 2026
@AztecBot AztecBot changed the base branch from next to merge-train/spartan-v5 July 6, 2026 17:39
@AztecBot AztecBot force-pushed the cb/fix-flake-e2e-ha-full-docker-pull branch from ecbdb98 to 112db05 Compare July 6, 2026 20:10
@spalladino spalladino marked this pull request as ready for review July 6, 2026 20:37
@spalladino spalladino requested a review from charlielye as a code owner July 6, 2026 20:37
@spalladino spalladino enabled auto-merge (squash) July 6, 2026 20:42
@spalladino spalladino merged commit 7533c61 into merge-train/spartan-v5 Jul 6, 2026
12 checks passed
@spalladino spalladino deleted the cb/fix-flake-e2e-ha-full-docker-pull branch July 6, 2026 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants