Stop registry blips from failing integration test jobs - #70524
Merged
Conversation
A Docker Hub timeout while pulling the integration images fails the job before a single test runs, and the retry that follows is a full re-run of the suite that starts within a minute of the first one - too soon to outlast the outage that caused it. Pulling the images up front retries the operation that actually fails, and waiting before the re-run gives a transient outage time to clear.
potiuk
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jason810496,
jedcunningham,
jscheffl and
vatsrahul1001
as code owners
July 27, 2026 14:19
shahar1
approved these changes
Jul 27, 2026
Miretpl
approved these changes
Jul 27, 2026
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
potiuk
added a commit
that referenced
this pull request
Jul 27, 2026
…70524) (#70557) A Docker Hub timeout while pulling the integration images fails the job before a single test runs, and the retry that follows is a full re-run of the suite that starts within a minute of the first one - too soon to outlast the outage that caused it. Pulling the images up front retries the operation that actually fails, and waiting before the re-run gives a transient outage time to clear. (cherry picked from commit 772a865) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An integration job dies whenever Docker Hub is briefly unreachable from the runner, before a single test executes. Example — run 30267566636, "Integration core otel":
docker compose runhad to pullotel/opentelemetry-collector-contrib(andpostgreson the retry) and the registry did not answer. Two things made that fatal:docker compose run, so there is no retry at the level where the failure happens — recovery costs a full re-run of the suite;run_integration_tests_with_retry.shstarts that re-run immediately, so both attempts land inside a minute (13:24:45 and 13:25:29). Anything longer than a one-minute blip cannot be survived.This changes both:
breeze testing {core,providers}-integration-testsnow pulls the third-party images the compose project needs before running, retrying each with a 15/30/45/60s backoff. The locally built CI image is excluded, and images already present locally are not touched, so a warm runner or a local run pulls nothing.RETRY_DELAY_SECONDS, default 60) before the second attempt.Behaviour is unchanged when the registry is healthy. If an image genuinely cannot be pulled the run proceeds and fails with docker compose's own error, as before.