Skip to content

Stop registry blips from failing integration test jobs - #70524

Merged
potiuk merged 1 commit into
mainfrom
harden-integration-image-pulls
Jul 27, 2026
Merged

Stop registry blips from failing integration test jobs#70524
potiuk merged 1 commit into
mainfrom
harden-integration-image-pulls

Conversation

@potiuk

@potiuk potiuk commented Jul 27, 2026

Copy link
Copy Markdown
Member

An integration job dies whenever Docker Hub is briefly unreachable from the runner, before a single test executes. Example — run 30267566636, "Integration core otel":

13:24:45  otel-collector Error Get "https://registry-1.docker.io/v2/":
          net/http: request canceled (Client.Timeout exceeded while awaiting headers)
13:25:29  postgres Error Get "https://registry-1.docker.io/v2/": context deadline exceeded
13:25:39  The core integration tests failed for the second time! Giving up

docker compose run had to pull otel/opentelemetry-collector-contrib (and postgres on the retry) and the registry did not answer. Two things made that fatal:

  • the pull is done implicitly by docker compose run, so there is no retry at the level where the failure happens — recovery costs a full re-run of the suite;
  • run_integration_tests_with_retry.sh starts that re-run immediately, so both attempts land inside a minute (13:24:45 and 13:25:29). Anything longer than a one-minute blip cannot be survived.

This changes both:

  • breeze testing {core,providers}-integration-tests now pulls the third-party images the compose project needs before running, retrying each with a 15/30/45/60s backoff. The locally built CI image is excluded, and images already present locally are not touched, so a warm runner or a local run pulls nothing.
  • the retry script waits (RETRY_DELAY_SECONDS, default 60) before the second attempt.

Behaviour is unchanged when the registry is healthy. If an image genuinely cannot be pulled the run proceeds and fails with docker compose's own error, as before.

A Docker Hub timeout while pulling the integration images fails the job
before a single test runs, and the retry that follows is a full re-run of
the suite that starts within a minute of the first one - too soon to
outlast the outage that caused it. Pulling the images up front retries
the operation that actually fails, and waiting before the re-run gives a
transient outage time to clear.
@potiuk
potiuk merged commit 772a865 into main Jul 27, 2026
155 checks passed
@potiuk
potiuk deleted the harden-integration-image-pulls branch July 27, 2026 18:50
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

potiuk added a commit that referenced this pull request Jul 27, 2026
…70524) (#70557)

A Docker Hub timeout while pulling the integration images fails the job
before a single test runs, and the retry that follows is a full re-run of
the suite that starts within a minute of the first one - too soon to
outlast the outage that caused it. Pulling the images up front retries
the operation that actually fails, and waiting before the re-run gives a
transient outage time to clear.
(cherry picked from commit 772a865)

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants