Build the docs CI image once and let the registry share it - #70618
Conversation
A docs publish paid for two CI image builds of the same sources, one for the docs and one for the provider registry, and only started the second once the docs were already live. Building it once up front and handing the same image to both lets them run side by side, which is most of the wait a release manager sits through after the docs have landed. Keeping the mount cache means consecutive publishes stop re-downloading the whole dependency set, and pinning both to one Python removes the second image that only differed by interpreter.
The build the docs job used to run inline fell back to a plain buildx build whenever an old ref's pyproject.toml had drifted far enough from main that main's registry cache could not satisfy it. Moving the build into its own job dropped that safety net, which would turn a slow-but-correct build of an old ref into a hard failure.
Expressing the no-cache retry as its own job put a permanently-skipped job in every docs publish and forced both consumers to reason about which of two builds produced the image. The retry belongs where the build is, and callers that do not want it are unaffected because it is off by default.
boring-cyborg
Bot
added
area:dev-tools
area:registry
backport-to-v3-3-test
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker c6449eb v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
| needs: [publish-docs-to-s3, build-info] | ||
| # Runs alongside the docs build rather than after it: the registry reads nothing the docs | ||
| # publish produces, so waiting only added its whole duration to the release manager's wait. | ||
| needs: [build-info, build-ci-image] |
There was a problem hiding this comment.
On the gate question: the sync steps look like the cheap place for it. Measured on run 30335779728, everything the registry job does after the image is 4m30s, and only 1m21s of that is Sync registry to S3 plus Publish version metadata (which is host-side, no CI image needed). The site already goes up as the registry-site artifact, so those two steps could move into a job that needs publish-docs-to-s3, and the docs branch (~27m of Sphinx plus 4m publish) is long enough that the registry would be idle waiting for it anyway, so the wall clock does not change.
That also retires the #70587 caveat that the registry becomes the critical path on 5 of 11 runs. That was only true while it built its own image.
| # cached against main's registry cache, which the regular Test workflow keeps warm. | ||
| checkout-ref: ${{ inputs.ref }} | ||
| push-image: "false" | ||
| upload-image-artifact: "true" |
There was a problem hiding this comment.
Splitting the build into its own job means the docs path now pays an image round-trip it did not before. On run 30335779728, Export CI docker image plus Stash CI docker image was 4m26s, the consumer side (Prepare breeze & CI image in the registry job, minus the breeze install and disk cleanup it shares) about 5m, and the new job repeats the checkout / free-disk / breeze preamble for another ~1m50s. For a provider wave that is a clear net win against a 13-32m second build, but build-ci-image has no if:, so an apache-airflow-only publish skips update-registry and pays the whole ~11m for a stash that only build-docs reads. Per #70587 that cohort medians 21m, so it is a large relative hit. Worth gating the job on registry-providers != '' || registry-full-build == 'true' and leaving the inline build in build-docs for that case?
| # skipping that job used to enforce for us on a dispatch. | ||
| if: > | ||
| !cancelled() && needs.build-ci-image.result != 'failure' && ( | ||
| github.event_name == 'workflow_call' || |
There was a problem hiding this comment.
github.event_name in a called workflow is the caller's event (workflow_dispatch here), never workflow_call, so this disjunct never fires and the job ends up gated on github.event.sender.login on both paths. Same net behaviour as before, since the old needs: [build-ci-image] inherited the identical check, but it does mean this allowlist is load-bearing for the shared-image path and it has drifted from the one on build-info in publish-docs-to-s3.yml, which also lists vatsrahul1001. A wave dispatched by them would publish docs and silently skip the registry.
3 participants
Addresses Question 1 of #70587, and takes Question 2's parallel option.
A provider docs publish built two CI images from the same sources — the docs job inline at
publish-docs-to-s3.yml:301-318(python3.10) and the registry's ownbuild-ci-imagejob (python3.12) — and only started the registry once the docs were already live on S3. Per #70587 the registry sub-workflow medians 23m, of which ~13m is that second image build, and it is 100% additive because it is strictly serial after the publish.What changes
build-ci-imagejob inpublish-docs-to-s3.ymlbuilds frominputs.refand stashes it;build-docsrestores it instead of building inline, andupdate-registrypassesci-image-already-built: truesoregistry-build.ymlskips its own build. A standalone registry dispatch has no such image and still builds one.default-python-versionmoves 3.10 → 3.12, which is what the registry has always extracted on. This removes the second image that differed only by interpreter. Notedev/registry/extract_metadata.py:47andextract_versions.py:49fall back totomlibelow 3.11 — that path is# pragma: no cover, so 3.12 is the better-exercised side.update-registrymoves fromneeds: [publish-docs-to-s3, build-info]toneeds: [build-info, build-ci-image]. The registry reads nothing the docs publish produces.ci-image-build.ymlgains aretry-without-cacheinput (defaultfalse, so every other caller is unaffected); when the cached build fails it retries once withDOCKER_CACHE=disabled. This keeps the behaviour of the inlinebreeze ci-image build || docker buildx build --pullit replaces, for old refs whosepyproject.tomlhas drifted from main.upload-mount-cache-artifact: "true", so the next publish on the branch restores it rather than re-downloading the dependency set.docker-cache: registryagainstbranch: main, the cache the regular Test workflow keeps warm.Resulting graph:
Consequences worth reviewing
needs: [publish-docs-to-s3]previously gated the registry, so a failed docs build skipped it. Running in parallel removes that: the registry can publish/stable/links to docs that never landed. Get provider registry build off the docs publish critical path #70587 measures the exposure window at median ~3 min, up to ~21 min, negative on 5 of 11 sampled runs. If that is not acceptable, the gate should move onto the S3 sync step rather than back onto the whole job.needs:a job the allowlist skipped; now that job is also skipped on the shared-image path, so the check is stated explicitly onbuild-and-publish-registry.Not done here, and worth a separate look: restoring the canary image instead of building at all. It does not work through the stash — the Tests (AMD) run stashes only the default Python (
ci-image-save-v3-linux_amd64-3.10-…), the stash name is branch-suffixed with no input to select a branch (current + base/default only), and artifacts expire in ~2 days. The cross-run-viable route would be pullingghcr.io/apache/airflow/<branch>/ci/python<ver>:latest, whichpush-image-cache.ymlpublishes for every canary Python.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines