Make registry-backfill workflow actually publish backfilled pages

[kaxil]

Summary

The Registry Backfill workflow has been reporting success while uploading nothing. Three related bugs combine:

1. Cached providers.json overwrite drops the backfill version. registry-backfill.yml's Download data files from S3 for build step writes the cached S3 providers.json to registry/src/_data/providers.json AFTER extraction has run. The cached copy predates the backfilled version, so provider.versions[] doesn't list it. registry/src/_data/providerVersions.js:48-50 then intersects provider.versions ∩ availableSet and drops the on-disk metadata.json for the new version. Eleventy emits no per-version page.

2. Silent aws s3 sync on missing source. With (1) preventing the page from being built, the next step's aws s3 sync exits 0 on the missing source and the job goes green having uploaded nothing.

3. A redundant Extract version metadata from git tags step that was also broken. It duplicated breeze registry backfill's internal call to extract_versions.py AND was broken for multi-version matrix entries because extract_versions.py:446 declares --version as single-valued (no action="append"). When the workflow looped --version A --version B argparse only kept the last one. The trailing || true masked the failure.

Fix

• New dev/registry/patch_providers_json.py runs after the S3 download and appends the backfilled version(s) into provider.versions[]. Defensively keeps provider.version (latest) in the array so patching a previously-empty list doesn't leave the latest field excluded once the array becomes authoritative for providerVersions.js.

• Pre-sync guards assert (a) the per-version index.html was emitted, (b) the API directory exists, (c) parameters.json is non-empty, before each aws s3 sync. Silent no-ops become hard failures with ::error:: annotations.

• Delete the redundant standalone Extract version metadata step. breeze registry backfill already runs _run_extract_versions internally and propagates exit codes correctly.

Why patch providers.json instead of changing the intersection in providerVersions.js

The intersection is shared by registry-build.yml (full builds) and registry-backfill.yml. Changing it affects both flows. Patching is scoped to backfill, mirrors the mental model that providers.json IS the source of truth for "which versions to expose", and matches what breeze registry publish-versions is supposed to do at the end of the run anyway.

Gotchas

• packaging is now declared as a runtime dep of dev/registry/. It was already available transitively via pydantic, but the patch script uses packaging.version for newest-first sorting and shouldn't rely on a transitive dep.
• The pre-sync guard's content checks deliberately don't assert classes is non-empty: providers like common-compat legitimately have zero modules ("Configuration-only provider"), and the page renders a placeholder for them.

Known follow-up (separate PR)

The publish-versions job runs in a fresh checkout where providers.json is gitignored, so publish_registry_versions.py:119-127 raises FileNotFoundError after the per-version sync. Tracked separately and shipping right after this PR; not in scope here.

Verification

This PR alone makes the per-version pages land on S3 correctly. Once the follow-up publish-versions fix lands, the smoke test (amazon/9.24.0 google/21.0.0 common-compat/1.14.2) should go fully green:

gh workflow run registry-backfill.yml -R apache/airflow \
  -f destination=staging \
  -f provider-versions="amazon/9.24.0 google/21.0.0 common-compat/1.14.2"

Tests in dev/registry/tests/test_patch_providers_json.py cover 11 cases including the empty-list / latest-field edges.

Linked prior fixes that built the chain to here: #65972, #65975, #65987, #65984, #65989.

[github-actions]

Backport failed to create: v3-2-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
❌	v3-2-test

You can attempt to backport this manually by running:

cherry_picker 7e49884 v3-2-test

This should apply the commit to the v3-2-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.