Skip to content

[v3-2-test] Split ci-amd-arm.yml into ARM canary and AMD per-PR workflows (#66348)#66664

Merged
potiuk merged 1 commit into
apache:v3-2-testfrom
potiuk:backport-8f93637-v3-2-test
May 10, 2026
Merged

[v3-2-test] Split ci-amd-arm.yml into ARM canary and AMD per-PR workflows (#66348)#66664
potiuk merged 1 commit into
apache:v3-2-testfrom
potiuk:backport-8f93637-v3-2-test

Conversation

@potiuk
Copy link
Copy Markdown
Member

@potiuk potiuk commented May 10, 2026

Summary

Backport of #66348 (merged 2026-05-07, commit `8f93637`) to `v3-2-test`.

Splits the combined `ci-amd-arm.yml` workflow into two thin wrappers (`ci-arm.yml` for the canary cron + `ci-amd.yml` for per-PR pushes) calling a reusable `_ci-pipeline.yml`. Adds the `check-ci-workflows-in-sync` prek hook to keep the two wrappers in lockstep. See the source PR for full motivation.

Conflicts resolved during cherry-pick

Three conflicts plus one post-resolve drift fix:

  1. `README.md` — both branches updated the badge table; took the cherry-pick's side because v3-2-test's HEAD still pointed at `v3-1-test`-branch badges (stale from when the branch was cut).
  2. `generated/PYPI_README.md` — same pattern as `README.md`, same resolution.
  3. `dev/update_github_branch_config.py` — modify/delete: file was deliberately removed on `v3-2-test`, kept it deleted (the backport doesn't need it).
  4. `ci-arm.yml` aligned to v3-2-test's `ci-amd.yml` — the new `check-ci-workflows-in-sync` hook flagged two divergences after the rename:
    • Removed `run-remote-logging-opensearch-e2e-tests` outputs (2 places) — `v3-2-test` already removed those tests.
    • Bumped `slackapi/slack-github-action` from `v3.0.1` → `v3.0.3` (3 places) — `v3-2-test` already had the newer pin.

Test plan

  • Cherry-pick clean; pre-commit (incl. the new `check-ci-workflows-in-sync` hook) passes locally
  • mypy-dev hook fails locally only — pre-existing (`dev/airflow_mypy/` plugin reference exists in `dev/pyproject.toml` on both `main` and `v3-2-test` but the directory isn't checked in; CI installs it via `uv sync`)
  • Wait for CI on `v3-2-test`

🤖 Generated with Claude Code (Opus 4.7, 1M context)

@potiuk
[Experimental] Split ci-amd-arm.yml into ARM canary and AMD per-PR wo…
8925eb2 
…rkflows (apache#66348)

* Split ci-amd-arm.yml into ci-arm.yml (canary) and ci-amd.yml (per-PR) wrappers

Replaces the combined ci-amd-arm.yml workflow with two thin wrappers carrying
the full pipeline inline:

- ci-arm.yml: schedule (canary cron) + workflow_dispatch only;
  runs on linux/arm64.
- ci-amd.yml: pull_request + push (to v3-N-test etc) + workflow_dispatch;
  runs on linux/amd64. No schedule for now; ARM keeps the canary slot.

Each wrapper carries the full pipeline jobs identically except for platform /
runner-type / triggers, so per-job UI grouping is preserved on the GitHub
Actions page. The runtime cross-mapping in selective_checks.runner_type is
left in place because is_disabled_integration still consumes it; the API
lookup against the now-removed ci-amd-arm.yml falls back to AMD harmlessly.

References to the old file name are updated in:
README.md, dev/update_github_branch_config.py, ci-notification.yml,
e2e-flaky-tests-report.yml, scripts/ci/analyze_e2e_flaky_tests.py,
dev/README_AIRFLOW3_DEV.md, dev/README_RELEASE_AIRFLOW.md,
generated/PYPI_README.md, update-constraints-on-push*.yml,
release_dockerhub_image.yml.

* Add prek hook that asserts ci-arm.yml and ci-amd.yml stay in sync

The two CI workflow files are physical copies of each other (GH Actions
has no cross-file YAML include) — they should differ only in:

- header intro comment
- workflow name (ARM vs AMD)
- triggers (schedule vs pull_request+push)
- concurrency group prefix
- build-info platform / runner-type outputs
- print-platform job name + echo

Drift outside this set is a bug. The new
`scripts/ci/prek/check_ci_workflows_in_sync.py` normalizes both files
against the documented divergences (regex line rules + arch-only block
removals) and asserts the rest matches byte-for-byte. Wired up as a
local prek hook gated on either workflow file changing.

Local UX: when run interactively, the script also diffs each file
against `upstream/main` (falls back to `origin/main`, then `main`) and
attributes the drift — "only ci-arm.yml changed since baseline; mirror
to ci-amd.yml" with the exact diff to copy, OR "add to LINE_RULES /
ARM_ONLY_BLOCK in this script if intentional". Falls back to the
non-attribution view in CI / non-TTY runs. Set `FORCE_INTERACTIVE=1`
to opt in from a non-TTY shell.

(cherry picked from commit 8f93637)
@potiuk potiuk merged commit a927451 into apache:v3-2-test May 10, 2026
8 checks passed
@potiuk potiuk deleted the backport-8f93637-v3-2-test branch May 10, 2026 18:19
@potiuk potiuk mentioned this pull request May 10, 2026
Merged
2 tasks
@vatsrahul1001 vatsrahul1001 added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label May 18, 2026
@vatsrahul1001 vatsrahul1001 added this to the Airflow 3.2.2 milestone May 18, 2026
vatsrahul1001 pushed a commit that referenced this pull request May 20, 2026
@potiuk @vatsrahul1001
[Experimental] Split ci-amd-arm.yml into ARM canary and AMD per-PR wo…
291b968 
…rkflows (#66348) (#66664)

* Split ci-amd-arm.yml into ci-arm.yml (canary) and ci-amd.yml (per-PR) wrappers

Replaces the combined ci-amd-arm.yml workflow with two thin wrappers carrying
the full pipeline inline:

- ci-arm.yml: schedule (canary cron) + workflow_dispatch only;
  runs on linux/arm64.
- ci-amd.yml: pull_request + push (to v3-N-test etc) + workflow_dispatch;
  runs on linux/amd64. No schedule for now; ARM keeps the canary slot.

Each wrapper carries the full pipeline jobs identically except for platform /
runner-type / triggers, so per-job UI grouping is preserved on the GitHub
Actions page. The runtime cross-mapping in selective_checks.runner_type is
left in place because is_disabled_integration still consumes it; the API
lookup against the now-removed ci-amd-arm.yml falls back to AMD harmlessly.

References to the old file name are updated in:
README.md, dev/update_github_branch_config.py, ci-notification.yml,
e2e-flaky-tests-report.yml, scripts/ci/analyze_e2e_flaky_tests.py,
dev/README_AIRFLOW3_DEV.md, dev/README_RELEASE_AIRFLOW.md,
generated/PYPI_README.md, update-constraints-on-push*.yml,
release_dockerhub_image.yml.

* Add prek hook that asserts ci-arm.yml and ci-amd.yml stay in sync

The two CI workflow files are physical copies of each other (GH Actions
has no cross-file YAML include) — they should differ only in:

- header intro comment
- workflow name (ARM vs AMD)
- triggers (schedule vs pull_request+push)
- concurrency group prefix
- build-info platform / runner-type outputs
- print-platform job name + echo

Drift outside this set is a bug. The new
`scripts/ci/prek/check_ci_workflows_in_sync.py` normalizes both files
against the documented divergences (regex line rules + arch-only block
removals) and asserts the rest matches byte-for-byte. Wired up as a
local prek hook gated on either workflow file changing.

Local UX: when run interactively, the script also diffs each file
against `upstream/main` (falls back to `origin/main`, then `main`) and
attributes the drift — "only ci-arm.yml changed since baseline; mirror
to ci-amd.yml" with the exact diff to copy, OR "add to LINE_RULES /
ARM_ONLY_BLOCK in this script if intentional". Falls back to the
non-attribution view in CI / non-TTY runs. Set `FORCE_INTERACTIVE=1`
to opt in from a non-TTY shell.

(cherry picked from commit 8f93637)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@vincbeck vincbeck Awaiting requested review from vincbeck vincbeck is a code owner

@shahar1 shahar1 Awaiting requested review from shahar1 shahar1 is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@bugraoz93 bugraoz93 Awaiting requested review from bugraoz93 bugraoz93 is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@gopidesupavan gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

@amoghrajesh amoghrajesh Awaiting requested review from amoghrajesh amoghrajesh is a code owner

@jason810496 jason810496 Awaiting requested review from jason810496 jason810496 is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@choo121600 choo121600 Awaiting requested review from choo121600 choo121600 is a code owner

@vatsrahul1001 vatsrahul1001 Awaiting requested review from vatsrahul1001 vatsrahul1001 is a code owner

@kaxil kaxil Awaiting requested review from kaxil kaxil is a code owner

Assignees

No one assigned

Labels

area:dev-tools changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..)

Projects

None yet

Milestone

Airflow 3.2.2

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk @vatsrahul1001