From 8ed2cf07fcefb7e2076ca835405c0eda44ac7675 Mon Sep 17 00:00:00 2001 From: danielmeppiel Date: Thu, 23 Apr 2026 15:41:51 +0200 Subject: [PATCH 1/2] ci(build-release): gate smoke to tag/schedule/dispatch only Push-time smoke in build-release.yml's build-and-test job (Linux x86_64, Linux arm64, Windows) duplicated the merge-time smoke gate already enforced by ci-integration.yml on the same SHA content, while burning ~15 redundant codex-binary downloads per active day and amplifying network-flake exposure. Smoke now runs only at promotion boundaries: - tags (pre-ship release gate; only validation tag-cut releases receive) - schedule (nightly drift catch for upstream openai/codex URL changes) - workflow_dispatch (manual safety net) Push-to-main retains unit tests on all build-and-test platforms for platform-regression signal; smoke coverage on Linux at merge_group time (ci-integration.yml) and on Linux x86_64 nightly (ci-runtime.yml) is unchanged. Multi-platform smoke (arm64 + Windows) shifts from per-push to per-tag, narrowing the time-to-detection window for platform-specific regressions in scripts/runtime/setup-codex.sh by hours-to-days but trading that for a meaningful reduction in network noise. The gating expression matches the existing canonical pattern used by the macOS Intel/ARM jobs and integration-tests/release-validation jobs in this same workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/instructions/cicd.instructions.md | 2 +- .github/workflows/build-release.yml | 20 ++++++++++++++++++-- CHANGELOG.md | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/.github/instructions/cicd.instructions.md b/.github/instructions/cicd.instructions.md index 7c5980f1f..d98c27884 100644 --- a/.github/instructions/cicd.instructions.md +++ b/.github/instructions/cicd.instructions.md @@ -41,7 +41,7 @@ integration suite runs only at merge time via GitHub Merge Queue - `.github/CODEOWNERS` requires Lead Maintainer review for any change to `.github/workflows/**`. 4. **`build-release.yml`** - `push` to main, tags, schedule, `workflow_dispatch` - - **Linux + Windows** run combined `build-and-test` (unit tests + binary build in one job). + - **Linux + Windows** run combined `build-and-test` (unit tests + binary build in one job). Unit tests run on every push for platform-regression signal; **smoke tests are gated to tag/schedule/dispatch only** (promotion boundaries) to avoid duplicating `ci-integration.yml`'s merge-time smoke and to cut redundant codex-binary downloads. - **macOS Intel** uses `build-and-validate-macos-intel` (root node, runs own unit tests - no dependency on `build-and-test`). Builds the binary on every push for early regression feedback; integration + release-validation phases conditional on tag/schedule/dispatch. - **macOS ARM** uses `build-and-validate-macos-arm` (root node, tag/schedule/dispatch only - ARM runners are extremely scarce with 2-4h+ queue waits). Only requested when the binary is actually needed for a release. - Secrets always available. Full 5-platform binary output (linux x86_64/arm64, darwin x86_64/arm64, windows x86_64). diff --git a/.github/workflows/build-release.yml b/.github/workflows/build-release.yml index c5357540a..e41dd6600 100644 --- a/.github/workflows/build-release.yml +++ b/.github/workflows/build-release.yml @@ -69,11 +69,27 @@ jobs: - name: Install dependencies run: uv sync --extra dev --extra build - - name: Run tests + # Unit tests run on every push for fast platform-regression signal. + # Smoke is intentionally NOT included here: it duplicates ci-integration.yml's + # merge-time smoke gate and burns a real codex binary download per platform + # per push (~15 redundant runs/day). Smoke is gated to promotion boundaries + # below (tag/schedule/dispatch) where it actually serves as a pre-ship gate. + - name: Run unit tests + env: + GITHUB_TOKEN: ${{ secrets.GH_MODELS_PAT }} + GITHUB_APM_PAT: ${{ secrets.GH_CLI_PAT }} + run: uv run pytest tests/unit tests/test_console.py -n auto --dist worksteal + + # Smoke runs only at promotion boundaries: + # - tags (pre-ship release gate; only place tag-cut releases get smoke validation) + # - schedule (nightly regression catch for upstream codex URL drift) + # - workflow_dispatch (manual safety net) + - name: Run smoke tests + if: github.ref_type == 'tag' || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' env: GITHUB_TOKEN: ${{ secrets.GH_MODELS_PAT }} GITHUB_APM_PAT: ${{ secrets.GH_CLI_PAT }} - run: uv run pytest tests/unit tests/test_console.py tests/integration/test_runtime_smoke.py -n auto --dist worksteal + run: uv run pytest tests/integration/test_runtime_smoke.py -v - name: Install UPX (Linux) if: matrix.platform == 'linux' diff --git a/CHANGELOG.md b/CHANGELOG.md index 30d8ef985..e316fd4e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- CI: smoke tests in `build-release.yml`'s `build-and-test` job (Linux x86_64, Linux arm64, Windows) are now gated to promotion boundaries (tag/schedule/dispatch) instead of running on every push to main. Push-time smoke duplicated the merge-time smoke gate in `ci-integration.yml` and burned ~15 redundant codex-binary downloads/day. Tag-cut releases still run smoke as a pre-ship gate; nightly catches upstream codex URL drift; merge-time still gates merges into main. - CI docs: clarify that branch-protection ruleset must store the check-run name (`gate`), not the workflow display string (`Merge Gate / gate`); document the merge-gate aggregator in `cicd.instructions.md` and mark the legacy stub workflow as deprecated. ### Removed From 58c40e728dfdb187d2d2db5822b2d8a484fed449 Mon Sep 17 00:00:00 2001 From: danielmeppiel Date: Thu, 23 Apr 2026 17:28:28 +0200 Subject: [PATCH 2/2] ci(build-release): sync .apm canonical + add PR ref to changelog Address PR #878 review: 1. Sync .apm/instructions/cicd.instructions.md (canonical source per #823) with .github/instructions/cicd.instructions.md so future apm install --target copilot regenerations don't revert the build-release smoke-gating doc note (and to bring along the stub-removal changes from #875 + branch-protection refinement from #874 that had also drifted). 2. Append (#878) suffix to the new CHANGELOG entry, matching the established Keep-a-Changelog convention used by neighbouring entries. No workflow behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .apm/instructions/cicd.instructions.md | 43 +++++++++++++++----------- CHANGELOG.md | 2 +- 2 files changed, 26 insertions(+), 19 deletions(-) diff --git a/.apm/instructions/cicd.instructions.md b/.apm/instructions/cicd.instructions.md index 8c60cbda5..d98c27884 100644 --- a/.apm/instructions/cicd.instructions.md +++ b/.apm/instructions/cicd.instructions.md @@ -6,7 +6,7 @@ description: "CI/CD Pipeline configuration for PyInstaller binary packaging and # CI/CD Pipeline Instructions ## Workflow Architecture (Tiered + Merge Queue) -Four workflows split by trigger and tier. PRs get fast feedback; the heavy +Five workflows split by trigger and tier. PRs get fast feedback; the heavy integration suite runs only at merge time via GitHub Merge Queue (microsoft/apm#770). @@ -24,27 +24,28 @@ integration suite runs only at merge time via GitHub Merge Queue cross-workflow artifact plumbing across triggers. - **Never add a `pull_request` or `pull_request_target` trigger here.** This file holds production secrets (`GH_CLI_PAT`, `ADO_APM_PAT`). - Required-check satisfaction at PR time is handled by the inert stub - `ci-integration-pr-stub.yml` instead. -3. **`ci-integration-pr-stub.yml`** - inert PR-time stub for required checks - - Triggers on `pull_request_target` so the YAML is read from `main` - (admin-controlled) regardless of PR head contents - applies retroactively - to existing fork PRs without rebase. - - `permissions: {}`, no secrets, no checkout, four no-op `echo` jobs whose - names match the four Tier 2 required checks. Reports success in seconds. - - Concurrency group keyed on PR number cancels in-flight stub runs on - subsequent pushes. - - Activity types include `labeled/unlabeled/edited` so maintainers can - re-trigger the stub without forcing contributors to push commits. + Required-check satisfaction at PR time is handled by `merge-gate.yml`, + which aggregates all required signals into a single `gate` check. +3. **`merge-gate.yml`** - single-authority PR-time aggregator + - Triggers on `pull_request` only (single trigger - dual-trigger with + `pull_request_target` produces SUCCESS+CANCELLED check-run twins via + `cancel-in-progress` and poisons branch protection's rollup). + - One job named `gate`. Polls the Checks API for all entries in the + workflow's `EXPECTED_CHECKS` env var; aggregates pass/fail into a + single check-run. + - Branch protection requires ONLY this one check (`gate`). Adding, + renaming, or removing an underlying check is a `merge-gate.yml` edit, + never a ruleset edit. Tide / bors single-authority pattern. + - Recovery if the `pull_request` webhook is dropped: empty commit, + `gh workflow run merge-gate.yml -f pr_number=NNN`, or close+reopen. - `.github/CODEOWNERS` requires Lead Maintainer review for any change - to `.github/workflows/**` to prevent inadvertent additions of secrets, - checkout, or PR-data interpolation to this file. -3. **`build-release.yml`** - `push` to main, tags, schedule, `workflow_dispatch` - - **Linux + Windows** run combined `build-and-test` (unit tests + binary build in one job). + to `.github/workflows/**`. +4. **`build-release.yml`** - `push` to main, tags, schedule, `workflow_dispatch` + - **Linux + Windows** run combined `build-and-test` (unit tests + binary build in one job). Unit tests run on every push for platform-regression signal; **smoke tests are gated to tag/schedule/dispatch only** (promotion boundaries) to avoid duplicating `ci-integration.yml`'s merge-time smoke and to cut redundant codex-binary downloads. - **macOS Intel** uses `build-and-validate-macos-intel` (root node, runs own unit tests - no dependency on `build-and-test`). Builds the binary on every push for early regression feedback; integration + release-validation phases conditional on tag/schedule/dispatch. - **macOS ARM** uses `build-and-validate-macos-arm` (root node, tag/schedule/dispatch only - ARM runners are extremely scarce with 2-4h+ queue waits). Only requested when the binary is actually needed for a release. - Secrets always available. Full 5-platform binary output (linux x86_64/arm64, darwin x86_64/arm64, windows x86_64). -4. **`ci-runtime.yml`** - nightly schedule, manual dispatch, path-filtered push +5. **`ci-runtime.yml`** - nightly schedule, manual dispatch, path-filtered push - **Linux x86_64 only**. Live inference smoke tests (`apm run`) isolated from release pipeline. - Uses `GH_MODELS_PAT` for GitHub Models API access. - Failures do not block releases - annotated as warnings. @@ -86,6 +87,12 @@ integration suite runs only at merge time via GitHub Merge Queue - **Artifact Retention**: 30 days for debugging failed releases - **Cross-workflow artifacts**: ci-integration.yml builds the binary inline (no cross-workflow artifact transfer); build-release.yml jobs share artifacts within the same workflow run. +## Branch Protection & Required Checks +- **Single required check**: branch protection (`main-protection` ruleset id 9294522) requires exactly one status check context: `gate` from `merge-gate.yml`. All other PR-time signals are aggregated by that workflow's poll loop. +- **CRITICAL ruleset gotcha**: the ruleset `context` must be the literal check-run name `gate`. `Merge Gate / gate` is only how GitHub may render the workflow and job together in the UI; it is not the context value to store in the ruleset. If the ruleset stores `Merge Gate / gate`, GitHub waits forever with "Expected - Waiting for status to be reported" because no check-run with that literal name is posted. +- **How the name is derived**: GitHub matches the check by `integration_id` (`15368` = github-actions) plus the emitted check-run name. That emitted name comes from the job `name:` if one is set; otherwise it falls back to the job id. In `merge-gate.yml` the job id is `gate` and `name: gate`, so the emitted check-run name is `gate` -- that is the exact string the ruleset must require. +- **Adding a new aggregated check**: add it to `EXPECTED_CHECKS` in `merge-gate.yml`. Do not change the ruleset unless you intentionally rename the merge gate job's emitted check-run name, in which case the ruleset `context` must be updated to the new exact name. + ## Trust Model - **PR push (any contributor, including forks)**: Runs Tier 1 only. No CI secrets exposed. PR code is checked out and tested in an unprivileged context. - **merge_group (write access required)**: Runs Tier 1 + Tier 2. Tier 2 sees secrets. The `gh-readonly-queue/main/*` ref is created by GitHub from the PR merged into main; only users with write access can trigger this by enqueueing a PR. diff --git a/CHANGELOG.md b/CHANGELOG.md index e316fd4e4..8a590e534 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- CI: smoke tests in `build-release.yml`'s `build-and-test` job (Linux x86_64, Linux arm64, Windows) are now gated to promotion boundaries (tag/schedule/dispatch) instead of running on every push to main. Push-time smoke duplicated the merge-time smoke gate in `ci-integration.yml` and burned ~15 redundant codex-binary downloads/day. Tag-cut releases still run smoke as a pre-ship gate; nightly catches upstream codex URL drift; merge-time still gates merges into main. +- CI: smoke tests in `build-release.yml`'s `build-and-test` job (Linux x86_64, Linux arm64, Windows) are now gated to promotion boundaries (tag/schedule/dispatch) instead of running on every push to main. Push-time smoke duplicated the merge-time smoke gate in `ci-integration.yml` and burned ~15 redundant codex-binary downloads/day. Tag-cut releases still run smoke as a pre-ship gate; nightly catches upstream codex URL drift; merge-time still gates merges into main. (#878) - CI docs: clarify that branch-protection ruleset must store the check-run name (`gate`), not the workflow display string (`Merge Gate / gate`); document the merge-gate aggregator in `cicd.instructions.md` and mark the legacy stub workflow as deprecated. ### Removed