ci: adopt GitHub Merge Queue with tiered CI (#770)

[danielmeppiel]

Implements #770.

Summary

Restructure CI into two tiers gated by GitHub native Merge Queue:

• Tier 1 (ci.yml) — unit tests + binary build. Runs on every PR push and on merge_group events. No secrets, fork-safe, fast feedback (~2-4 min).
• Tier 2 (ci-integration.yml) — smoke + integration + release-validation. Runs only on merge_group events against the queue's tentative merge commit. Heavy suite no longer fires on every WIP push.

What this fixes

1. Manual "Update branch" clicks — merge queue tests the tentative merge commit, so PRs are always validated against current main. No more rebase loops.
2. Per-PR-push approval bottleneck — Tier 2 was previously gated by the integration-tests environment requiring manual approval on every push. With merge queue, the trust boundary becomes the write-access grant: only users with write can enqueue, and only enqueued runs see CI secrets. Fork PRs without write access run only Tier 1.

Files changed

• .github/workflows/ci.yml — added merge_group trigger.
• .github/workflows/ci-integration.yml — rewritten as merge_group-only; inlines the binary build so the suite tests exactly the tentative merge SHA without cross-workflow artifact plumbing; drops workflow_run, approve-fork, approve-internal, report-status jobs.
• CONTRIBUTING.md — added "How merging works" section.
• .github/instructions/cicd.instructions.md — rewrote workflow architecture, trust model, release flow, and performance sections.
• CHANGELOG.md — Unreleased > Changed entry.

Required follow-up (repo Settings, cannot be done in YAML)

After this PR merges, a maintainer must:

1. Enable merge queue on main in branch protection (Settings -> Branches -> main).
2. Update required status checks to the names produced by the first merge_group run. Expected:
   • From ci.yml (PR + merge_group): Build & Test (Linux)
   • From ci-integration.yml (merge_group only): Build (Linux), Smoke Test (Linux), Integration Tests (Linux), Release Validation (Linux)
3. Disable "Require branches to be up to date before merging" — the queue handles that.
4. Retire the integration-tests environment (Settings -> Environments) — no longer used.

Risks

The ci-integration.yml rewrite is the riskiest piece — losing the workflow_run machinery means any bug ships at merge time. The first few merges should be watched closely. If the queue stalls, disabling merge queue in branch protection takes one click and reverts to the previous PR-merge model without any YAML changes.

Out of scope

• Mergify/Kodiak — GitHub native merge queue removes the need.
• Required reviews policy — orthogonal.
• Test parallelization / sharding — separate optimization.

Closes part of #770.

[Copilot]

Pull request overview

This PR restructures the repository CI to support GitHub Merge Queue with a tiered model: fast PR checks (Tier 1) and merge-queue-only integration checks (Tier 2), plus accompanying documentation updates.

Changes:

• Add merge_group trigger to Tier 1 CI so unit tests + binary build run on merge queue tentative merge commits.
• Rewrite Tier 2 workflow to be merge_group-only and build the binary inline before smoke/integration/release-validation.
• Document the merge queue flow in CONTRIBUTING and CI internal instructions, and add a changelog entry.

Show a summary per file

File	Description
CONTRIBUTING.md	Documents the merge queue-based merge process for contributors.
CHANGELOG.md	Adds an Unreleased entry describing the CI/merge-queue change.
.github/workflows/ci.yml	Enables Tier 1 to run on merge_group in addition to PRs.
.github/workflows/ci-integration.yml	Converts Tier 2 to merge_group-only and inlines binary build + artifact passing.
.github/instructions/cicd.instructions.md	Updates CI architecture/trust model docs to reflect tiered merge queue design.

Copilot's findings

Comments suppressed due to low confidence (4)

.github/instructions/cicd.instructions.md:65

• This section uses non-ASCII arrows (→) in the job sequence description. Repo encoding rules require printable ASCII in .github/ Markdown files; replace → with -> (and keep the rest of the section ASCII-only).

## Release Flow Dependencies
- **PR workflow**: Tier 1 only — ci.yml (build-and-test, Linux-only) provides fast feedback. Tier 2 does not run until enqueued.
- **Merge queue workflow**: ci.yml (Tier 1 against tentative merge ref) + ci-integration.yml (Tier 2: build → smoke-test → integration-tests → release-validation). Queue auto-merges on success; ejects on failure.
- **Push/Release workflow (Linux + Windows)**: build-and-test → integration-tests → release-validation → create-release → publish-pypi → update-homebrew (gh-aw-compat runs in parallel, informational)

.github/instructions/cicd.instructions.md:88

• Non-ASCII arrow character (→) is used here ("setup copilot→ npm install"). Encoding rules require printable ASCII in.github/instruction files; replace with->`.

- **Native uv caching**: `setup-uv` action with `enable-cache: true` replaces manual `actions/cache@v3` blocks.
- **Targeted setup-node usage**: Node.js is only installed in `ci-runtime.yml`, macOS consolidated jobs, and integration-tests/release-validation phases (for `apm runtime setup copilot` → npm install).
- **macOS runner consolidation**: Each macOS arch has a single consolidated job (build + integration + release-validation). Intel (`build-and-validate-macos-intel`) runs on every push since Intel runners are plentiful. ARM (`build-and-validate-macos-arm`) is gated to tag/schedule/dispatch only since ARM runners are extremely scarce (2-4h+ queue waits). This avoids serial re-queuing of runners across multiple jobs.

.github/workflows/ci-integration.yml:167

• Same nesting issue as the integration job: downloading apm-mq-linux-x86_64 into /tmp/apm-isolated-test/dist/apm-linux-x86_64/ will likely create /tmp/apm-isolated-test/dist/apm-linux-x86_64/apm-linux-x86_64/apm, but the subsequent chmod +x ./dist/apm-linux-x86_64/apm assumes the binary is at /tmp/apm-isolated-test/dist/apm-linux-x86_64/apm. Download into /tmp/apm-isolated-test/dist/ (or preserve dist/ in the artifact) so the binary ends up at the expected location.

      - name: Download binary
        uses: actions/download-artifact@v4
        with:
          name: apm-mq-linux-x86_64
          path: /tmp/apm-isolated-test/dist/apm-linux-x86_64/

.github/instructions/cicd.instructions.md:60

• Non-ASCII characters appear here (e.g., the multiplication sign in 8 inference executions × 2% failure rate and em dashes). Encoding rules for .github/ docs require printable ASCII only; use x instead of × and replace em dashes with -/--.

## Inference Testing (Decoupled)
- Live inference tests (`apm run`) are **isolated** in `ci-runtime.yml` — they do NOT gate releases
- `APM_RUN_INFERENCE_TESTS=1` env var enables inference in test scripts; absent = skipped
- `GH_MODELS_PAT` is only used in `ci-runtime.yml` and Tier 2 smoke-test job — NOT in integration-tests or release-validation
- Rationale: 8 inference executions × 2% failure rate = 14.9% false-negative per release; APM core UVPs require zero live inference

• Files reviewed: 5/5 changed files
• Comments generated: 5

[Copilot]

The artifact download destination creates an extra nested directory. This workflow uploads ./dist/apm-linux-x86_64 (which becomes apm-linux-x86_64/ at artifact root because dist/ is the shared prefix), but downloads into ./dist/apm-linux-x86_64/, producing ./dist/apm-linux-x86_64/apm-linux-x86_64/apm. scripts/test-integration.sh expects ./dist/apm-linux-x86_64/apm, so the integration job will fail. Download the artifact into ./dist/ (or adjust the uploaded paths so dist/ is preserved) so the final path is ./dist/apm-linux-x86_64/apm.

This issue also appears on line 162 of the same file.

Suggested change

	path: ./dist/apm-linux-x86_64/
	path: ./dist/