fix: route merge_group jobs to the dedicated arc-merge-queue pool (BLO-22428)

[kkroo]

Replacement for app-authored #1080 so the Ally GitHub App can provide the required independent review/approval.

Exact code head copied from #1080: d73f38008bcd3e48dd02bd8e341c0f9f98c3c3b1.

Thinking Path

The presenting symptom was Blockcast/paperclip's master merge queue sitting 53 entries deep with a 10h47m gap between merges, while arc-default was pinned at 30/30 runner pods. The obvious reading — "we are out of CI capacity, add runners" — is wrong, and following it would have cost a cross-team capacity trade for no benefit.

The disconfirming measurement is GitHub's merge-queue max_entries_to_build, which is 1 on ruleset 20487141. The queue builds one entry at a time, so its runner demand is flat and small regardless of whether it is 5 deep or 53 deep. Queue depth is not check demand. What starved the queue was not insufficient total capacity but ordering: merge_group jobs competed for the same default label as an unbounded pull_request backlog, and lost.

That reframes the fix. The queue does not need capacity proportional to its depth; it needs a small pool that pull_request traffic cannot touch. So instead of raising arc-default past its documented 30-slot ceiling — which arc/arc-default-values.yaml states is derived imagefs arithmetic (2625Gi x 1.10 = 2887Gi max; 30 x 38Gi = 2876Gi, 11Gi slack) and whose raise "is a cross-team decision about whose capacity shrinks, not a knob turn" — the companion PR carves arc-default 30 -> 22 and adds arc-merge-queue at 8. Total stays 30, the ledger is untouched, and no team's capacity shrinks. This PR is the other half: routing the traffic onto that new pool.

Rejected alternatives: raising the ceiling by shrinking a named idle pool (spends a cross-team trade on the wrong axis, and a single 0-pod sample does not establish sustained idleness); and accepting the ceiling while throttling PR CI demand (treats the symptom, leaves the starvation mechanism intact).

What Changed

.github/workflows/pr.yml — the six heavy jobs shared between the pull_request and merge_group events (typecheck_release_registry, worktree_install, general_tests, build, verify_serialized_server, canary_dry_run) move from runs-on: default to a conditional:

runs-on: ${{ github.event_name == 'merge_group' && 'arc-merge-queue' || 'default' }}

merge_group runs land on the new dedicated pool; pull_request runs keep the shared default label. On pull_request the ternary evaluates to default, so PR CI behaviour is unchanged byte-for-byte.

pr.yml is the only workflow in the repo carrying a merge_group: trigger, so no straggler job is left starving on default. The runs-on values not touched here are 3x arc-light and 1x arc-e2e, both already partitioned.

scripts/check-github-runner-labels.mjs — adds arc-merge-queue to the allow-list, and teaches the guard to parse exactly the ${{ <cond> && 'X' || 'Y' }} shape so it validates both literal branches. Previously it treated any ${{ ... }} value as one opaque runner name, which would have flagged the new expression as an unknown label. Any other expression shape still falls through to the opaque-string path and fails closed.

scripts/check-github-runner-labels.test.js — three regression cases for the new parsing: a valid ternary with both branches allowed, one with a disallowed branch, and an unrelated opaque expression the parser cannot decompose (must still be rejected).

Risks

• Ordering risk, and it is the real one. This PR must not merge before Blockcast/onprem-k8s#2115, which creates the arc-merge-queue scale set. If this routing goes live first, every merge_group job targets a label with zero live listeners and queues indefinitely — the same starvation this change exists to fix, relocated to a new label. This PR must therefore not merge until feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 is merged and arc-merge-queue pods are confirmed live in arc-runners. That gate is enforced by the sequencing hold on the Paperclip issue and by board approval 25fe072c — not by draft status, which has been lifted so the review gates can run.
• Unavoidable degraded window between the two merges. Once feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 merges, arc-default drops to 22 while master's pr.yml still routes all six merge_group jobs to default — strictly worse than today until this PR lands. Running both pools at full size to avoid the window is not available: 30+8 slots is 3180Gi against a 2887Gi ceiling. The window is structural; the only lever is its duration, so this PR should land within minutes of feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115, not via the 53-deep queue.
• Mild oversubscription of the new pool. Peak merge_group concurrency is ~10 jobs, not 8: the 4 verify_serialized_server shards are gated behind general_tests, but typecheck + worktree_install + 6x general_tests + build + canary_dry_run overlap. Against maxRunners: 8 two jobs wait one slot-turn — materially better than starving behind 53 PRs, but the companion values-file comment's word "comfortably" is optimistic.
• Guard parsing is deliberately narrow. The regex matches one ternary shape only. A future author writing a differently-shaped expression gets a hard failure rather than a silent pass — intended (fail closed), but it will look like a false positive to whoever hits it first.
• Rollback is cheap. git revert this commit and all six jobs return to runs-on: default immediately; the arc-merge-queue pool then simply sits idle. No CI breakage. Reverting feat: PAPERCLIP_APP_NAME — rebrand self-hosted deployment without forking paperclipai/paperclip#2115 separately restores arc-default to 30.

Model Used

• Authored by PlatformSREEngineer (claude-sonnet-5[1m]), which also verified the ledger arithmetic and ran the guard suites locally.
• Reviewed, and this description restructured, by CTO (claude-opus-5[1m]). The CTO review independently confirmed the max_entries_to_build: 1 measurement that motivates partitioning over a capacity raise, checked that the companion ledger change nets to zero, and flagged the ~10-vs-8 peak-concurrency gap above.

Verification

• python3 -c "import yaml; yaml.safe_load(open('.github/workflows/pr.yml'))" — YAML still parses.
• node ./scripts/check-github-runner-labels.mjs — Validated 20 workflows: all runner labels use ARC.
• node --test scripts/check-github-runner-labels.test.js — 8/8 pass, including the 3 new ternary-parsing cases.
• node --test scripts/__tests__/pr-verify-lane-outcome.test.mjs — unaffected, 13/13 pass (does not assert on runs-on).
• Full build/typecheck is not runnable in the authoring environment (no local Postgres/build toolchain); CI is the gate for the rest.

Post-merge verifying signal: a merge_group run observed picking up a runner on the arc-merge-queue pool, and merge-queue drain rate measured by merged_at (not commit dates — under a REBASE queue a commit's committer date is its enqueue time).

Paperclip issue: https://paperclip.blockcast.net/BLO/issues/BLO-22428
Related: https://paperclip.blockcast.net/BLO/issues/BLO-21953

Related PRs

Searched the open and recent PR list for duplicates before filing this checklist. Nothing duplicates this change; two are adjacent and worth reading together:

• #1042 — ci: load-shed stale merge_group generations superseded by a queue re-stage (BLO-21953). Same incident, different mechanism (sheds superseded generations rather than partitioning capacity). Complementary, not overlapping.
• #1077 — fix(ci): run commitperclip-review against the merge queue's landing commit. Touches the review gate, not runs-on: routing.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template — described in-PR above; tracking issues live in Paperclip (BLO-22428 / BLO-21953), not GitHub Issues
• I have run tests locally and they pass — the guard suites listed under Verification; the full build/typecheck is not runnable in the authoring environment, so CI is the gate for the rest
• I have added or updated tests where applicable — 3 new cases for the ternary parsing
• If this change affects the UI, I have included before/after screenshots — N/A, CI configuration and a build script only
• I have updated relevant documentation to reflect my changes — N/A; the rationale is captured in inline runs-on: comments and in the companion arc-merge-queue-values.yaml
• I have considered and documented any risks above
• All Paperclip CI gates are green — not yet; four verify_serialized_server shards are still running and the companion PR's gate is red for an unrelated reason (see BLO-22428)
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups — not run on this repo
• I will address all Greptile and reviewer comments before requesting merge

[kkroo]

@ally please review head d73f380 for fix: route merge_group jobs to the dedicated arc-merge-queue pool (BLO-22428).

This replaces app-authored #1080 under independent PR author kkroo so the Ally GitHub App can approve.

[allyblockcast]

🔗 Paperclip issue: BLO-21953
🔗 Paperclip issue: BLO-22428

[allyblockcast]

🔗 Paperclip issue: BLO-21953
🔗 Paperclip issue: BLO-22428

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[allyblockcast]

Ally — Consolidated PR Review

Lenses: pr-review-toolkit (code, tests, comments, errors, types) + gstack/review + native-codex.
Reviewed head: d73f380

Critical Issues (0)

Important Issues (1)

• [pr-review-toolkit + gstack/review + native-codex] scripts/check-github-runner-labels.mjs:18 — The new regex can accept a chained expression containing an earlier forbidden runner branch. Its unrestricted prefix backtracks to the final && 'arc-merge-queue' || 'default', so ${{ cond && 'ubuntu-latest' || other && 'arc-merge-queue' || 'default' }} is accepted while the checker validates only arc-merge-queue and default. This makes the ARC-only policy fail open despite the comment's fail-closed guarantee.
  • Restrict the condition prefix to the one supported expression (or parse and validate every branch), and add a regression test with a forbidden earlier branch in a chained expression.

Suggestions (1)

• [native-codex] .github/workflows/pr.yml:212 — Keep merge blocked until Blockcast/onprem-k8s#2115 has landed and live arc-merge-queue listeners are verified. The current routing is correct, but merging it first would send all six merge-group lanes to an unavailable label.

Strengths

• The current workflow expression correctly keeps pull-request jobs on default and routes all six shared heavy merge-group jobs to the dedicated pool.
• The tests cover the intended two-branch expression, a forbidden selected branch, and unsupported opaque expressions.
• CI is green on the reviewed head, including the policy guard and all serialized server shards.

Recommended Action

1. Tighten the runner-expression parser and add the chained-expression regression test before merge.
2. Preserve the documented infrastructure-first rollout order.