Skip to content

feat(devx): patrol the merge queue's head entry for a missing merge_group build - #7857

Merged
os-sam merged 1 commit into
mainfrom
claude/issue-7010-merge-queue-head-patrol
Sep 6, 2026
Merged

feat(devx): patrol the merge queue's head entry for a missing merge_group build#7857
os-sam merged 1 commit into
mainfrom
claude/issue-7010-merge-queue-head-patrol

Conversation

@claude

@claude claude Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #7010

Scope is the patrol half only, per the central triage ruling of 2026-09-04T22:32Z on that card. The other half — why GitHub declines to dispatch merge_group for these entries — is a repository/Actions-settings reading no agent seat can take, it is deliberately not attempted here, and this pull request is not hung on it.

What lands

file what it is
scripts/check-merge-queue-head.mjs the detector, pure + injectable, with an offline --self-test
scripts/__tests__/check-merge-queue-head.test.ts the pin, mostly anti-vacuity (see below)
.github/workflows/merge-queue-head-patrol.yml the standing caller, every 15 minutes
package.json check:merge-queue-head (offline) and merge-queue-head (live read)
content/docs/guide/ci-cd-pipeline.md inventory row + section — required, not optional: ci-cd-pipeline-doc.test.ts fails any workflow with no heading naming it

The reading, re-derived rather than transcribed

The card's 2026-08-31T13:12Z comment describes a two-call check. It was re-derived against the live API instead of trusted, and three things in the transcription turned out to be wrong or incomplete:

  1. "healthy ⇒ ~15 runs in the same second" is over-precise. Measured 2026-09-05T23:3xZ over the 18 most recent queue entries: 17 workflows now, and the gap between an entry's queue commit and its first merge_group run is 3–24 seconds, not the same second. The conclusion survives; the number in the constant's comment is the measured range.
  2. ⭐ "zero merge_group runs" is NOT by itself a wedge signature. One healthy entry in that population (pr-7815, sixth in a six-deep chain) waited 877 seconds for its first run, because GitHub builds only the first few entries speculatively and it was outside that window. A patrol that judged every queue entry would report a wedge on every healthy busy queue. This is why the check is taken on the head entry only — position 1 is always inside the build window. The card's ruling already said "head entry"; this is the measurement of why it has to be.
  3. The tip commit's timestamp is not when main advanced. A queue commit is created when its group forms and lands minutes later — measured 15m51s apart on 52cac388. "How long has main been static" is read from the newest push workflow run instead, and a push run whose head_sha is not the current tip is refused as stale rather than used.

Two API shapes were also measured and are load-bearing:

  • GET /actions/runs?branch= answers total_count: 0 with HTTP 200 for a branch that does not exist. So the head's ref is taken verbatim from git/matching-refs and never assembled from a pull number and a sha — a constructed ref one character off would report a wedge on a healthy queue, with a completely plausible message.
  • GET /commits/{ref} answers 422 for a ref containing slashes, encoded or not, and every queue ref contains three. The entry is read by the sha the refs listing returns alongside it, which costs no extra call.

Proven against the live API, in both directions

head pr-7815, live       -> total_count 17, verdict CLEAR
a ref that does not exist -> total_count 0, HTTP 200   (the trap above)
queue empty (live, 23:52Z) -> verdict EMPTY, exit 0

The CLEAR leg replays the 23:31Z moment with the run count and the commit read hitting the live API. Full end-to-end live run against objectstack-ai/objectui returned exit 0 with a correct EMPTY reading.

The threshold: one constant, both boundaries beside it

WEDGE_THRESHOLD_MS = 5 minutes, ruled by the lane. Its docblock carries the two readings it sits between so the next person retunes it against evidence: 3–24 seconds (measured healthy dispatch, n=18) below, 60 minutes (the ruleset status-check timeout that self-heals it, measured on the 09-02 instance) above. A suspected wedge is confirmed by a second run count 60 seconds later — 2.5x the slowest measured dispatch — so "wedged for an hour" is never confused with the seconds after a head change, which is the one case where both clocks read old but the entry is new.

⭐ The anti-vacuous leg — the queue is healthy, so the corpus is clean

origin/main was measured advancing normally the whole time this was written. There is no live wedge to point the detector at, and there may not be another for weeks. A detector in that position fails in exactly one direction: it stops being able to see a wedge, and every run afterwards reports a healthy queue with total confidence.

Three things stand against that:

  1. In the script. assertGrounded refuses the verdict before it is rendered. clear is unreachable without an identified head and a positive run count; wedged is unreachable without two zero samples and both clocks past the threshold. "Could not identify the head" is its own verdict with its own wording and never renders as a healthy queue — the same rule half-state-patrol.yml states as could-not-read must never look like clean.
  2. In the pin. The wedged fixture is the live healthy corpus with exactly one field changed (run count 17 to 0), plus the recorded 2026-09-02 #7283 instance in its own shape. If the parser narrows, the head selector stops resolving, or the count is read off a renamed field, those go red.
  3. Ablated, to prove the pin can fail. With classifyQueue's wedge branch replaced by settling on disk (verified by grep count 1 to 0 and a changed blob hash, not by the editor's exit code), the suite goes 6 failed / 34 passed, headed by the anti-vacuity test. Restored by git checkout HEAD --, verified byte-identical (same blob hash, empty git diff HEAD), and green again at 40/40.

What it does when it fires — one anchor, never a card per run

⛔ It never opens an issue on any code path (pinned). Delivery is: the run summary always; a PATCH of one pinned issue body when the repository variable MERGE_QUEUE_ANCHOR_ISSUE names one; and a red job.

The red job is a deliberate divergence from half-state-patrol.yml, which is report-only throughout. The reason is specific to this defect: the remedy is a human removing the entry from the queue inside a 60-minute window, and an issue-body edit notifies nobody. It cannot become routine — an empty queue, a settling head and an unreadable reading all exit 0, and the finding has occurred four times in three weeks.

The anchor is optional here, which is the second divergence: half-state-patrol.yml fails when its anchor variable is unset. This one does not, because a patrol running every 15 minutes would then be red 96 times a day over a missing setting, and this repository has ruled on that shape (objectui#6596). ⚠️ That is not hypothetical — see the finding filed below.

Optional install step for a maintainer: open an issue and set MERGE_QUEUE_ANCHOR_ISSUE to its number (Settings, then Secrets and variables, then Actions, then Variables). Nothing else changes.

Cost, since it was raised as unmeasured

This repository is public, so scheduled Actions minutes are not billed. The job is a checkout plus one node call — no pnpm install, and check:pre-install-import-graph now derives this step into its population and holds the property. A healthy run makes 3 API reads.

Cadence arithmetic, so it is retunable against the same numbers: the wedge self-heals in ~60 minutes and the threshold is 5, so */15 reports it 5–20 minutes in, with two thirds of the wasted hour still recoverable. */5 buys ~10 minutes at three times the runs; hourly would routinely report a wedge that had already healed. The minute is offset off :00 so the patrol does not queue behind everyone else's top-of-hour schedules.

⛔ No pull_request leg — and why that is not laziness

Every job of a pull_request-triggered workflow produces a check run, and scripts/dependabot-merge-gate.mjs requires every produced name to be classified across REQUIRED_CONTEXTS / OPTIONAL_CONTEXTS / NOT_A_GATE — a partition dependabot-merge-gate.test.ts asserts exactly. A leg here would mean editing that declaration from a card that does not hold that file. Instead the offline --self-test runs on every pull request through the pin, and the live transport is proven by workflow_dispatch. The pin asserts the absence and names the other half of the work if anyone adds a leg later.

Gates run locally

Derived from objectui's own root package.json and workflow files for the five files actually changed. ⛔ Not inherited from a list, and objectstack's dispatch-gates.mjs refuses a cross-repo assertion, so nothing was derived from it.

gate result
scripts/__tests__/ (whole directory) 105 files / 3101 tests passed
tsc -p tsconfig.scripts.json exit 0
eslint --no-inline-config (narrowed, see below) exit 0, 1 file, 0 errors / 0 warnings
check-changeset-presence.mjs exit 0 — 5 files changed, 0 published source, no changeset owed
check-changeset-fixed / -no-major / -overwrite exit 0
check-entry-guard.mjs exit 0
check-control-bytes.mjs exit 0
check-pre-install-import-graph.mjs exit 0 — 24 steps / 22 gates / 25 modules, every non-relative leaf a node builtin
check-shell-escape-residue.mjs exit 0
check-doc-links.mjs exit 0
check-governed-queue-guard.mjs --test (5 changed paths) NOT GOVERNED — none of the 5 governed surfaces matched
check-merge-queue-head.mjs --self-test exit 0, 41 cases

Verdicts are quoted from each gate's own printed line; exit codes were captured with no pipe in between.

The eslint narrowing is a measurement, not a skip — the three readings that make it one, taken at 03364a4:

  1. Universe 4199 files, decided by eslint's own config (ESLint#isPathIgnored over every tracked .ts/.tsx), not by a guess.
  2. Files linted 1, read from --format json, exit 0.
  3. Invariance: eslint.config.js configures no type-aware linting — no project, no projectService, no parserOptions anywhere — so this diff cannot move the verdict on any file it did not touch. The new .mjs is outside eslint's files: ['**/*.{ts,tsx}'] entirely.

⚠️ No changeset, and no skip-changeset label. The gate says none is owed (root scripts/, a workflow, a doc and a root-manifest script entry publish nothing), and the label is confirmed dead on this repository with a pin test forbidding it.

Deviation from the dispatched file surface, declared

content/docs/guide/ci-cd-pipeline.md was not in the dispatched surface. It is not optional: scripts/__tests__/ci-cd-pipeline-doc.test.ts fails any workflow in .github/workflows/ that no heading on that page names, and its DOCUMENTATION_EXEMPT map is deliberately empty. Checked before editing — no open pull request touches that file, root package.json, or any of the new paths.

Out of scope, filed not fixed

#7852 — the half-state patrol has been red on all 50 scheduled runs since 2026-08-28 (its sweep passes; HALF_STATE_ANCHOR_ISSUE is unset so the findings land nowhere), and this same doc page describes a PM_SWEEP_CLOSED_WINDOW_PAGES knob the workflow no longer sets. Found while reading that workflow as the precedent for this one; ⛔ not touched here.


Generated by Claude Code

…roup build

A queue entry can sit at the HEAD of the merge queue for which GitHub never
dispatches `merge_group`. Nothing is red, nothing is ejected, and every entry
behind it builds green and never merges, because a merge queue is strictly
ordered. Four occurrences are on record; the worst ran four hours; nothing was
watching for any of them.

`scripts/check-merge-queue-head.mjs` takes the two-call reading the incident
record already verified in both directions, on the HEAD entry only: list the
live `gh-readonly-queue/<base>/*` refs, pick the one stacked on the base
branch's current tip, and count its `merge_group` runs. Zero runs, past a
5-minute threshold on both clocks and confirmed by a second sample, is a wedge.

Only the head is judged, and that is correctness rather than economy: zero runs
is the NORMAL state of an entry outside the queue's speculative build window
(measured — one healthy entry waited 877s for its turn).

The verdict is refused before it is rendered: `clear` is unreachable without an
identified head and a positive run count, so a patrol looking at nothing cannot
report a healthy queue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDq78vMMSzCGWGmhUYBabh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ci] the merge queue has landed NOTHING since 07:58Z — five merge_group builds completed success and main has not moved in 90 minutes

1 participant