feat(devx): patrol the merge queue's head entry for a missing merge_group build - #7857
Merged
Merged
Conversation
…roup build A queue entry can sit at the HEAD of the merge queue for which GitHub never dispatches `merge_group`. Nothing is red, nothing is ejected, and every entry behind it builds green and never merges, because a merge queue is strictly ordered. Four occurrences are on record; the worst ran four hours; nothing was watching for any of them. `scripts/check-merge-queue-head.mjs` takes the two-call reading the incident record already verified in both directions, on the HEAD entry only: list the live `gh-readonly-queue/<base>/*` refs, pick the one stacked on the base branch's current tip, and count its `merge_group` runs. Zero runs, past a 5-minute threshold on both clocks and confirmed by a second sample, is a wedge. Only the head is judged, and that is correctness rather than economy: zero runs is the NORMAL state of an entry outside the queue's speculative build window (measured — one healthy entry waited 877s for its turn). The verdict is refused before it is rendered: `clear` is unreachable without an identified head and a positive run count, so a patrol looking at nothing cannot report a healthy queue. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDq78vMMSzCGWGmhUYBabh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7010
Scope is the patrol half only, per the central triage ruling of 2026-09-04T22:32Z on that card. The other half — why GitHub declines to dispatch
merge_groupfor these entries — is a repository/Actions-settings reading no agent seat can take, it is deliberately not attempted here, and this pull request is not hung on it.What lands
scripts/check-merge-queue-head.mjs--self-testscripts/__tests__/check-merge-queue-head.test.ts.github/workflows/merge-queue-head-patrol.ymlpackage.jsoncheck:merge-queue-head(offline) andmerge-queue-head(live read)content/docs/guide/ci-cd-pipeline.mdci-cd-pipeline-doc.test.tsfails any workflow with no heading naming itThe reading, re-derived rather than transcribed
The card's 2026-08-31T13:12Z comment describes a two-call check. It was re-derived against the live API instead of trusted, and three things in the transcription turned out to be wrong or incomplete:
merge_grouprun is 3–24 seconds, not the same second. The conclusion survives; the number in the constant's comment is the measured range.merge_groupruns" is NOT by itself a wedge signature. One healthy entry in that population (pr-7815, sixth in a six-deep chain) waited 877 seconds for its first run, because GitHub builds only the first few entries speculatively and it was outside that window. A patrol that judged every queue entry would report a wedge on every healthy busy queue. This is why the check is taken on the head entry only — position 1 is always inside the build window. The card's ruling already said "head entry"; this is the measurement of why it has to be.mainadvanced. A queue commit is created when its group forms and lands minutes later — measured 15m51s apart on52cac388. "How long hasmainbeen static" is read from the newestpushworkflow run instead, and a push run whosehead_shais not the current tip is refused as stale rather than used.Two API shapes were also measured and are load-bearing:
GET /actions/runs?branch=answerstotal_count: 0with HTTP 200 for a branch that does not exist. So the head's ref is taken verbatim fromgit/matching-refsand never assembled from a pull number and a sha — a constructed ref one character off would report a wedge on a healthy queue, with a completely plausible message.GET /commits/{ref}answers 422 for a ref containing slashes, encoded or not, and every queue ref contains three. The entry is read by the sha the refs listing returns alongside it, which costs no extra call.Proven against the live API, in both directions
The
CLEARleg replays the 23:31Z moment with the run count and the commit read hitting the live API. Full end-to-end live run againstobjectstack-ai/objectuireturned exit 0 with a correctEMPTYreading.The threshold: one constant, both boundaries beside it
WEDGE_THRESHOLD_MS = 5 minutes, ruled by the lane. Its docblock carries the two readings it sits between so the next person retunes it against evidence: 3–24 seconds (measured healthy dispatch, n=18) below, 60 minutes (the ruleset status-check timeout that self-heals it, measured on the 09-02 instance) above. A suspected wedge is confirmed by a second run count 60 seconds later — 2.5x the slowest measured dispatch — so "wedged for an hour" is never confused with the seconds after a head change, which is the one case where both clocks read old but the entry is new.⭐ The anti-vacuous leg — the queue is healthy, so the corpus is clean
origin/mainwas measured advancing normally the whole time this was written. There is no live wedge to point the detector at, and there may not be another for weeks. A detector in that position fails in exactly one direction: it stops being able to see a wedge, and every run afterwards reports a healthy queue with total confidence.Three things stand against that:
assertGroundedrefuses the verdict before it is rendered.clearis unreachable without an identified head and a positive run count;wedgedis unreachable without two zero samples and both clocks past the threshold. "Could not identify the head" is its own verdict with its own wording and never renders as a healthy queue — the same rulehalf-state-patrol.ymlstates as could-not-read must never look like clean.#7283instance in its own shape. If the parser narrows, the head selector stops resolving, or the count is read off a renamed field, those go red.classifyQueue's wedge branch replaced bysettlingon disk (verified by grep count 1 to 0 and a changed blob hash, not by the editor's exit code), the suite goes 6 failed / 34 passed, headed by the anti-vacuity test. Restored bygit checkout HEAD --, verified byte-identical (same blob hash, emptygit diff HEAD), and green again at 40/40.What it does when it fires — one anchor, never a card per run
⛔ It never opens an issue on any code path (pinned). Delivery is: the run summary always; a PATCH of one pinned issue body when the repository variable
MERGE_QUEUE_ANCHOR_ISSUEnames one; and a red job.The red job is a deliberate divergence from
half-state-patrol.yml, which is report-only throughout. The reason is specific to this defect: the remedy is a human removing the entry from the queue inside a 60-minute window, and an issue-body edit notifies nobody. It cannot become routine — an empty queue, a settling head and an unreadable reading all exit 0, and the finding has occurred four times in three weeks.The anchor is optional here, which is the second divergence:⚠️ That is not hypothetical — see the finding filed below.
half-state-patrol.ymlfails when its anchor variable is unset. This one does not, because a patrol running every 15 minutes would then be red 96 times a day over a missing setting, and this repository has ruled on that shape (objectui#6596).Optional install step for a maintainer: open an issue and set
MERGE_QUEUE_ANCHOR_ISSUEto its number (Settings, then Secrets and variables, then Actions, then Variables). Nothing else changes.Cost, since it was raised as unmeasured
This repository is public, so scheduled Actions minutes are not billed. The job is a checkout plus one
nodecall — nopnpm install, andcheck:pre-install-import-graphnow derives this step into its population and holds the property. A healthy run makes 3 API reads.Cadence arithmetic, so it is retunable against the same numbers: the wedge self-heals in ~60 minutes and the threshold is 5, so
*/15reports it 5–20 minutes in, with two thirds of the wasted hour still recoverable.*/5buys ~10 minutes at three times the runs; hourly would routinely report a wedge that had already healed. The minute is offset off:00so the patrol does not queue behind everyone else's top-of-hour schedules.⛔ No
pull_requestleg — and why that is not lazinessEvery job of a
pull_request-triggered workflow produces a check run, andscripts/dependabot-merge-gate.mjsrequires every produced name to be classified acrossREQUIRED_CONTEXTS/OPTIONAL_CONTEXTS/NOT_A_GATE— a partitiondependabot-merge-gate.test.tsasserts exactly. A leg here would mean editing that declaration from a card that does not hold that file. Instead the offline--self-testruns on every pull request through the pin, and the live transport is proven byworkflow_dispatch. The pin asserts the absence and names the other half of the work if anyone adds a leg later.Gates run locally
Derived from objectui's own root
package.jsonand workflow files for the five files actually changed. ⛔ Not inherited from a list, and objectstack'sdispatch-gates.mjsrefuses a cross-repo assertion, so nothing was derived from it.scripts/__tests__/(whole directory)tsc -p tsconfig.scripts.jsoneslint --no-inline-config(narrowed, see below)check-changeset-presence.mjscheck-changeset-fixed/-no-major/-overwritecheck-entry-guard.mjscheck-control-bytes.mjscheck-pre-install-import-graph.mjscheck-shell-escape-residue.mjscheck-doc-links.mjscheck-governed-queue-guard.mjs --test(5 changed paths)check-merge-queue-head.mjs --self-testVerdicts are quoted from each gate's own printed line; exit codes were captured with no pipe in between.
The eslint narrowing is a measurement, not a skip — the three readings that make it one, taken at
03364a4:ESLint#isPathIgnoredover every tracked.ts/.tsx), not by a guess.--format json, exit 0.eslint.config.jsconfigures no type-aware linting — noproject, noprojectService, noparserOptionsanywhere — so this diff cannot move the verdict on any file it did not touch. The new.mjsis outside eslint'sfiles: ['**/*.{ts,tsx}']entirely.skip-changesetlabel. The gate says none is owed (rootscripts/, a workflow, a doc and a root-manifest script entry publish nothing), and the label is confirmed dead on this repository with a pin test forbidding it.Deviation from the dispatched file surface, declared
content/docs/guide/ci-cd-pipeline.mdwas not in the dispatched surface. It is not optional:scripts/__tests__/ci-cd-pipeline-doc.test.tsfails any workflow in.github/workflows/that no heading on that page names, and itsDOCUMENTATION_EXEMPTmap is deliberately empty. Checked before editing — no open pull request touches that file, rootpackage.json, or any of the new paths.Out of scope, filed not fixed
#7852 — the half-state patrol has been red on all 50 scheduled runs since 2026-08-28 (its sweep passes;
HALF_STATE_ANCHOR_ISSUEis unset so the findings land nowhere), and this same doc page describes aPM_SWEEP_CLOSED_WINDOW_PAGESknob the workflow no longer sets. Found while reading that workflow as the precedent for this one; ⛔ not touched here.Generated by Claude Code