Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 19 additions & 3 deletions docs/BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8289,19 +8289,35 @@ PR mergeState failing pending auto-merge

(Nine at the hand survey; eight when [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) ran ~20 minutes later, because #120 had been re-synced in between. The set moves — the condition does not.)

Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again. A hand-coordinated merge freeze across five sessions *did* hold `main` still for a full window — and #119 still failed, on an unrelated timeout (#344) — which is the evidence that hand coordination is not the fix.
Two worked instances the same day. **#74** went green on 2026-07-30 and sat unmergeable until 2026-08-01, found only by someone hunting "stuck CI" by hand; it took three merges from `main` to land. **#119** was green with 25 passing checks and armed, stalled, was re-synced, and lost the window again — on two separate heads its last required check completed *after* another PR had already merged (by 2m11s on `2a2900cb`, by 2m38s on `8c407fb5`), so it was never simultaneously green and up to date. A hand-coordinated merge freeze was declared across the parallel sessions and did **not** hold `main` still: `main` advanced four times between #119's first fully-green head and its merge, one of those 8m26s after the freeze was recorded in a work claim. #119 did land in the end, at 2026-08-02T01:45:00Z, 12h15m after auto-merge was armed. That is the evidence that hand coordination is not the fix.

**Why:** the cost is finished work sitting undelivered while everyone believes it is landing. This is the repo's recurring defect shape — a signal accurate about what it looks at and silent about what it does not ([`Secure_Development_Standards`](Secure_Development_Standards.md) §3) — but the worst variant, because every other instance has *someone waiting on a result*. Here the author already had their full pass and has no reason to look again. It also scales the wrong way: the more sessions working in parallel, the more often `main` moves, so the race gets harder to win exactly as throughput rises.

**Measured cost, re-derived 2026-08-02.** The first account of this instance was wrong in every load-bearing figure, and a peer caught it rather than its author — so every number below is re-derived from the Actions API and carries its unit.

*One worked instance.* #132 dispatched **nine CI runs — ten attempts — across eleven head moves** in a 3h28m open window. `main` moved **five times** underneath it. Five of the eleven head moves were rebases, four of them onto a tip that had landed 1–14 minutes earlier; **four runs were cancelled in flight** by `ci.yml`'s `cancel-in-progress` concurrency when a newer push superseded them. Cost: **212.7 min of run wall-clock**, or **442.5 min of job wall-clock** summed across matrix legs — the two differ by 2.08x, so neither is meaningful without its unit, and neither is *billable* minutes (`/timing` reports 0 billable ms on every run; these legs are self-hosted, so no billable figure exists to quote). Note what this instance is **not**: two of its attempts failed on real defects — one of them on the very head it opened at — so it is not a change that was right on the first pass and re-run anyway. The cost that generalises is the four superseded runs and the four rebase-driven re-runs — the ones that carried no new information about the change.

*The general form.* N ready PRs become N sequential suite-length cycles, because each merge invalidates every other. `main` moved **22 times on 2026-08-01**. Re-measured 2026-08-02T04:10Z: **0 of 14 open PRs were `CLEAN`** — not one could merge. Nine were green (0 failing, 0 pending) and stopped only by staleness or conflict; separately, nine of the fourteen carried armed auto-merge that could not fire. The cohort tabulated above **did not drain**: all eight are still open, and #96 has now been armed 39h at 26 commits behind. Re-read **nine hours later at 13:25Z** — 15 open, 10 armed, still **0 `CLEAN`**. The membership churns constantly; the condition has not lifted once.

*It is also a measurement cost, which is how this joins #344.* Supersession-and-re-run turns this population into one that **two different filters prune in two different ways**, and a margin read off either without naming it is not a margin. Filtering by **job** conclusion deletes rows where the job was cancelled but the *step* succeeded — the tightest rows, by construction, because a step near the cap is exactly what pushes its job past the job cap. The default **latest-attempt** view (`actions/runs/{id}/jobs`, `gh run list`) instead hides *failed earlier* attempts; it does not move a step-success maximum, but it conceals that the sample is **right-censored** — the largest step observable is the largest that *fit* under the cap, never the largest the suite wanted. Read `?filter=all` to see the censoring, and key on the **step's** own conclusion, not the job's.

*The largest cost is protocol, not throughput.* With no queue, sessions invent an ordering ritual to compensate — and the ritual is less reliable than the mechanism it replaces. Self-reported instance from the same night: a session assured a peer it would not jump the queue **while its own PR had auto-merge armed** and would have landed with nobody deciding anything. [`WORKTREES.md`](WORKTREES.md) already names that failure — *"'Don't do X' is the wrong primitive when automation already has X armed"* — and that session had read the line, about this very freeze, hours earlier. Effort spent negotiating a merge order is effort a queue spends for free, and it is where the night's stale facts and unenforceable promises came from. Wall-clock a reader can dismiss as impatience; this is not that.

**What a merge queue does not fix.** Stated level with the cost, because filing an overclaim inside the ticket about overclaiming would be its own instance:

- The overnight run of eight merges was strictly serialized (min gap 26m06s, mean 35m16s over 7 gaps) — but **serialization is not evidence that hand coordination worked.** It is entailed by `strict = true` plus a full-suite cycle, human or no human. `strict` also does not always serialize: #110 and #111 merged **24 seconds apart** on 2026-08-01, #111's squash commit parented directly on #110's. That mechanism is **not established** — only the timestamps and the parentage are.
- A queue does not shorten the suite. It reorders the same cycles; the wall-clock bounds tracked in #344 are untouched by it.
- **A queue would not report at all today.** Measured 2026-08-02: **no workflow under `.github/workflows/` carries a `merge_group:` trigger**, against 13 required contexts in branch protection — so by the mechanism step 2 describes, zero of them would report. Enabling the queue before step 2 lands would wedge every open PR. Step 2 is a precondition, not a follow-up.

**Proposed:**
1. Enable a merge queue on `main` (branch protection → *Require merge queue*), squash method to match the existing history.
2. Reconcile the required set against it: a queue runs checks on a `gh-readonly-queue/**` ref, so any workflow that must gate the queue needs a `merge_group:` trigger. Every context in [`.github/required-contexts.txt`](../.github/required-contexts.txt) lacking one will never report there — that file's own required-but-absent trap, in a new place.
3. Decide the interaction with `strict = true`. A queue makes it largely redundant; leaving both on is safe but keeps the re-sync burden for anything bypassing the queue.
4. Once landed, [`check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) goes quiet on its own. Keep it — it is the detector for this class returning.

**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that actually killed #119); #320.
**Related:** [`scripts/ci/check_stalled_prs.py`](../scripts/ci/check_stalled_prs.py) + [`.github/workflows/stalled-prs.yml`](../.github/workflows/stalled-prs.yml) (built alongside this filing — it reports the condition, it does not remove it); [`.github/required-contexts.txt`](../.github/required-contexts.txt); [`scripts/ci/check_required_workflow_state.py`](../scripts/ci/check_required_workflow_state.py) (sibling: "can this context ever report?" to this one's "can this PR ever merge?"); #344 (the wall-clock bounds that killed one of #119's runs, and the sibling half of the measurement cost above); #320.

**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence.
**Source:** stuck-CI triage, 2026-08-01. Measured live against `MEFORORG/MessageFoundry` branch protection and the open-PR set that date; independently reached by three parallel sessions from separate evidence. Cost evidence and both corrections above re-derived from the Actions API on 2026-08-02 under adversarial verification, after the first hand account of #132 proved wrong in every load-bearing figure.

---

Expand Down
Loading