Skip to content

ci: run the backlog status invariant ungated, and correct #320 to the measurement - #128

Open
wshallwshall wants to merge 1 commit into
mainfrom
ci-ungate-backlog-status
Open

ci: run the backlog status invariant ungated, and correct #320 to the measurement#128
wshallwshall wants to merge 1 commit into
mainfrom
ci-ungate-backlog-status

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Two fixes from the same investigation, both correcting my own work.

1. The guard gap that red main

Two guards police docs/BACKLOG.md. The number-space gate is deliberately ungated, with a comment saying gating it "would skip it on exactly the pull requests it exists to police". The status invariant was reachable only through pytest, which is gated on code == 'true' — so on a BACKLOG-only PR, the exact shape it exists to check, it did not run at all.

On 2026-08-01 that let #117 merge with a #320 banner using an emoji the invariant rejects. Docs-only, so the leg went green in seconds without compiling the suite. main was then red for every session in the repo until another session's PR tripped over it. A 0.6s markdown parse would have caught it.

One hardened guard and one unhardened guard over the same file is worse than neither — the hardened one makes it reasonable to assume BACKLOG changes are covered.

The fix turned out to be clean. The invariant already lives in a standalone stdlib script, scripts/docs/backlog_status_check.py — only argparse/re/sys/pathlib, with its own CLI. So the neighbouring step's advertised property ("git + stdlib python (no install), costs seconds") is preserved exactly. And tests/test_backlog_status_check.py imports that same module, so there is one implementation, the unit tests keep covering its edge cases, and this only adds the always-on invocation.

2. #320's diagnosis was wrong

It claimed windows-2025 "cannot service 60/s" and is "~10x slower". A 36-run sweep across all three hosted SKUs (run 30705885914, 3 repeats per cell) refutes both.

Stranded fraction:

offered ubuntu windows-2022 windows-2025
60/s 0, 0, 0 0, 0, 0 0, 0, 0
150/s 0, 0, 0 0, 0, 0 0, 0, 0
300/s 0, 0, 0 0, 0, 0 0, 0, 0
600/s 0, 0, 0 4.6 / 8.9 / 19.2% 25.4 / 30.7 / 31.1%

Wall time — the cleaner signal, since all three still ingest everything up to 300/s:

offered ubuntu win-2022 win-2025 2025 vs ubuntu
60/s 2.8s 4.7s 4.9s 1.8x
150/s 4.3s 10.1s 11.9s 2.8x
300/s 8.4s 22.0s 29.8s 3.5x

What holds: a consistent ubuntu > win-2022 > win-2025 ordering, and ~3× more stranding than its sibling at saturation. It is genuinely the slowest leg.

What is refuted: the gap is a latency gap of 1.8×–3.5×, not a capacity cliff. windows-2025 ingests everything up to 300/s — five times the rate the failing test offers — and strands 0% at 60/s in three of three runs.

So the 60/s failures remain unexplained. Real (byte-identical twice) but not a property of the SKU at that rate, or the sweep would show them. The untested variable is contention: the probe runs the engine alone on an idle runner, while the CI failure happens with the full suite alongside it in a ~20-minute job. That also matches the developer-box outlier already retracted in this item.

Consequence. This is a tolerance problem, not a throughput one — exactly what #115's widened stranding budget absorbs, so that fix is better supported now than when it was made. And the product concern this item originally raised is not supported: a 1.8×–3.5× gap on a hosted VM image says nothing about Windows Server 2025 as a deployment target.

Verification

  • scripts/docs/backlog_status_check.py exits 0 — 244 items, each declaring exactly one status.
  • New step asserted present, ungated (runner.os == 'Linux' only, no code condition).
  • 122 backlog / required-context / workflow guard tests pass.
  • Cherry-picked onto current main (2a6649fb) rather than committed against a stale base — another session had edited ci.yml meanwhile; auto-merged cleanly and re-verified after.

… measurement

Two fixes from the same investigation, both correcting my own work.

1. THE GUARD GAP THAT RED main.

Two guards police docs/BACKLOG.md. The number-space gate is DELIBERATELY ungated,
with a comment saying gating it "would skip it on exactly the pull requests it
exists to police". The STATUS invariant was reachable only through pytest, which
IS gated on `code == 'true'` -- so on a BACKLOG-only PR, the exact shape it exists
to check, it did not run at all.

On 2026-08-01 that let #117 merge with a #320 banner using an emoji the invariant
rejects. Docs-only, so the leg went green in seconds without compiling the suite;
main was then red for every session until another session's PR tripped over it. A
0.6s markdown parse would have caught it. One hardened and one unhardened guard
over the same file is worse than neither: the hardened one makes it reasonable to
assume BACKLOG changes are covered.

Adds an ungated step running scripts/docs/backlog_status_check.py. The step above
advertises "git + stdlib python (no install), costs seconds" and that property is
preserved exactly -- the script imports only argparse/re/sys/pathlib and has its
own CLI. tests/test_backlog_status_check.py imports the SAME module, so there is
one implementation, the unit tests keep covering the edge cases, and this only
adds the always-on invocation.

2. #320's DIAGNOSIS WAS WRONG.

It claimed windows-2025 "cannot service 60/s" and is "~10x slower". A 36-run sweep
across all three hosted SKUs (run 30705885914, 3 repeats per cell) refutes both:

  stranded    ubuntu    win-2022              win-2025
  60/s        0,0,0     0,0,0                 0,0,0
  150/s       0,0,0     0,0,0                 0,0,0
  300/s       0,0,0     0,0,0                 0,0,0
  600/s       0,0,0     4.6/8.9/19.2%         25.4/30.7/31.1%

  wall        ubuntu    win-2022   win-2025   2025 vs ubuntu
  60/s        2.8s      4.7s       4.9s       1.8x
  150/s       4.3s      10.1s      11.9s      2.8x
  300/s       8.4s      22.0s      29.8s      3.5x

What holds: a consistent ubuntu > win-2022 > win-2025 ordering, and ~3x more
stranding than its sibling at saturation. It is genuinely the slowest leg.

What is refuted: the gap is a LATENCY gap of 1.8x-3.5x, not a capacity cliff.
windows-2025 ingests everything up to 300/s -- five times the rate the failing
test offers -- and strands 0% at 60/s in three of three runs.

So the 60/s failures are still unexplained: real, byte-identical twice, but not a
property of the SKU at that rate or the sweep would show them. The untested
variable is CONTENTION -- the probe runs the engine alone on an idle runner, while
the CI failure happens with the full suite alongside it in a ~20-minute job. That
also matches the developer-box outlier already retracted in this item.

Consequence: this is a TOLERANCE problem, not a throughput one, which is what
#115's widened stranding budget already absorbs -- that fix is better supported
now than when it was made. And the product concern this item originally raised is
not supported: a 1.8x-3.5x gap on a hosted VM image says nothing about Windows
Server 2025 as a deployment target.

Verified: backlog_status_check exits 0 (244 items); the new step parses and is
ungated (`runner.os == 'Linux'` only, no `code` condition); 122 backlog /
required-context / workflow guard tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@wshallwshall
wshallwshall enabled auto-merge (squash) August 1, 2026 16:13
wshallwshall added a commit that referenced this pull request Aug 2, 2026
…s that can never merge (#131)

* ci: report pull requests that are green, armed, and can never merge

Measured on this repo 2026-08-01: nine open pull requests with zero failing
checks and zero pending checks, not one of which could merge. Six had
auto-merge ARMED, which will never fire. #74 had been in that state since
2026-07-30 and was found only because somebody went hunting for "stuck CI"
by hand.

The mechanism is that `strict = true` plus no merge queue plus a ~20-minute
suite makes merging a race: a PR is mergeable only between going green and
the next thing landing on main. Losing that race is silent. Armed auto-merge
does NOT update a BEHIND branch -- it waits on checks that already passed --
so the PR sits with no failing check, no notification, and no run in flight.

No existing signal can see it, because every existing signal is a check
OUTCOME and nothing has failed. nightly-notice.yml watches CI runs and there
is no failing run to watch; the author's last signal was a full pass. A green
dashboard and a wedged repository are indistinguishable unless something asks
"can this still merge at all?".

This does not fix the race -- only a merge queue does, filed separately as
BACKLOG #340. It converts a SILENT failure into a LOUD one, which is the part
that let #74 sit for days.

Scheduled rather than per-PR: the stall arrives when a DIFFERENT pull request
merges, so the affected PR has no run to hang a check on. Advisory by
placement and must never become required -- it reports on OTHER pull requests,
so a stall on #71 would block #128, wedging the repo with the tool meant to
unwedge it.

Verified against the live repo: 14 scanned, 8 stalled, 6 armed, exit 1. The
count differs from the hand survey's 9 because #120 was re-synced in between,
which the check correctly excluded. Tests carry a positive control (the exact
stall shape MUST be detected) alongside negative controls for failing,
pending, BLOCKED, DIRTY and closed PRs, and assert that an unclassifiable
rollup node counts as unsettled rather than green.

* ci: raise the Windows step cap, which had 1.06x margin while claiming 2x

PR #119 was killed at 26:07 against ci.yml's 26:00 `step_timeout` with ZERO
tests failing. What moved was the suite, not the code under test: #74 landed
tests/test_worktree_prune_merged.py (1,506 lines) and windows-2025 went
19:35 -> 26:07 on the same branch.

The comment beside the cap said the Windows legs were "unchanged because 26
min against the same suite is still ~2x headroom". Measured over the 11
PASSING windows-2025 runs on 2026-08-01:

  leg              max passing step   old cap   old margin
  ubuntu-latest    12:27              19:00     1.53x
  windows-2022     18:39              26:00     1.39x
  windows-2025     24:35              26:00     1.06x

windows-2025 had already PASSED at 24:35 -- 85 seconds of margin -- before
#119 died. The "2x" figure matched no leg when it was written. The same file
records this exact failure happening on the ubuntu leg on 2026-07-31 (775s
green against a 780s cap) and concludes a watchdog that cannot separate
"deadlocked" from "slow today" becomes a coin flip; ubuntu's budget was
raised then and Windows was left alone on the false claim.

Raised to step_timeout 36 (1.46x over the 24:35 max) and job_timeout 40,
preserving the nesting invariant that the step must expire strictly before
the job. Both Windows legs take the same number: windows-2022 is faster, so
sizing on windows-2025 only leaves it more room. The replacement comment
states the measured value and its date rather than a multiple -- a bare
multiple is what let this rot undetected.

Timing note recorded in the comment because it cost two sessions an error
during triage: step_timeout gates the STEP, not the job. c53f752's JOB ran
28:41 and PASSED, against job cap 30 / step cap 26.

Also files BACKLOG #340 (enable a merge queue -- 9 green PRs could not merge,
6 armed and never firing) and #344 (fixed wall-clock bounds as a class, with
this cap as instance 1 and test_stage_dispatcher.py's hardcoded 8.0s poll
budget against an injected ManualClock as instance 2).

COORDINATION: ci.yml and docs/BACKLOG.md were each held by another live
session, and the collision gate (scripts/hooks/collision_gate.ps1) refuses an
Edit while a live session's BRANCH carries a diff to the file -- it cannot
represent "coordinated, verified disjoint". Both counterparties gave explicit
written consent before these edits were applied outside the Edit tool:
zizmor-1280-adoption ("You land it. I'm standing down on ci.yml timeouts",
its only hunk being a one-line pin comment ~160 lines away) and
ha-construct-pickle-sandbox ("Go ahead with #340 now -- append after #338
exactly as you planned. I'll absorb the conflict"). No hooks were skipped;
this commit ran the full pre-commit suite.

* ci: record that #119's leg passed on a re-run at the same cap

The ADR-0154 session re-ran #119's windows-2025 leg on the SAME commit
against the SAME 26:00 cap. Attempt 1 was killed at the cap; attempt 2
concluded success. Same code, same config, same ceiling, two outcomes.

This closes the one gap in the case for raising the cap: it rules out "that
PR's tests are just slow". The leg was not failing, it was coin-flipping
against the ceiling -- which is precisely the state the ubuntu note above
already names, now demonstrated rather than argued.

It also disposes of "re-run it and see" as a diagnosis. A green re-run at
26:00 does not show the suite fits; it shows that runner was fast enough
that time. Recorded in the comment so the next person reaching for a retry
knows what a green retry does and does not prove.

Evidence contributed by the session holding #119, which is deliberately
holding its branch update until this lands so it re-rolls under the raised
cap rather than spending another coin flip at 26.

Merged main (8f01cef, #120) in the same push: #131 had gone BEHIND, which
is the stall this PR exists to report -- the fix for the cap has to survive
the cap, and the fix for silent stalls can itself stall silently.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant