Restore release-pipeline cleanup health (issue #2952)

The cleanup-release-fragments release job no longer kills itself mid-wait on every
release: its timeout-minutes now covers the sum of its own dispatched-run wait budgets
(with round-trip overhead and margin), the ci.yml wait's poll budget now outlasts the
dispatched run it waits for, and a new static lint keeps the invariant enforced on every
workflow-touching PR.

What changed

Wait budgets scaled to their targets (.github/workflows/release-and-publish.yml):
wait_for_dispatched_run takes per-call poll parameters (fn <workflow> <iterations> <sleep_seconds>, defaults preserved at 120×5 s). The ci.yml wait now polls up to
30 min (360×5 s — the dispatched ci.yml run observed 16m05s against the old 10-minute
poll budget), and the pr-standards/drift-check waits stay at 10 min. Each call site
declares its realistic duration via a # target-duration: <minutes> comment.
Job timeout covers the sum: cleanup-release-fragments timeout-minutes 10 → 70.
The old value sat below every wait's effective cost once the observed per-iteration
gh run list round-trip overhead is counted (600 s nominal vs ~660-720 s effective
per wait), and 3-3.6× below the 50-minute sum (3,000 s declared → 3,900 s required at
the 1.2× overhead factor plus margin); every release run was cancelled at ~10 minutes
mid-wait, release-health-summary went red, and each run stranded a cleanup PR
awaiting manual rescue.
New static lint scripts/check-workflow-timeouts.ts (invoked as
bun run scripts/check-workflow-timeouts.ts, the exact form CI runs): for every job
that polls via a seq×sleep wait loop, asserts
timeout-minutes ≥ ceil(Σ wait budgets × 1.2) + 5 min and every wait budget ≥ its
declared # target-duration. Fail-closed throughout: parameterized helpers with
${N:-default} defaults resolve; unresolvable defaults, expression-valued timeouts,
inline (non-function) poll loops, oversized files, and unattributable constructs are
all findings, never silent passes — and the same holds for parser ambiguity
(unparsable job regions, unterminated wait functions, line-continued loops, empty
scans, and findings overflow all fail closed). A wait call site without a
# target-duration directive is a violation (deliberate strengthening: every wait
must declare its target).
CI wiring: the lint runs in the drift job of drift-check.yml (pull_request,
push, merge_group, workflow_dispatch) via direct script invocation — the same
pattern as the drift check step beside it. A package.json script alias is not
added: the release-owner guard (#2677) forbids non-release PRs from touching
package.json.
Tests (tests/unit/scripts/check-workflow-timeouts.test.ts plus
tests/unit/scripts/check-workflow-timeouts-robustness.test.ts): 34 tests covering
the pre-fix red shape, the fixed green shape, the exact shipped parameterized helper
(anti-vacuousness: the lint demonstrably sees the workflow it ships with, and a
dropped-directive mutation reds), all fail-closed findings (including non-literal
wait-call arguments), partial parameterization, parser edges, the parser-hardening
regressions (commented jobs: keys, alternate job indentation, commented or missing
function braces, line-continued loops, comment-shadowed budgets, empty scans,
truncated output), and a ratchet over the repository's real workflows.

User-visible effect

On the next release after this lands, cleanup-release-fragments should conclude
success and release-health-summary should be green instead of failing with
"cleanup-release-fragments concluded cancelled"; the docs/releases archive materializes
automatically and the per-release manual rescue of chore/release-fragments-* PRs ends.

2026-10 flaky-test cohort: coverage drift posture, Windows teardown EBUSY, macOS probe re-quarantine, detector dedupe (issue #3248)

Issue: #3248

What

Resolves the open "Auto-detected flaky tests (merge-group) — review for
quarantine" issue backlog: 84 issues in the triage cohort (2026-09-09..
2026-10-14) plus 2 more auto-filed while this change was in flight (86 at
publication; the detector dedupe below stops further duplicates). Triage
(per-run check-run annotations + validation artifacts from 16 sampled failed
merge-group runs) found three independent root-cause classes behind the
detections, plus one workflow defect that turned them into one issue per
failed run, plus two policy quarantine decisions and one production
follow-on the evidence demanded:

Coverage-shard checkout-drift enforcement (the retry-flake wave). The
checkout-drift tripwire (#3146, 2026-10-08/09) enforces "creating .swarm/
where none existed". The unit job pre-writes .swarm/repository-validation/
manifests before tests; scripts/ci/run-coverage-gate.sh did not, so on
every fresh coverage shard the first file whose execution materialized
.swarm/ (writers tracked in #3165-A) failed attempt 1 and passed the
in-job retry — ubuntu/coverage-only retry-flakes riding along on every
failed run (cli-version/commands/run-command: 22 detections each;
delegation-gate-scope/guardrails-modified-files: 16 each). The gate now
pre-creates .swarm/coverage-gate/ with the shard file list (unit-job
posture alignment; coverage-gate joins IGNORED_SWARM_PREFIXES). This is
an interim alignment: #3165-A retires the writers and then flips the
.swarm diff from warn to enforce, restoring creation detection — until
then the coverage job cannot detect a test creating .swarm/ where none
existed (the same trade-off the unit job already accepts).
Windows teardown EBUSY. safeRmRecursive
(tests/helpers/safe-test-dir.ts) retried EBUSY/EPERM/ENOTEMPTY on a flat
~2 s budget with GC only on the first attempt; windows-latest AV/cold-FS
handle holds outlived it and hard-failed merge-group shards
(close-active-state-unlink-retry: 21 detections, hard fail run
38507292215). The budget is now a ~10 s deadline (100 attempts) with
collectGarbageBestEffort() on every failed attempt (a hedge for
GC-reclaimable handles; only elapsed time helps AV-held ones). The
19-detection hydration-populate-guard flake was its raw-rmSync afterAll
with no retry at all (hard fail run 38425263922, 3/3 attempts) — it now
routes through safeRmRecursive (the PR #2190 / #2807 pattern). A
recurrence sweep across all 77 flagged files found 27 further raw
recursive-rmSync cleanup sites; they are dispositioned (not converted) in
the tracker and deferred to follow-up (#3252).
Production follow-on: /swarm close's clean stage had the same
too-short budget — ACTIVE_STATE_UNLINK_RETRY_DELAYS_MS grew from
[25, 50, 100, 200] (~375 ms) to a ~15.2 s schedule, with GC on every
failed attempt. Local instrumented Windows runs show the Bun sqlite
handle release after closeProjectDb is bimodal (~5 ms or ~11 s); on the
long tail the old schedule threw EBUSY from
unlinkActiveStateFileWithRetry (observed ~3/10 local runs). The success
path is unchanged — the extra budget is spent only where the old schedule
produced an error. Worst case, a stuck artifact now delays /swarm close's clean stage by up to ~15 s instead of failing it.
macOS native-probe race (re-quarantine). bun-compat-exit-first-2530
(10 detections; hard fails on merge-group runs 38449931439 + 38421990284,
both unit (macos-latest, 5)): the overflow child exits 0 before the kill
lands — a different signature than the timeout flake the #2973 retirement
cured, so the retirement absence-guard's "fresh merge-group macos-latest
failure evidence" condition is met and the entry is re-added to the macOS
ledger with OWNER/EXPIRY (retire on a nativeOutputLimitProbe root fix or
three consecutive green macOS merge-group runs). Pin tests
ci-yml-quarantine-2730/-2740 were reworked from retired→active pins.
Windows retry-flake quarantine. dispatch-lanes-pr-review-resilience-
followups (9 detections, windows-latest retry-only evidence, no assertion
text survives the retry loop) joins the windows ledger with OWNER/EXPIRY
per the delegation-gate-background-coder retry-only precedent; the ledger's
STATUS header now declares 3 active entries.
Detector dedupe (stops the flood). detect-and-quarantine-flakes.sh
created a new issue per failed merge-group run. It now searches for an
OPEN issue with the exact title that is bot-authored (the structural
author.is_bot flag — the gh CLI and the REST API render the Actions
actor with different login strings, so the flag is the only portable
identity; the v7.188.0 lookalike-issue guard; server-side
--search in:title, gh --template extraction, no jq) and comments the
full per-run candidate
list plus the triggering run URL there. A new issue is created when no
bot-authored match is open — and, with a ::warning::, when the lookup
fails or the only same-titled issues are not bot-authored. Concurrent
detections can still race one duplicate (accepted, warn-only).

All 86 open detection issues (84 cohort + 2 in-flight filings) are
dispositioned in the tracker with per-file
evidence (root-fixed-by / quarantined-by / already-ledgered / stale).

Why

Failed merge-group runs were being filed as duplicate issues continuously
(22 runs flagged the coverage cohort alone during 2026-10-09..14, out of 34
failed merge-group runs in those five days), and the two Windows/macOS
hard-failers were repeatedly ejecting PR groups from the queue. The classes
had concrete, evidenced root causes; quarantining the coverage-shard cohort
(5+ files including two core hooks suites) would have suppressed them
everywhere for an infrastructure bug.

Migration

No config or API changes. Behavior note: /swarm close's clean stage can now
spend up to ~15.2 s retrying a handle-locked artifact (previously it errored
after ~375 ms); the success path is unchanged. The quarantine entries carry
Check 7 OWNER/EXPIRY metadata (2026-11-15) with retirement criteria tracked
in #3248 (discharged by #3251 for the macOS entry); quarantined macOS files
are skipped only on macos-latest runners and the windows entry only on
windows-latest. Follow-ups: #3250 (coverage lcov aggregator failures),
#3251 (native-probe root fix), #3252 (windows single-hit diagnoses + the
27-site raw-rmSync hardening worklist), #3165-A (the .swarm-writer
cleanup that eventually restores the drift guard's full enforcement).

Navigation Menu

Repository navigation

v7.197.2

Choose a tag to compare

Sorry, something went wrong.