[storify] Daily Storify — 2026-08-29: Avenger's fourth crash, a cascade detector's first real test

[github-actions[bot]]

The last 24 hours across github/gh-aw's workflow fleet read like a system quietly correcting itself: a fleet-health baseline showed measurable improvement even as the same handful of chronic problems — a crash-prone Avenger, a metrics job stuck on a memory-cap treadmill, and a mislabeled Windows runner — kept resurfacing in new shapes. Meanwhile a burst of ten simultaneous smoke-test failures around midnight triggered the automated cascade-detector for the first time this window, correctly flagging correlated infra noise instead of ten separate incidents.

Episode Highlights

Avenger's third distinct crash, still unrepaired. Avenger (self-repair workflow) failed four separate times in the window — §33170291463, §33190548025, §33211194205, §33215860939, §33216999126 — all with the identical signature: "the claude engine terminated before producing output." GitHub API core-quota consumption climbed steeply run over run (+292% → +724% vs. baseline) while token usage stayed at zero, suggesting the engine dies before doing any real work, only after burning API calls. Four filed issues (#56694, #56728, #56737, plus earlier ones) each read as a fresh transient-infra guess, but the pattern is now old enough (yesterday: symlink bind-mount fix; today: a fourth crash) that it looks structural rather than transient.

Code Scanning Fixer hits its tool-denial guardrail three times. Runs §33192281598, §33214165512, §33225306296, and §33239595658 all stopped early at "Excessive Tool Denials (3/3)" — denied git checkout/git status calls — while duration ballooned from 5m34s to ~25m and cache-read tokens exploded from 0 to 600K+. The denial pattern plus the runaway duration suggests the agent is retrying the same blocked git operation in a loop rather than failing fast.

Metrics Collector's cap treadmill produces its worst run yet. §33229192909 burned 1.5M tokens (rebuild factor 25x) and still failed at push_repo_memory, plus newly blocked domains (ab.chatgpt.com, api.github.com, chatgpt.com). This is the same root cause flagged yesterday (#56537: daily snapshot outgrowing the 100KB cap) — now compounding with a token-usage blowout, reinforcing that a cap bump alone won't fix it (issue #56815).

Windows Runner: closed not_planned, failed again next day. #56502 was closed not_planned on 2026-08-28, and the identical Setup Scripts failure recurred the very next cycle — independently caught by both the Agent Job Health Monitor (discussion #56744) and Deep Report (#56848). A same-day PR (#56667, merged) added a new windows-grower.md workflow to probe and auto-file fixes for this exact failure, suggesting the team pivoted from "close as not planned" to "build a dedicated self-repair loop" within 24 hours.

Cascade detector fires for the first time on a 10-issue smoke-test burst. At 00:27–01:27 UTC, ten [aw] * failed issues landed within 60 minutes (Smoke Drive, Smoke Gemini, Smoke Cursor, Smoke GitHub Claude, Smoke Codex Sub Agents, and others) — the automated cascade rollup (#56767) correctly grouped them as a likely common-cause cluster rather than letting them scatter as isolated triage items, applying cascade-suspected labels for batch handling.

Feedback Loops Across Workflows

• avenger-self-repair (escalating): incremental fix → new distinct crash symptom, four days running now. Direction: stalled, not improving — the symlink fix from yesterday didn't touch today's claude-engine-exit-early symptom.
• memory-cap-treadmill (chronic): Metrics Collector: push_repo_memory fails 6 consecutive runs — daily JSON exceeds 100KB file-size cap #56537's 6th failure yesterday is joined today by a 1.5M-token blowout on the same job. Direction: worsening — the cost dimension is now compounding the size dimension.
• code-scanning-fixer-tool-denial-stall (new this cycle, related to prior "silent-stall" loop): three runs in a row hit the exact same 3/3 denial guardrail on git commands, with duration scaling ~5x. Direction: new pattern, needs investigation before it's assumed transient.
• windows-runner-close-reopen (new): a not_planned closure was immediately falsified by recurrence, but the response was a new dedicated self-repair workflow (Add daily windows-grower workflow that probes and fixes Windows runner support #56667) rather than a repeat close. Direction: improving process, though the underlying Setup Scripts bug is still unfixed.
• cascade-detection-maturing: the automated cascade rollup ([aw] Failure cascade detected #56767) worked as designed on its first real 10-issue burst this window — a structural win for noise reduction. Direction: improving.
• fleet-health-net-positive: per the Agent Job Health Monitor (discussion [agent-job-health] Agent Job Health Monitor - 2026-08-28: fleet improved, 1 novel Windows Runner failure, 3 schedule blind spots #56744), run-weighted failure rate dropped 9.22% → 7.41% and workflow-weighted mean dropped 16.8% → 8.84% vs. the 2026-08-23 baseline, even with the above chronic issues persisting. Direction: improving overall, masking localized regressions.

Human Interventions That Mattered

• pelikhan merged Add daily windows-grower workflow that probes and fixes Windows runner support #56667 (Windows-grower workflow) same-day as the Windows failure recurrence — turning a not_planned closure into an active self-repair mechanism instead of repeating the dismissal.
• Copilot's Avoid CodeQL false positive in manifest path validation #56690 fixed a CodeQL false positive in manifest path validation, closing out the UK AI Resilience Tier-B alert flagged the prior day.
• Deep Report's intel-agent filed a structured 8-issue batch ([deep-report] Windows Runner Integration Test: 100% failure at Setup Scripts step recurred after #56502 closed not_planned #56848–[deep-report] metrics-glossary.md says Daily Firewall Report covers last 7 days, but workflow actually uses 24h window #56855) rather than one noisy report, each with acceptance criteria and cited discussion evidence — a template that's kept documentation drift (e.g., metrics-glossary 7-day vs 24h mismatch) actionable instead of buried in a wall of text.
• Safe Output Health Monitor's [deep-report] Improve safe-outputs failure classification: policy-decline vs hard failure, and add_comment error detail #56576 (policy-decline miscounting) remains open with a proposed structural fix — a second occurrence of the same E099 miscategorization bug from two days ago, still awaiting an engineering decision rather than another workaround.

Signals to Watch Next

• Does Avenger's next run surface a fourth distinct crash mode, confirming this needs an engine-level fix rather than case-by-case patches?
• Does the new windows-grower.md workflow actually diagnose and fix Setup Scripts, or just re-file the same issue on a schedule?
• Does Metrics Collector get a growth-control fix before its next scheduled run, given today's 1.5M-token/still-failing result?
• Does Code Scanning Fixer's tool-denial loop recur a 4th time, or was #33239595658 the last occurrence before a guardrail fix lands?

References:

• §33229192909 — Metrics Collector worst-run-yet
• §33216999126 — Avenger's latest claude-engine crash
• §33239595658 — Code Scanning Fixer's 4th tool-denial stop

Generated by 📚 Daily Storify · copilot · auto · 86.2 AIC · ⌖ 6.83 AIC · ⊞ 10.2K · ◷

• expires on Aug 31, 2026, 12:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Daily Storify.

A newer discussion is available at Discussion #57112.