[audit-workflows] 🚨 Fleet-wide ~7h hang incident + daily audit (2026-08-06): 43 workflows, 5 engines affected

[github-actions[bot]]

🚨 Headline: ~7-hour fleet-wide hang/reap incident (14:41–21:47 UTC)

The single biggest story in today's window: 68 of 91 failures (75%) across 43 distinct workflows and 5 engines took 15 minutes to 4.8 hours to fail — instead of the usual few minutes — between 2026-08-06 14:41:43Z and 21:47:20Z (~7h1m). Several successful runs in the same window also stretched to 1–4.8h. This burned 9,645 action-minutes and 1,075.1 AI-credits (63% of the day's total AIC) for near-zero output.

Three independent lines of evidence point to one shared infra event rather than many coincidental per-workflow regressions:

1. Synchronized end-times — unrelated workflows on unrelated triggers finished within the same 60–90s buckets (e.g. multiple distinct workflows all ending 19:19:25–26Z, others at 20:55:58–59Z) — the strongest signal, consistent with a shared watchdog/reap sweep rather than independent timeouts.
2. Cross-engine, cross-workflow scope — Copilot, Claude, Pi, and Crush engines all affected simultaneously, plus 48 runs where engine classification itself failed (a symptom, not a cause).
3. Sharp before/after boundary — run durations snap back to normal (minutes) immediately outside the window.

This reframes most of today's "top failing workflows" as symptoms of one incident, not independent bugs: PR Code Quality Reviewer (11 fails), Design Decision Gate 🏗️ (7), PR Sous Chef (5), Issue Monster (5), Auto-Triage Issues (5), Impeccable Skills Reviewer (4), Avenger (4).

Recommendation (P0/new): investigate GitHub Actions runner-pool / MCP-gateway / shared-infra health for this window — check platform status history and any shared MCP-gateway/proxy logs for a stall or leak starting ~14:41Z — and add a per-job hard timeout well below the ~5h worst case so a recurrence fails fast instead of burning hours of compute.

Fleet health this cycle

Metric	Raw	Adjusted (excl. intentional-fail test workflows)
Runs	378	377
Success	287	287
Failure	91	90
Success rate	75.9%	76.1%

prod-main (main-branch) success rate: 84.8% (162/191). Window: partial ~19.8h (03:29–23:17 UTC) — note a 31-day gap since the prior audit cycle (2026-07-06), so trend comparisons below span that gap.

Of the 91 raw failures, 68 (75%) are the fleet-wide hang incident above; the remaining 23 are baseline (non-incident) failures, almost all matching the pre-existing chronic copilot-sdk-driver-failures signature (see below).

Chronic issue status (click to expand)

• copilot-sdk-driver-failures (chronic since 2026-06-02, still open 2+ months later): the long-run "agent does real work then Execute-CLI/session-idle fails 0-turn" signature remains the dominant baseline fail cause outside today's incident — seen in PR Code Quality Reviewer, Impeccable Skills Reviewer, Avenger, and Auto-Triage Issues, now with real token usage (20K–73K tokens) confirming the long-run variant rather than the original instant 0-token variant.
• avenger-err-config-no-structured-logs: Avenger ran clean 4/4 before today's incident window, then failed only during/adjacent to it (with hang-length durations, not the classic err-config signature) — inconclusive whether the underlying root cause is actually fixed or simply didn't get a chance to fire today.
• smoke-ci-copilot-cli-100pct-fail-on-push: zero push-triggered runs occurred in today's sample — no data either way this cycle; left open/unconfirmed rather than marked resolved.
• pi-gpt54-0tok-agentjob-fail (RECOVERED-WATCH): Auto-Triage Issues failed 5×, but 4 of 5 fall inside/adjacent to the hang window; the 1 baseline fail had non-zero tokens (16,592), unlike the original 0-tok signature — low-confidence, not a confirmed relapse.
• NEW watch — Design Decision Gate 🏗️ baseline risky-fail: this workflow had 32 runs / 0 fails on 2026-07-06. Today it had 7 fails total; 3 fall inside the incident window, but 4 are standalone baseline failures (7.5m, 7.5m, 11.3m/15,001 tok, 12.3m/15,819 tok) independent of the incident — a genuinely new signal not present a month ago, flagged for next-cycle watch.

Trend charts

Workflow Health — daily success/failure counts and success rate:

Success rate dropped from 89.8% (07-06) to 75.9% today, and failure count jumped to 91 — but this is almost entirely attributable to the single hang incident, not a broad reliability regression. Strip out the incident's 68 failures and the day's underlying baseline rate is consistent with the prior chronic-issue trend (~92-95%, gated mainly by the still-unresolved copilot-sdk-driver-failures family).

Token Usage — daily AI-credits (AIC) with 7-day moving average:

AIC roughly halved from the 07-06 peak (29,611 → 15,076.6), but 1,075.1 of that (7%) was burned purely by the hang incident's stalled/retried runs with no useful output — the true "useful work" cost is even lower than the raw number suggests.

Full baseline (non-incident) failing-workflow breakdown

Workflow	Total fails today	In-incident	Baseline (non-incident)
PR Code Quality Reviewer	11	~9	~2
Design Decision Gate 🏗️	7	3	4 (NEW watch)
PR Sous Chef	5	4	1
Issue Monster	5	4	1
Auto-Triage Issues	5	4	1
Impeccable Skills Reviewer	4	3	1
Avenger	4	3	1
Test Quality Sentinel	2	1	1
Smoke Copilot	2	0	2
Matt Pocock Skills Reviewer	2	1	1
Agent Persona Explorer	2	2	0

Engine breakdown (378 runs): GitHub Copilot CLI 203 (176✓/27✗), Claude Code 64 (57✓/7✗), Pi 47 (40✓/7✗), Codex 12 (12✓/0✗), Crush 2 (1✓/1✗), Goose 2 (1✓/1✗), plus 48 runs where engine classification failed (all failures, all inside the incident window).

Recommendations (deduplicated, most recent first)

1. [P0-NEW] Investigate the fleet-wide hang/reap incident — see headline above. Highest-ROI action this cycle by far.
2. [Chronic, 2+ months open] Root-cause copilot-sdk-driver-failures: instrument the Execute-CLI/session-idle path to emit stderr + exit code so the "does real work then fails 0-turn" signature stops being opaque. Affects PR Code Quality Reviewer, Impeccable Skills Reviewer, Avenger, Auto-Triage Issues as baseline failures independent of today's incident.
3. [Watch, new] Re-check Design Decision Gate 🏗️ next cycle — 4 baseline fails today vs 0/32 a month ago.
4. [Low priority, ongoing] Fleet token-usage reporting gap (metrics.TokenUsage null) persists; today's 14.0M total tokens came through cleanly via token_usage_summary, so this may already be improving — worth confirming next cycle.

Repo memory (known-issues.json, anomalies.json, recommendations.json, workflow-trends.json, metrics-summary.json, audit-history.jsonl) has been updated and pushed with all of the above, preserving prior history and recurrence counters.

References:

• §31121507958 — PR Code Quality Reviewer baseline failure
• §31124209624 — Design Decision Gate 🏗️ long-hang failure (incident window)
• §31126468057 — Avenger failure adjacent to incident window

Generated by 🔍 Agentic Workflow Audit Agent · agent · 344.6 AIC · ⊞ 8.4K · ◷

• expires on Aug 7, 2026, 3:59 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #51213.