[aw-failures] [aw] Failure Investigator Report — 2026-08-06 (6h)

[github-actions]

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Fix the two production workflows still running the removed opencode engine first — they fail 100% of scheduled runs, and it's the only P0 with zero tracking coverage.

5 distinct failure clusters analyzed from this window's failed runs; 2 already covered by in-flight/tracked fixes, 1 confirmed recurrence of a tracked issue, 1 new P0 self-inflicted regression, 1 new P1 hardening gap folded into this report (create_issue budget: 2/run).

Cluster summary

Sev	Cluster	Runs	Status
P0	engine: opencode removed from the runtime, but Daily Reliability Review + Daily Security Red Team still declare it — fails every scheduled run	§31055598155, §31057366689	Untracked — sub-issue filed below
P1	PR Code Quality Reviewer: grumpy-coder sub-agent hits "No model available", exhausts 4 retries	§31054795358	Tracked — recurrence of #49022, not yet fixed
P1	Detection Analysis Report: 20-min job timeout right after self-recovering from a matplotlib ModuleNotFoundError mid-run	§31056417218	Tracked — fix already in flight, PR #50716 (timeout-minutes: 45 + single-log-fetch prompt), not yet merged
P1	Avenger Install Make step fails outright when Microsoft's azure-cli apt mirror 403s, even though make ships preinstalled on the runner — same fragile step exists verbatim in hourly-ci-cleaner.md	§31061056320	Untracked, no separate issue this cycle (budget) — see below

Evidence

opencode engine removal breaks 2 in-repo workflows (P0, untracked)

• CHANGELOG.md confirms an intentional, already-shipped breaking change: "Removed the experimental opencode engine... Workflows using engine: opencode must migrate to copilot, claude, codex, gemini, antigravity, or pi."
• .github/workflows/daily-security-red-team.md (line 20: id: opencode) and .github/workflows/daily-reliability-review.md (line 20: id: opencode) were never migrated.
• Both compiled .lock.yml files fail identically at "Verify OpenCode CLI installation" across all 3 jobs (agent, detection, evals) — runs §31057366689 and §31055598155.
• This is not a downstream-repo problem — issue [aw-compat] Daily AW compat audit (2026-08-05): opencode engine removal has no migration codemod; 4 --strict findings are test-harness artif [Content truncated due to length] #50531 (opencode compat audit) only flagged that external repos lack a migration codemod. It missed that gh-aw's own scheduled workflows are broken by the same change. Every future scheduled run of these 2 workflows will fail until migrated.

PR Code Quality Reviewer — recurrence of #49022 (P1, tracked)

• Run §31054795358: all 4 grumpy-coder sub-agent launch attempts failed identically with [copilot-sdk-driver] error: Execution failed: Error: No model available..., failureClass=partial_execution, isSDKSessionIdleTimeoutError=false (ruling out the other [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 failure mode). Retries exhausted 3→2→1→0, job exited 1.
• Exact match to [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 mode (b). [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 last updated 2026-08-04, still open, hardening fix not shipped.

Detection Analysis Report — timeout, fix already in flight (P1, tracked)

• Run §31056417218: agent hit ModuleNotFoundError: No module named 'matplotlib', self-recovered by building a venv and reinstalling packages, successfully generated the chart at 23:51:30 — then hit ##[error]The action 'Execute Claude Code CLI' has timed out after 20 minutes 4 seconds later.
• PR fix: detection-analysis-report timeout and redundant log downloads #50716 ("fix: detection-analysis-report timeout and redundant log downloads") already diagnosed this precisely — 89 turns / 28.6 min real runtime against a 20-min default timeout, plus 6 redundant agenticworkflows logs calls — and proposes timeout-minutes: 45 plus a single-fetch prompt constraint. Fixes (closed) [aw] Detection Analysis Report failed #50710. Not yet merged — no new issue needed, just needs review/merge.

Avenger — fragile "Install Make" step (P1, untracked, no separate issue this cycle)

• Run §31061056320: sudo apt-get update failed because packages.microsoft.com/repos/azure-cli returned 403 Forbidden ("no longer signed"), unrelated to make. apt-get update treats any repo failure as fatal for the whole command, so the subsequent apt-get install -y make never ran — ##[error]Process completed with exit code 100.
• make ships preinstalled on GitHub-hosted ubuntu-latest runners; the apt-get update in this step is unnecessary in the common case and turns an unrelated Microsoft-mirror outage into a full workflow failure.
• Identical step exists in .github/workflows/hourly-ci-cleaner.md (same fragility, not yet triggered this window).
• Not filed as a separate sub-issue this cycle (create_issue budget: 1 parent + 1 P0 sub-issue). Recommend filing next window if it recurs, or fixing proactively: guard with command -v make || (sudo apt-get update && sudo apt-get install -y make).

Existing issue correlation

Don't touch the 6 other curated open issues — none show fresh evidence of a fix or staleness this window. #49583, #49144, #49446, #49023, #49096, #49246 re-checked against this window's failures; no matches, no closed-elsewhere or merged-fix signals. Left open, untouched. #49022 confirmed still-recurring (see above), left open. Prior parent #50077 (2026-08-03) left open — its own untracked items are unrelated to this window's findings.

Fix roadmap

P0 — do this first:

1. Migrate daily-security-red-team.md and daily-reliability-review.md off engine: opencode to a supported engine and recompile the lock files — both workflows currently fail 100% of scheduled runs. See sub-issue below.

P1 — next:
2. Merge #50716 (Detection Analysis Report timeout fix) — already written, just needs review.
3. Ship the #49022 grumpy-coder hardening (bounded sub-agent timeout + best-effort degrade) — still open, recurring.
4. Harden Avenger's (and hourly-ci-cleaner's) Install Make step to skip apt-get update when make is already present, so an unrelated Microsoft-mirror outage can't fail the job.

P2 — monitor only:

• None this window.

Sub-issues created

• opencode engine migration for daily-security-red-team.md / daily-reliability-review.md (P0)

References:

• §31055598155 — Daily Reliability Review
• §31057366689 — Daily Security Red Team
• §31061056320 — Avenger

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 146.7 AIC · ⊞ 5.2K · ◷

• expires on Aug 12, 2026, 5:23 PM UTC-08:00

---

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

---

Update — 2026-08-06 07:41 UTC pass: reopen #49022, it's still the dominant failure

Reopen #49022 immediately — closing it "not planned" 3 hours ago didn't stop it; it just fired 6 more times across 3 workflows and is still the single biggest source of failures in gh-aw. Done — reopened this pass with fresh evidence.

12 failed runs analyzed this window (2026-08-06 01:41–07:41 UTC). 8 clusters. 1 reopened (was wrongly closed), 1 new sub-issue filed, 1 corroborated (no new issue), 3 no-action (expected behavior / transient / already in-flight fix), 2 unconfirmed-but-presumed part of the reopened cluster.

Cluster summary

Sev	Cluster	Runs	Status
P0	Copilot "No model available" — PR Code Quality Reviewer (x4), Code Simplifier, GPL Dependency Cleaner	§31075980405 (confirmed), §31075406960, §31074728575, §31070819933, §31070779226, §31070577803	Reopened #49022 — was closed NOT_PLANNED 2026-08-06T03:45:57Z, recurred within hours
P1	Daily VulnHunter Scan — AI credit budget (1000) exceeded mid-scan, hard 403	§31077438828	New sub-issue filed (this issue)
P1	Smoke Copilot — dispatch_workflow safe-output failure	§31076914297, §31074756937	Tracked by #49583 — corroborating comment added, no new issue
P2	Auto-Triage Issues — push_evals_state GraphQL rebase conflict on evals.jsonl	§31074178819	No action — same evals.jsonl append-only mechanism is being reworked in open PR #50766
P2	Test Quality Sentinel — pre_activation checkout DNS resolution failure ("Could not resolve host: github.com")	§31076026746	No action — isolated, transient runner network blip
—	Daily Container Image Security Scan — gate failed on real critical CVEs/license violations	§31076719592	Not a bug — gate working as designed

Evidence

Copilot "No model available" — P0, reopened #49022

agent-stdio.log for §31075980405:

[copilot-sdk-driver] [sdk-driver] error: Execution failed: Error: No model available. Check policy enablement under GitHub Settings > Copilot
[copilot-harness] attempt 4: process closed exitCode=1 duration=50s ... failureClass=partial_execution
[copilot-harness] all 3 retries exhausted — giving up (exitCode=1)

audit cohort-comparison flags this run risky against a same-day successful baseline (§31075085084): posture flipped write_capable→read_only, run_unsuccessful.

The other 5 runs in this cluster share the same workflow/engine (Copilot) and failed within tight time clusters (04:11–04:17 UTC and 05:38–06:02 UTC) but were not individually log-fetched this pass (audit-call budget) — presumed same signature, not independently confirmed.

This is the same signature #49022 was opened for and #48897 before it — 3rd consecutive 6h window with a recurrence, now spanning 3 workflows instead of 1.

Daily VulnHunter Scan — AI credits exceeded (P1, new sub-issue)

agent-stdio.log for §31077438828:

API Error: 403 Maximum AI credits exceeded (1032.294175 / 1000).

14 turns, total_cost_usd: 6.19, harness classified this as non-retryable isAuthenticationFailedError=true and gave up on attempt 1 — no partial output salvaged. Full detail and remediation in the new sub-issue.

Auto-Triage Issues — evals.jsonl push conflict (P2, no action, in-flight fix exists)

audit for §31074178819:

Failed to push evals results after 4 attempts: pushSignedCommits: failed to rebase commit range onto current GraphQL parent ... Root cause: Auto-merging evals.jsonl

Isolated this window. Open PR #50766 ("Append persisted eval results across workflow runs") reworks push_experiment_state.cjs to merge evals.jsonl as append-only instead of overwrite — directly targets this failure mode. No new issue needed; flag for re-check next window if #50766 hasn't merged and this recurs.

Existing issue correlation

• [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 — reopened this pass (see above).
• [aw-failures] dispatch_workflow safe-output fails the whole Smoke Copilot run when the ephemeral trigger branch is gone #49583 — still open, 2 more corroborating occurrences this window, comment added, no new issue.
• [aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144, [aw-failures] Fix: unguarded arithmetic on github.event.pull_request.stack in applyPullRequestStackFilter #49446, [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096, [aw-failures] MCP Gateway github SSE reconnect hang + Ollama BYOK backend 503 fast-fail #49246, [aw-failures] Migrate daily-security-red-team.md and daily-reliability-review.md off removed opencode engine #50735, [aw] [aw] Failure Investigator (6h) timed out #50736 — re-checked, no fresh evidence of fix or staleness this window, left untouched.
• [aw] No-Op Runs #48838, [reliability] Daily Reliability Review - 2026-08-04 #50420, [ambient-context] Daily Ambient Context Optimizer - 2026-08-05 #50687, [agentic-token-audit] Daily AIC Usage Audit — 2026-08-04 #50243, [evals] Daily Evals Feature Report - 2026-08-05 #50521 — unrelated report/audit issues, not failure signatures, left untouched.

Fix roadmap

P0 — do this first:

1. Ship the [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 hardening now — treat Copilot model-allocation failure as non-fatal/best-effort instead of 4x full-session retries. This is ~50% of this window's failures and has recurred across 3 consecutive 6h windows.

P1 — next:
2. Raise or gracefully enforce Daily VulnHunter Scan's AI credit budget (currently 1000, this run needed ~1032) — see new sub-issue.
3. Merge #50766 to fix the evals.jsonl push-conflict mode hitting Auto-Triage Issues (and likely other state-pushing workflows).
4. Land the #49583 dispatch_workflow ref-pinning fix — still recurring, 2 more hits this window.

P2 — monitor only:

• Test Quality Sentinel DNS resolution blip — isolated, no action unless it recurs.

Sub-issues created

• Daily VulnHunter Scan AI credit budget exceeded (P1) — parented to this issue.

References:

• §31075980405 — PR Code Quality Reviewer
• §31077438828 — Daily VulnHunter Scan
• §31074178819 — Auto-Triage Issues> Generated by 🔍 [aw] Failure Investigator (6h) · agent · 148.1 AIC · ⊞ 5.2K · ◷