[aw-failures] [aw] Failure Investigator Report — 2026-08-03 (6h)

[github-actions]

Fix the Workflow Portfolio Analyst Sentry loop first — it's the only P0 with zero tracking coverage.

8 failed runs in the last 6 hours, 3 unrelated root-cause clusters already tracked, 1 untracked P0, 4 untracked lower-severity findings folded into this report instead of filed separately (create_issue budget: 2/run).

Cluster summary

Sev	Cluster	Runs	Status
P0	Sentry MCP sum() on string field → 400 loop → exhausts 20/20 LLM invocation cap	§30833881004	Untracked — sub-issue filed below
P0	Task-tool sub-agent "No model available" (cross-workflow)	§30839283638, §30826409804, §30817695295	Tracked — broadened #49022 today
P1	resolve_pull_request_review_thread gets a PRRC_ id instead of PRRT_	§30838483274	Untracked, no separate issue this cycle — see below
P1	CI Optimization Coach: prompt.txt ENOENT, total run failure	§30820313785	Untracked, no separate issue this cycle — see below
P2	Impeccable Skills Reviewer: threat-detect curl connection-reset install	§30824511155	Non-fatal (GH_AW_DETECTION_CONTINUE_ON_ERROR=true), no action needed
P2	Slide Deck Maintainer: git lfs fetch connection-reset during checkout	§30834427727	Transient network, no action needed

Evidence

Workflow Portfolio Analyst — Sentry MCP invocation-cap exhaustion (P0, untracked)

• Run §30833881004: agent repeatedly calls the Sentry list_events tool with sum(gh-aw.aic) / sum(gh-aw.action_minutes) — both string-typed fields. Sentry returns HTTP 400 each time. The agent doesn't recognize the error as terminal and retries the same malformed query shape with minor variations until it hits the 20/20 LLM invocation cap, at which point the harness kills the run with zero useful output.
• This is not a one-off. gh api "/repos/github/gh-aw/actions/workflows/portfolio-analyst.lock.yml/runs?per_page=10" shows this workflow has failed every scheduled run since 2026-06-22 (7+ consecutive failures). Last success was 2026-06-15 (run 27574552812).
• audit-diff between this run and the prior failure (§30285971411) shows the trend getting worse, not just persisting: GitHub API core-rate-limit consumption +75%, total API call count +18% run-over-run — consistent with the agent doing more wasted retry work each cycle as it burns invocations on the same dead-end query.

PR Sous Chef — resolve_pull_request_review_thread wrong node-id type (P1, untracked)

• Run §30838483274: the resolve_pull_request_review_thread safe-output tool expects a PullRequestReviewThread GraphQL node id (PRRT_...) but was passed a PullRequestReviewComment id (PRRC_...) instead, and the GraphQL mutation rejected it. Looks like a type-confusion bug where the agent (or the tool-call construction code) is grabbing the comment id instead of walking up to the parent thread id.
• Not filed as a separate sub-issue this cycle (create_issue budget exhausted by the two issues below). Recommend filing next window if it recurs, pointing at wherever resolve_pull_request_review_thread's input id is sourced from in the safe-outputs processor.

CI Optimization Coach — prompt.txt ENOENT (P1, untracked)

• Run §30820313785: the agent job fails immediately, before any LLM invocation, because the compiled prompt file is missing on disk (ENOENT: no such file or directory, open '.../prompt.txt'). This is a total run failure, not a partial/degraded one — worth prioritizing over the P2s below despite being a single occurrence, since it means the workflow never got a chance to do anything.
• Not filed as a separate sub-issue this cycle (budget). Recommend checking the compile/render step for this specific workflow's .lock.yml for a path or working-directory mismatch if it recurs.

P2s — no action needed

• Impeccable Skills Reviewer §30824511155: threat-detect binary install via curl hit Recv failure: Connection reset by peer. Non-fatal — GH_AW_DETECTION_CONTINUE_ON_ERROR=true let the run continue. Classic transient network blip, not a code defect.
• Slide Deck Maintainer §30834427727: git lfs fetch origin refs/remotes/origin/main exited 2 during checkout. Also transient network, self-resolves on retry.

Existing-issue correlation

Don't touch the other 6 curated open issues — none show fresh evidence of a fix or staleness this window. #49583, #49144, #49446, #49023, #49096, #49246 were each re-checked against this window's failures; none matched, and none have merged-fix or closed-elsewhere signals. Left open, untouched.

Fix roadmap

P0 — do these first:

1. Workflow Portfolio Analyst: fix (or guard against) the Sentry sum()-on-string-field query, and cap retries on repeated identical 400s so the harness fails fast instead of burning the full invocation budget. See sub-issue below.
2. Task-tool "No model available": already tracked in [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 (broadened today to cross-workflow scope) — no new issue needed.

P1 — next:
3. resolve_pull_request_review_thread: fix id-type resolution so it always passes a PRRT_ id, not PRRC_.
4. CI Optimization Coach: fix whatever compile/path step is failing to produce prompt.txt for the agent job.

P2 — monitor only:
5. threat-detect curl reset and git-lfs reset are both transient; no fix needed unless they start recurring.

Sub-issues created

• One new sub-issue filed for the Workflow Portfolio Analyst Sentry MCP query loop (P0, untracked) — linked as a child of this report.

References:

• §30833881004 — Workflow Portfolio Analyst (P0)
• §30838483274 — PR Sous Chef (P1)
• §30820313785 — CI Optimization Coach (P1)

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 224.7 AIC · ⌖ 41.3 AIC · ⊞ 6.8K · ◷

• expires on Aug 10, 2026, 11:28 AM UTC-08:00

---

Update — 2026-08-04 (6h window, 01:05–01:13 UTC batch)

File the two new P1 sub-issues below first — they're one-off config bugs, not infra drift. 5 failed runs this window: 2 brand-new untracked P1s, 2 recurring-but-already-tracked hits, 0 P0s.

Cluster summary

Sev	Cluster	Runs	Status
P1	update_pull_request denied 6x (Permission Denied) — PR Description Updater	§30867631807	Untracked — sub-issue filed
P1	push_to_pull_request_branch rejects fix touching actions/setup/** (outside allowed-files) — Design Decision Gate	§30867235474	Untracked — sub-issue filed
P1	Background task-tool sub-agent hang (~19.4m, 1 turn) — PR Code Quality Reviewer	§30861967269, §30859127480	Tracked — still recurring under #49022, not yet fixed
P1	Ollama backend 503 burns full 4×5 retry budget — Daily BYOK Ollama Test	§30859350200	Tracked — still recurring under #49246 (Fix B), not yet fixed

Evidence

PR Description Updater — permission_denied (P1, untracked, sub-issue filed)

• Run §30867631807: agent synthesized a ready PR description, then update_pull_request was denied 6 times (failureClass=permission_denied, hasNumerousPermissionDenied=true). Harness correctly stopped retrying and emitted missing_tool + report_incomplete instead of looping — the remaining fix is the underlying permission grant, not the retry logic.

Design Decision Gate — allowed-files rejection (P1, untracked, sub-issue filed)

• Run §30867235474: agent job succeeded; only safe_outputs/push_to_pull_request_branch failed because the fix bundle touched actions/setup/js/parse_token_usage.{cjs,test.cjs} and actions/setup/sh/audit_pre_agent_workspace{,_test}.sh, outside the configured allow-list.
• audit-diff vs nearest successful run of the same workflow (§30866929279): 0 firewall/domain drift, 0 anomalies, GitHub API call volume within 7% of baseline — confirms this is a scope-config bug, not infra regression.
• Distinct from [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096 (LLM invocation cap via permission-denied retries) and [aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144 (pre-fetch step fails before agent starts) — third distinct Design Decision Gate safe-output failure mode this week.

PR Code Quality Reviewer — sub-agent hang (P1, tracked under #49022, still recurring)

• Runs §30861967269 (19.4m, task tool call_count=1, turns=1) and §30859127480: both match the idle-timeout hang pattern already documented in [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022. [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 was last updated today (2026-08-03T19:28Z) and is not yet closed — the hardening fix hasn't landed.

Daily BYOK Ollama Test — 503 retry-budget burn (P1, tracked under #49246 Fix B, still recurring)

• Run §30859350200: raw log confirms exact match to [aw-failures] MCP Gateway github SSE reconnect hang + Ollama BYOK backend 503 fast-fail #49246 Fix B — models fetch returned 503 on api-proxy:10002/v1/models, then 4 attempts × 5 sub-retries each of "Request failed due to a transient API error... 503 Service Unavailable" before giving up (9.4m total). [aw-failures] MCP Gateway github SSE reconnect hang + Ollama BYOK backend 503 fast-fail #49246 was filed 2026-07-31 and proposed a fast-fail health gate; it hasn't shipped — this is the same outage pattern recurring 3+ days later.

Existing issue correlation

• No issues closed this window — [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 and [aw-failures] MCP Gateway github SSE reconnect hang + Ollama BYOK backend 503 fast-fail #49246 are confirmed still-open and still-recurring (fixes not yet shipped), not stale.
• 2 new sub-issues filed under this parent for the untracked P1 gaps above.

Fix roadmap

• P0: none this window.
• P1: (1) grant/fix update_pull_request permission for PR Description Updater — sub-issue filed; (2) widen or pre-flight-check Design Decision Gate's allowed-files scope — sub-issue filed; (3) ship the [aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022 sub-agent hardening (still open, recurring); (4) ship the [aw-failures] MCP Gateway github SSE reconnect hang + Ollama BYOK backend 503 fast-fail #49246 Ollama fast-fail gate (still open, recurring).
• P2: none flagged this window.

Sub-issues created

• PR Description Updater update_pull_request permission-denied fix
• Design Decision Gate allowed-files scope fix

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 122.7 AIC · ⌖ 41 AIC · ⊞ 5.5K · ◷

---

2026-08-04 (6h) update

Fix the awf PATH gap in Smoke Goose and bump the sandbox Go toolchain — those are the two new, confirmed, untracked root causes from this pass. Everything else in this window is either an expected gate firing or an already-tracked recurrence.

4 failed runs in the last 6 hours, 1 legitimate gate (no bug), 1 confirmed recurrence of a tracked issue, 2 new confirmed root causes filed as sub-issues below.

Cluster summary

Sev	Cluster	Runs	Status
P1	Smoke Goose: awf: command not found (exit 127) before Goose starts	§30880381421	Untracked — sub-issue filed: #aw_goose1
P1	Go Logger Enhancement: sandbox Go 1.25.12 < go.mod's required 1.26.5 blocks make build/make fmt	§30877164579	Untracked — sub-issue filed: #aw_golog1
P1	PR Code Quality Reviewer: task-tool sub-agent "No model available" exhausts 4/4 retries	§30876479583	Tracked — recurrence appended to #49022
—	Daily Container Image Security Scan: "Critical vulnerabilities detected" gate fired	§30883387143	No action — gate working as intended, real CVEs found upstream, not a workflow bug

Evidence

Smoke Goose — `awf: command not found` (P1, untracked, sub-issue filed)

Full job log (not just the tail) shows the "Execute Goose CLI" step's generated script invoking awf --config ... -- /bin/bash -c '... goose_harness.cjs goose', which fails at line 23 with awf: command not found, exit 127 — before Goose itself ever runs. No other engine analyzed in this window hit a missing-binary error at this stage; looks specific to the Goose engine's generated job steps. See #aw_goose1 for full remediation.

Go Logger Enhancement — GOTOOLCHAIN/go.mod drift (P1, untracked, sub-issue filed)

audit + audit-diff (vs. last success §30784179371) confirm the agent hit go: go.mod requires go >= 1.26.5 (running go 1.25.12; GOTOOLCHAIN=local), could not validate its change with make build && make fmt, and correctly called report_incomplete instead of opening an unvalidated PR. This is environment drift, not an agent bug — the CLI itself exited 0. See #aw_golog1 for full remediation.

PR Code Quality Reviewer — task-tool sub-agent model allocation (P1, tracked recurrence)

Exact repeat of the failure mode already diagnosed on #49022: background sub-agent grumpy-coder hit No model available. Check policy enablement under GitHub Settings > Copilot on all 4 harness retries, exhausting retriesRemaining=0. Appended as new evidence to #49022 rather than filing a duplicate — fix there is unchanged and now overdue given the repeat.

Daily Container Image Security Scan — critical CVE gate (no bug)

The "Enforce critical vulnerability and license gates" step is a real gate: it scans compiled scan output for : error: [Critical] and exits 1 when found. Log tail confirms ##[error]Critical vulnerabilities detected in container images. — this is the scanner correctly catching real critical CVEs in a scanned base image, not a script or tooling defect. No workflow fix needed; remediation (if any) is a base-image/dependency update tracked outside this bug queue.

Existing issue correlation

Checked all 17 open agentic-workflows-labeled issues against this window's 4 failures. Only one match: PR Code Quality Reviewer's task-tool failure is an exact recurrence of #49022 (updated in place). The other 3 failures (CVE gate, Smoke Goose, Go Logger) have no prior open-issue coverage — 2 of those 3 got dedicated sub-issues; the CVE gate needs none.

Fix roadmap

• P0: none this window.
• P1: fix the awf PATH gap in the Goose engine job setup (#aw_goose1); bump sandbox Go toolchain to >=1.26.5 or set GOTOOLCHAIN=auto (#aw_golog1); harden the task-tool sub-agent model-allocation retry path ([aw-failures] Fleet-wide: Copilot CLI subagent model allocation fails with 'No model available' (hit PR Code Quality Reviewer, L [Content truncated due to length] #49022, now recurring — treat as overdue).
• P2: none new this window.

Sub-issues created

• #aw_golog1 — Go Logger Enhancement: bump sandbox Go toolchain
• #aw_goose1 — Smoke Goose: awf: command not found

Tooling note

3 of 4 runs' pre-fetched truncated_error_logs captured only post-job cleanup / mid-task transcript noise instead of the actual failing step's output — real root cause required full-log audit calls each time. Worth a look at the log-truncation logic behind this investigator's pre-fetch step so future passes get the right tail on the first try; not filed as its own issue this cycle to stay inside the 2-issue create budget.

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 191.6 AIC · ⌖ 37.4 AIC · ⊞ 5.5K · ◷

---

---

2026-08-04 (6h) update — 07:27–12:14 UTC batch

File the Daily Max Ai Credits Test billing-proxy fix first — it's a P0 that's been silently failing every run for over a week, and had zero tracking until now. 5 failed runs this window: 1 new P0 (long-standing, just discovered), 1 new P1, 1 tracked recurrence, 1 tracked-issue evolution (bug changed, not fixed), 1 folded into narrative below budget.

Cluster summary

Sev	Cluster	Runs	Status
P0	Daily Max Ai Credits Test: Copilot provider-proxy 403 (billing/permission gap), 10/10 runs failing since 2026-07-27	§30906430299	Untracked — sub-issue filed
P1	Copilot Session Insights: all-or-nothing artifact check fails whole run on any missing session log (10/10 missing this run)	§30887922501	Untracked — sub-issue filed
P1	PR Code Quality Reviewer: task-tool sub-agent "No model available" exhausts retry budget (again)	§30906965561	Tracked — recurrence appended to #49022, still unfixed
P1	Smoke Goose: awf: command not found fix landed, but new blocker — goose_harness.cjs ENOENT on .goose/mcp.json	§30908234685	Tracked — evolution appended to #50214, still open
P2	Tidy: golangci-lint firewall 403 + sandboxed shell "Permission denied" on direct curl/go calls, agent recovers but step then hangs silently until 10-min timeout	§30887863573	Untracked, no separate issue this cycle — see below

Evidence

Daily Max Ai Credits Test — Copilot provider-proxy 403 (P0, untracked, sub-issue filed)

Raw log: Authentication failed with provider at (172.30.0.30/redacted) (HTTP 403), harness message explicitly names the cause — permissions.copilot-requests: write requests must clear the org's centralized Copilot billing configuration, and they aren't. audit-diff vs. the prior day's failure (§30812833453) shows 0 firewall drift, 0 anomalies — not a sandbox regression, a standing config gap. gh api .../workflows/daily-max-ai-credits-test.lock.yml/runs confirms 10/10 consecutive failures back to 2026-07-27 — this has been broken and unreported for over a week.

Copilot Session Insights — all-or-nothing artifact gate (P1, untracked, sub-issue filed)

Step samples 10 prior agent run IDs and fails outright when even one events.jsonl/transcript can't be downloaded. This run hit 10-for-10 missing, then hard-failed with Missing per-session logs for 10 of 10 agent runs; failing to prevent incomplete optimization analysis. Either the sampled run IDs fell outside artifact retention, or the fetch mechanism itself is broken — either way the check should degrade gracefully instead of an all-or-nothing exit 1.

PR Code Quality Reviewer — task-tool sub-agent model allocation (P1, tracked recurrence, still unfixed)

Exact repeat of #49022's signature: No model available. Check policy enablement under GitHub Settings > Copilot, hit by both the primary grumpy-coder sub-agent and a second grumpy-coder-1 spawn later in the same run. Appended as a new recurrence entry to #49022 — the three remediation items proposed there (explicit timeout, best-effort degradation, fail-fast instead of full-session replay) still haven't shipped.

Smoke Goose — awf PATH fix landed, new goose_harness.cjs ENOENT (P1, tracked evolution, still open)

Good news/bad news: awf: command not found (#50214's original signature) is gone — awf now runs. But the step still fails one line later, inside the harness itself: Error: ENOENT: no such file or directory, open '.goose/mcp.json' at goose_harness.cjs:26, a relative-path lookup that no longer resolves given the container-workdir change that likely fixed the PATH issue. Appended to #50214 rather than filing a new issue — same overall "Smoke Goose broken" cluster, evolved root cause, not a fix yet.

Tidy — golangci-lint firewall 403 → sandbox permission-denied → silent hang → 10-min timeout (P2, untracked, no separate issue)

Agent hit repeated Permission denied and could not request permission from user on direct curl/go install calls while golangci-lint's release download 403'd through the firewall — but it correctly worked around this (make fmt succeeded via the wrapped path; agent logged "Go linting can't run... infra limitation, not a code issue" and moved on). The real bug: after that log line, the job produced zero further output for 55 seconds and then hit ##[error]The action 'Execute GitHub Copilot CLI' has timed out after 10 minutes — a silent hang, not a clean failure, suggesting whatever came next (recompiling workflows / running tests) blocked on a firewall-denied call with no error surfaced. Not filed as its own sub-issue this cycle — worth filing next window if it recurs, pointing at why the CLI hangs silently instead of erroring when a subsequent tool call is sandbox-denied.

Existing-issue correlation

Checked all currently open agentic-workflows-labeled issues against this window's 5 failures. 2 are exact recurrences of already-open issues (#49022, #50214) — both updated in place, neither closed (neither is fixed). No issue in this window shows fix or staleness signals; nothing closed this cycle.

Fix roadmap

P0 — do this first:

1. Daily Max Ai Credits Test: fix copilot-requests: write billing/permission grant through the gh-aw API proxy — sub-issue filed, 8+ day outage.

P1 — next:
2. Copilot Session Insights: make the per-session artifact check degrade instead of hard-failing — sub-issue filed.
3. PR Code Quality Reviewer / #49022: ship the sub-agent hardening fix — now recurring across multiple windows, overdue.
4. Smoke Goose / #50214: fix goose_harness.cjs's relative .goose/mcp.json lookup — PATH fix landed, this is the new blocker.

P2 — monitor:
5. Tidy: silent hang after a sandbox-denied call, until 10-min timeout. Not filed yet — file if it recurs.

Sub-issues created

• Daily Max Ai Credits Test: Copilot provider-proxy 403 (P0)
• Copilot Session Insights: all-or-nothing artifact check (P1)

References:

• §30906430299 — Daily Max Ai Credits Test (P0)
• §30887922501 — Copilot Session Insights (P1)
• §30908234685 — Smoke Goose (P1, evolution)

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 232.9 AIC · ⌖ 42.2 AIC · ⊞ 5.2K · ◷