[aw-failures] [aw] Failure Investigator Report — 2026-07-31 (6h)

[github-actions]

Fix the MCP Gateway → github SSE reconnect hang — it wasted 41 min and 206.8 AIC on one run, and it's the 4th failure in 7 days.

Fix the Daily BYOK Ollama Test's blind retry against a dead backend — it burned 4 full retry cycles against an upstream that was 503'ing the whole time.

Both are new failures with zero existing tracking coverage — none of the 8 open agentic-workflows issues reference these workflows. One consolidated sub-issue is filed below for both (create_issue capped at 2/run).

Executive summary

• Window: last 6h (2026-07-30 19:57Z → 22:46Z), repo github/gh-aw.
• 3 failed runs, 2 distinct new P0 root causes + 1 isolated P2. No overlap with existing open issues.
• P0-A: Agentic Workflow Audit Agent hangs whenever the internal MCP Gateway loses its SSE stream to the github MCP server — recurring 4x in the last 7 scheduled runs, not a one-off.
• P0-B: Daily BYOK Ollama Test fails outright when the Ollama backend proxy (api-proxy:10002) 503s on /v1/models — no fast-fail/health-gate exists, so it burns the full retry budget every time this happens.
• P2: Smoke Copilot's safe_outputs job step failed despite the agent job succeeding and the artifact uploading cleanly — likely a flaky post-processing step, not a real data-loss bug. No deep-dive performed (below P0/P1 threshold this cycle).

Failure cluster table

Cluster	Severity	Workflow	Run ID	Failed step	Tracked?
A	P0	Agentic Workflow Audit Agent	30583831507	Execute Claude Code CLI	❌ new gap
B	P0	Daily BYOK Ollama Test	30587927074	Execute GitHub Copilot CLI	❌ new gap
C	P2	Smoke Copilot	30577084701	Process Safe Outputs (safe_outputs job)	❌ new gap, not deepened (P2)

Evidence — Cluster A: MCP Gateway SSE reconnect hang

agent-stdio.log for run 30583831507 shows the github MCP server connection cycling:

21:42:07 HTTP connection dropped after ... uptime — SSE stream disconnected: TimeoutError
21:47:08 HTTP connection dropped — SSE stream disconnected: TimeoutError (2/3)
21:52:09 HTTP connection dropped — SSE stream disconnected: TimeoutError (3/3) — transport closed
22:02:58 reconnected successfully, one tool call (get_commit) succeeds
22:07:07 HTTP connection dropped after 249s uptime — SSE stream disconnected: TimeoutError
22:07:22 Unable to connect. Is the computer able to access the url?
22:07:27 Maximum reconnection attempts (2) exceeded — SSE GET-stream reconnection exhausted

The transcript ends immediately after that line (no further activity), while the job itself didn't report completed_at until ~6.5 min later — consistent with a hang killed by an external timeout rather than a clean exit.

audit-diff against the last successful run of the same workflow (30492378500, 2026-07-29) confirms the cost of the hang:

metric	failed run (30583831507)	last success (30492378500)
duration	41m1s	(baseline)
GitHub API requests	59	17
AIC burned	206.78	0 (no completion recorded)
cache-read tokens	7,068,514	0

Workflow history (gh run list) shows this is recurring, not isolated: failures on 2026-07-25, 07-26, 07-27, and now 07-30 — 4 of the last 7 scheduled runs.

Root cause: the MCP Gateway (awmg-mcpg) → github MCP server SSE channel is not resilient to drops under longer-running sessions; reconnection gives up after only 2 attempts, and there is no step-level timeout to fail the run quickly once the channel is dead, so the CLI hangs burning tokens instead of failing fast.

Evidence — Cluster B: Ollama BYOK backend 503 loop

agent-stdio.log for run 30587927074 (Daily BYOK Ollama Test):

awf-reflect models fetch for (apiproxy/redacted) failed (attempt 1-4/5): 503 Service Unavailable
● Request failed due to a transient API error. Retrying... (x5)
Failed to get response from the AI model; retried 5 times ... Last error: 503 Service Unavailable
[copilot-harness] attempt 1 failed: exitCode=1 failureClass=partial_execution ... attemptDurationMs=37866

This 5-retry / 503 cycle repeats identically for all 4 harness attempts (durations 37s, 1m12s, 1m57s, 2m48s), each ending the same way, before the step finally gives up.

Root cause: the Ollama backend proxy (api-proxy:10002) was unhealthy/unavailable for the full run — an external dependency outage, not a gh-aw code bug — but there is no pre-flight health check or fast-fail circuit breaker, so the workflow burns its entire retry budget every time this happens instead of failing in one attempt.

Existing issue correlation

Checked against all 8 open agentic-workflows-labeled issues (#49189, #49144, #49143, #49096, #49022, #49023, #49095, #48897) — no title or error signature overlaps any of the 3 clusters above; all 3 are confirmed new gaps. No existing issues were closed this cycle — none of them relate to the workflows analyzed here, so there's no fresh evidence to justify closing any of them.

Fix roadmap

• P0 — Harden awmg-mcpg's SSE reconnection to the github MCP server (higher retry ceiling + backoff) and add a hard step-level timeout so a dead channel fails the run in minutes, not 40+. → tracked in sub-issue below.
• P0 — Add a pre-flight health check / circuit breaker for the Ollama BYOK backend proxy so a 503'ing backend fails fast instead of burning all 4 retry attempts. → tracked in sub-issue below.
• P2 — Investigate the Smoke Copilot safe_outputs job's "Process Safe Outputs" step marked failed despite a clean artifact upload (run 30577084701). Not deepened this cycle (below P0/P1 threshold); re-open investigation if it recurs.

Sub-issues created

• #aw_p0fix — consolidated P0 fix for Clusters A + B (linked below as sub-issue of this report).

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 131.7 AIC · ⌖ 20.5 AIC · ⊞ 6.8K · ◷

• expires on Aug 6, 2026, 5:28 PM UTC-08:00

---

Update — 2026-08-01 (6h pass)

Fix the investigator's own false-positive rate first — 63 of 68 "failures" this window were GitHub-cancelled Dependabot-burst matrix runs, not real bugs. Filed as a sub-issue with the exact line to change.

Fix dispatch_workflow's implicit ref before the next ephemeral-branch smoke run — a 15-minute Smoke Copilot run failed entirely because its trigger branch was deleted before the finishing dispatch fired. Filed as a sub-issue with a concrete remediation.

Correlation with open issues: #49143 (Daily Max Ai Credits Test) recurred again today (§30697984980) — no new signature, left as-is. #49022 (PR Code Quality Reviewer background sub-agent) also recurred (§30690753725 window-adjacent failure at 07:56:09) — already tracked with clear success criteria, no action needed here. Contribution Check (evals/Install GitHub Copilot CLI, §30693860813) and daily-experiment-report (§30692043079) each had one isolated failure with no error captured in the 50-line log tail — insufficient evidence for a sub-issue yet; re-check if either recurs.

Sub-issues: see linked children of this issue.

Generated by 🔍 [aw] Failure Investigator (6h) · agent · 191.1 AIC · ⌖ 25.1 AIC · ⊞ 6.8K · ◷