Fix A: Harden MCP Gateway → github SSE reconnection (Agentic Workflow Audit Agent)
Problem statement: Execute Claude Code CLI hangs and ultimately fails whenever the MCP Gateway (awmg-mcpg) loses its SSE connection to the github MCP server mid-run. Reconnection is abandoned after only 2 attempts, and there's no step-level timeout, so the process burns tokens for tens of minutes before dying instead of failing fast.
Affected workflows / run IDs:
Agentic Workflow Audit Agent (.github/workflows/audit-workflows.lock.yml) — run 30583831507 (2026-07-30, latest failure)- Same workflow also failed on 2026-07-25 (run 30175619709), 2026-07-26 (run 30221074895), 2026-07-27 (run 30307242647) — 4 of the last 7 scheduled runs.
- Last good baseline for comparison: run 30492378500 (2026-07-29, success).
Probable root cause: agent-stdio.log shows the github MCP server's SSE stream dropping repeatedly ("HTTP connection dropped after ~250-900s uptime", "SSE stream disconnected: TimeoutError"), recovering once via reconnect, then failing again and exhausting its 2-attempt reconnection budget ("Maximum reconnection attempts (2) exceeded"). The transcript ends immediately after that log line with no further activity, while the job doesn't report completion until ~6.5 minutes later — indicating the CLI process hangs rather than exiting cleanly. audit-diff against the last successful run confirms the cost: the failed run ran 41m1s, made 59 GitHub API calls, and burned 206.78 AIC / ~7M cache-read tokens before producing zero usable output.
Proposed remediation:
- Increase SSE reconnection attempts/backoff in the MCP Gateway's github MCP client, or make the reconnect budget configurable.
- Add a hard step-level timeout in the
Execute Claude/Copilot CLI step (or a gateway-side liveness check) so a permanently-dead MCP channel fails the run within a few minutes instead of hanging for 40+.
- Emit a clear, greppable error (not just a DEBUG log line) when reconnection is exhausted, so
Detect agent errors can classify it distinctly from generic partial_execution.
Success criteria / verification: Next 5 scheduled runs of Agentic Workflow Audit Agent complete without an SSE-reconnection-exhaustion hang; if the upstream MCP server does drop, the run fails within ~5 minutes (verifiable via agenticworkflows audit duration/AIC on the failing run) rather than burning 40+ minutes and 200+ AIC.
Fix B: Fast-fail / health-gate the Ollama BYOK backend (Daily BYOK Ollama Test)
Problem statement: Execute GitHub Copilot CLI fails after burning its entire retry budget (4 attempts × 5 sub-retries each) when the Ollama backend proxy is down, instead of detecting the outage and failing in one attempt.
Affected workflows / run IDs:
Daily BYOK Ollama Test (.github/workflows/daily-byok-ollama-test.lock.yml) — run 30587927074 (2026-07-30).
Probable root cause: agent-stdio.log shows awf-reflect models fetch for (apiproxy/redacted) returning 503 Service Unavailableon every attempt (1-4/5 retries each), for all 4 harness attempts (durations 37s, 1m12s, 1m57s, 2m48s — cumulative ~6 minutes), with the harness reportingfailureClass=partial_execution exitCode=1` each time. The upstream Ollama backend proxy itself was unhealthy for the full run — this is an external dependency outage, not a gh-aw logic bug — but there's no pre-flight health check, so the run wastes its full retry budget instead of failing fast on the first 503.
Proposed remediation:
- Add a pre-flight health check (single lightweight
GET /v1/models) before starting the Copilot CLI step; if it fails, short-circuit to a clear "Ollama backend unavailable" failure instead of running the full retry matrix.
- Reduce the retry ceiling for this specific 503-on-
/v1/models signature (it's a backend-down signal, not a transient blip worth 5×4 retries).
- Consider skipping/soft-failing the scheduled run (rather than hard-failing) when the backend health check fails, so this doesn't need human triage every time the shared Ollama backend has an outage.
Success criteria / verification: Next occurrence of an Ollama backend outage causes Daily BYOK Ollama Test to fail (or soft-skip) within ~1 minute of the first health-check failure, rather than running the full ~6-minute, 4-attempt retry matrix.
Related to #49245
Generated by 🔍 [aw] Failure Investigator (6h) · agent · 131.7 AIC · ⌖ 20.5 AIC · ⊞ 6.8K · ◷
Fix A: Harden MCP Gateway → github SSE reconnection (Agentic Workflow Audit Agent)
Problem statement:
Execute Claude Code CLIhangs and ultimately fails whenever the MCP Gateway (awmg-mcpg) loses its SSE connection to thegithubMCP server mid-run. Reconnection is abandoned after only 2 attempts, and there's no step-level timeout, so the process burns tokens for tens of minutes before dying instead of failing fast.Affected workflows / run IDs:
Agentic Workflow Audit Agent(.github/workflows/audit-workflows.lock.yml) — run 30583831507 (2026-07-30, latest failure)Probable root cause:
agent-stdio.logshows thegithubMCP server's SSE stream dropping repeatedly ("HTTP connection dropped after ~250-900s uptime", "SSE stream disconnected: TimeoutError"), recovering once via reconnect, then failing again and exhausting its 2-attempt reconnection budget ("Maximum reconnection attempts (2) exceeded"). The transcript ends immediately after that log line with no further activity, while the job doesn't report completion until ~6.5 minutes later — indicating the CLI process hangs rather than exiting cleanly.audit-diffagainst the last successful run confirms the cost: the failed run ran 41m1s, made 59 GitHub API calls, and burned 206.78 AIC / ~7M cache-read tokens before producing zero usable output.Proposed remediation:
Execute Claude/Copilot CLIstep (or a gateway-side liveness check) so a permanently-dead MCP channel fails the run within a few minutes instead of hanging for 40+.Detect agent errorscan classify it distinctly from genericpartial_execution.Success criteria / verification: Next 5 scheduled runs of
Agentic Workflow Audit Agentcomplete without an SSE-reconnection-exhaustion hang; if the upstream MCP server does drop, the run fails within ~5 minutes (verifiable viaagenticworkflows auditduration/AIC on the failing run) rather than burning 40+ minutes and 200+ AIC.Fix B: Fast-fail / health-gate the Ollama BYOK backend (Daily BYOK Ollama Test)
Problem statement:
Execute GitHub Copilot CLIfails after burning its entire retry budget (4 attempts × 5 sub-retries each) when the Ollama backend proxy is down, instead of detecting the outage and failing in one attempt.Affected workflows / run IDs:
Daily BYOK Ollama Test(.github/workflows/daily-byok-ollama-test.lock.yml) — run 30587927074 (2026-07-30).Probable root cause:
agent-stdio.logshowsawf-reflect models fetch for (apiproxy/redacted) returning503 Service Unavailableon every attempt (1-4/5 retries each), for all 4 harness attempts (durations 37s, 1m12s, 1m57s, 2m48s — cumulative ~6 minutes), with the harness reportingfailureClass=partial_execution exitCode=1` each time. The upstream Ollama backend proxy itself was unhealthy for the full run — this is an external dependency outage, not a gh-aw logic bug — but there's no pre-flight health check, so the run wastes its full retry budget instead of failing fast on the first 503.Proposed remediation:
GET /v1/models) before starting the Copilot CLI step; if it fails, short-circuit to a clear "Ollama backend unavailable" failure instead of running the full retry matrix./v1/modelssignature (it's a backend-down signal, not a transient blip worth 5×4 retries).Success criteria / verification: Next occurrence of an Ollama backend outage causes
Daily BYOK Ollama Testto fail (or soft-skip) within ~1 minute of the first health-check failure, rather than running the full ~6-minute, 4-attempt retry matrix.Related to #49245