[aw-failures] [aw] Failure Report 2026-05-05: APM Unpack Failures + Smoke Gemini Network Blocked

[github-actions]

Executive Summary

Investigation of the last 6 hours (2026-05-05 ~19:30–01:17 UTC) found 2 distinct failure clusters across 4 runs out of 29 total. A fifth run (Smoke CI §25351381782) was cancelled due to push supersession and is not actionable.

Critical (P0): APM unpack failures are blocking 3 runs across 2 different PRs, preventing agents from ever launching. This is systemic — present on both firewall v0.25.35 and v0.25.38. Separate P1: Smoke Gemini fails because the Gemini CLI can't reach localhost:8080 (MCP gateway) or the Gemini API via play.googleapis.com, resulting in 94% firewall block rate and exhausted retries.

Failure Clusters

Priority	Cluster	Affected Runs	Workflows	Root Cause
P0	APM Unpack Failure	3	Matt Pocock Skills Reviewer (×2), Smoke Claude	apm unpack failed for bundle 1 of 1 — exit code 1 on apm-default.tar.gz
P1	Gemini API Connectivity	1	Smoke Gemini	TypeError: fetch failed — MCP gateway at localhost:8080 blocked (267/301 requests blocked)
N/A	Smoke CI Cancelled	1	Smoke CI	Superseded by newer push to main; next run succeeded immediately

Evidence

Cluster A: APM Unpack Failures (P0) — 3 runs

Affected runs:

Run	Workflow	Branch	Firewall
§25350390860	Matt Pocock Skills Reviewer	copilot/fix-ai-moderator-workflow-allowlist	v0.25.35
§25351475330	Smoke Claude	copilot/bump-firewall-to-v0-25-38-and-mcpg-to-v0-3-6	v0.25.38
§25351482890	Matt Pocock Skills Reviewer	copilot/bump-firewall-to-v0-25-38-and-mcpg-to-v0-3-6	v0.25.38

Consistent error (all 3 runs):

##[error]APM action failed: apm unpack failed for bundle 1 of 1
  (path: /tmp/gh-aw/apm-bundles/apm-default.tar.gz, exit code: 1)

Failure point: agent job → step Restore APM packages (all bundles). APM download succeeds (bundles present: 3–78 KB), but unpack exits non-zero.

Audit-diff evidence (Smoke Claude vs Smoke Codex on same PR):

• Smoke Claude: 0 network requests — agent never activated
• Smoke Codex (success): 16+ API calls, 8 MCP tool invocations, api.openai.com:443 allowed
• Duration delta: −3m39s confirming early failure, not a slow run

Scope: Two distinct PRs + two distinct firewall versions rule out the firewall bump as the sole cause.

Cluster B: Smoke Gemini — API Connectivity Failure (P1)

Affected run: §25351475292 — Smoke Gemini on copilot/bump-firewall-to-v0-25-38-and-mcpg-to-v0-3-6

Repeated error (4 retries before termination):

Attempt N failed. Retrying with backoff...
Error: exception TypeError: fetch failed sending request
  at Models.generateContent → NumericalClassifierStrategy.route

Firewall analysis:

• 301 total requests; 283 blocked (94% block rate)
• localhost:8080 — 267 blocked (MCP gateway)
• 172.30.0.30:10003 — 15 blocked (MCP sidecar/proxy)
• play.googleapis.com:443 — 18 allowed (Gemini API attempts that still fail upstream)

Audit-diff vs Smoke Pi (success, same PR):

• 3 new anomaly domains in Gemini run vs Pi: localhost:8080, 172.30.0.30:10003, (unknown) — all newly blocked
• Pi had api-proxy:10000 and api-proxy:10002 blocked (expected internal proxy traffic) — absent in Gemini
• Duration: +3m3s longer than Pi (Gemini exhausted 4 retries before giving up)
• GitHub API rate limit usage: +627% vs Pi (retry amplification)

Root cause: The Gemini model router (NumericalClassifierStrategy) cannot complete its API classification call; MCP tools are inaccessible because the MCP gateway is blocked. This is likely a firewall allowlist gap for the Gemini engine setup.

Existing Issue Correlation

⚠️ GitHub API unavailable in this environment (HTTP 403) — existing open issue correlation could not be performed. Sub-issue created for the P0 APM cluster pending deduplication review.

Proposed Fix Roadmap

Priority	Fix	Area
P0	Investigate APM unpack failure — check bundle integrity, APM action version, and filesystem permissions	APM / Platform
P1	Add localhost:8080 and 172.30.0.30:10003 to Gemini allowed domains in firewall config	Firewall / Gemini engine
N/A	Smoke CI cancellation expected (superseded push)	—

Sub-issues Created

• [aw-failures] [aw] Fix: APM unpack fails for apm-default.tar.gz (exit code 1) — 3 workflows blocked #30252 — APM unpack failures (P0 fix sub-issue)

References:

• §25350390860 · §25351475330 · §25351475292