[aw-failures] AWF firewall image v0.25.47 broken: missing oidc-token-provider-base module

[github-actions]

Summary

• Window: last 6h ending 2026-05-16T01:20Z
• Failures in window: 10 (9 in a single PR cluster, 1 scheduled run already fixed)
• Root cause (cluster A): AWF firewall image v0.25.47 ships with a broken Node bundle — oidc-token-provider.js requires ./oidc-token-provider-base which is not present in /app/, so awf-api-proxy crashes at startup and the agent is never invoked.
• Status: PR Bump default AWF to v0.25.47 and MCP Gateway to v0.3.10 #32503 (the version bump that exposed this) was closed without merging at 2026-05-16T01:02:41Z, so main is not currently affected. The smoke gate worked as designed.
• Cluster B (Daily Model Inventory Checker): 1 pre-fix failure already addressed by commit f91a078 (PR Stabilize Daily Model Inventory Copilot startup by removing fragile quoted jq allow-tool #32505), merged at 2026-05-16T01:04:37Z — after the failed run started at 00:10Z.

Failure clusters

Cluster	Workflows affected	Runs	Trigger SHA	Existing tracking
A — AWF v0.25.47 MODULE_NOT_FOUND	Smoke OTEL, Smoke OTEL Backends, Agent Container Smoke Test, Smoke Pi, Smoke Gemini, Changeset Generator, Smoke Codex, Smoke Claude, Smoke Copilot	9	4be2b6b9 (PR #32503, closed)	#32508 #32509 #32510 #32511 #32512 #32513
B — Daily Model Inventory Copilot startup crash	Daily Model Inventory Checker	1	890db390 (main, scheduled)	Fixed by f91a078 (#32505)

Evidence (cluster A)

All 9 cluster-A runs share head_sha=4be2b6b9852ffe06efc8f59c2c5bdf1b5db12668 from branch copilot/bump-firewall-mcpg-versions (PR #32503, "Bump default AWF to v0.25.47 and MCP Gateway to v0.3.10"). The failure occurs before the agent runs — awf-api-proxy exits non-zero on startup with the same Node module-resolution error in every workflow.

Representative stack trace (Smoke OTEL, [§25948486975](https://github.com/github/gh-aw/actions/runs/25948486975))

Error: Cannot find module './oidc-token-provider-base'
Require stack:
- /app/oidc-token-provider.js
- /app/providers/openai.js
- /app/providers/index.js
- /app/server.js
    at Function._resolveFilename (node:internal/modules/cjs/loader:1430:15)
    ...
    at Object.<anonymous> (/app/oidc-token-provider.js:22:5)
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/app/oidc-token-provider.js',
    '/app/providers/openai.js',
    '/app/providers/index.js',
    '/app/server.js'
  ]

[ERROR] Fatal error: Error: AWF firewall failed to start: awf-api-proxy failed to start on both attempts. The agent was never invoked.

All 9 failed runs in cluster A

Workflow	Run
Smoke OTEL	§25948486975
Smoke OTEL Backends	§25948486951
Agent Container Smoke Test	§25948486948
Smoke Pi	§25948486928
Smoke Gemini	§25948486919
Changeset Generator	§25948486910
Smoke Codex	§25948486977
Smoke Claude	§25948486949
Smoke Copilot	§25948486944

Note: Smoke Pi, Smoke Codex, and Changeset Generator do not have individual auto-generated tracking issues; they share the same root cause as the linked sub-issues.

Evidence (cluster B)

Run §25947409109 (Daily Model Inventory Checker, scheduled, on main). The Copilot CLI spawned via copilot_harness.cjs exited with code 1 after ~1s, producing zero stdout/stderr — a silent startup crash. The harness flagged: no output produced — not retrying (possible causes: binary not found, permission denied, auth failure, or silent startup crash).

The spawn command included a deeply quoted --allow-tool 'shell(jq ".endpoints[] | select(.provider == \"copilot\") | .models" /tmp/gh-aw/model-inventory/reflect.json)', which matches the fix landed in commit f91a078 ("Stabilize Daily Model Inventory Copilot startup by removing fragile quoted jq allow-tool (#32505)"). The fix merged at 2026-05-16T01:04:37Z, after the 00:10Z failed run. No subsequent Daily Model Inventory run exists in the window to confirm the fix end-to-end, but the symptom matches exactly.

Existing issue correlation

• The per-workflow tracking issues [aw] Smoke OTEL failed #32508–[aw] Smoke Claude failed #32513 each report the same Cannot find module './oidc-token-provider-base' error against PR Bump default AWF to v0.25.47 and MCP Gateway to v0.3.10 #32503's run IDs. They are all symptoms of this single upstream image bug. They are linked below as sub-issues.
• No prior open agentic-workflows issue describes the AWF v0.25.47 bundle bug as an upstream/release problem — only per-workflow symptom reports.
• Cluster B has no separate tracking issue; the fix has already landed.

Proposed fix roadmap

P0 — Block ✅ already in place: the smoke gate caught the bad bump, PR #32503 closed without merging. No action.

P1 — Upstream fix needed before retrying v0.25.47:

• File or escalate an upstream issue against the AWF firewall image build pipeline: the oidc-token-provider-base.js (or its compiled counterpart) is missing from /app/ in the published v0.25.47 container image. The require call at /app/oidc-token-provider.js:22 resolves to nothing.
• Until upstream resolves, do not re-attempt the bump to v0.25.47. If a future PR retries the bump without upstream proof of fix, expect the same cluster of smoke failures.

P2 — Verification:

• Confirm that the next scheduled Daily Model Inventory Checker run (post f91a078) succeeds on main. If it fails again with zero stdout/stderr, re-open cluster B as a separate investigation.

Sub-issues linked

The 6 auto-generated per-workflow failure issues are linked as sub-issues of this report. They are kept open so each affected workflow's per-engine surface (copilot, claude, gemini) remains visible; closing is deferred to maintainers once the upstream image is confirmed fixed.

Confidence and unknowns

• High confidence that all 9 cluster-A failures share the same root cause (identical stack trace, identical SHA, identical container image).
• High confidence that the failure is in the upstream image, not in gh-aw code (the missing file is inside /app/ of awf-api-proxy, which is the published AWF v0.25.47 container).
• Medium confidence that cluster B is fully resolved by f91a078 — symptom matches the fix description but no post-fix run exists yet to verify.
• Unknown: whether the upstream image bug has already been reported externally.

References:

• §25948486975 — Smoke OTEL
• §25947409109 — Daily Model Inventory Checker
• PR Bump default AWF to v0.25.47 and MCP Gateway to v0.3.10 #32503 (closed), PR Stabilize Daily Model Inventory Copilot startup by removing fragile quoted jq allow-tool #32505 (f91a078, merged)

Generated by 🔍 [aw] Failure Investigator (6h) · ● 12M · ◷

• expires on May 23, 2026, 1:30 AM UTC