You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Critical action needed: codex engine is failing fleet-wide (~49% success), not tied to one workflow — see Finding 1.
Key findings
1. Codex engine driver_exit collapse — 49.4% success (41/83 runs), HIGH severity, NEW
42/83 codex runs died with driver_exit (engine crashed, not agent logic). Spans multiple models (gpt-5.4, copilot/mai-code-1-flash-picker, copilot/auto) and both branch types (main 42.9%, non-main 56.1%) — this rules out a single-workflow or single-model cause and points at the codex driver itself. Two distinct crash signatures:
Fast 0-turn/0-token startup crash
Runaway context-rebuild loop before crashing — worst case §32708737765 (Terminal Stylist) hit rebuild_factor 42.98x, burning ~5.07M tokens before dying
2. Daily Documentation Diagram — 100% npm registry firewall block + retry storm, MEDIUM severity, NEW
Run §32692387113: all 2,102,274 outbound requests to registry.npmjs.org blocked — the workflow's firewall allowlist only has {defaults, github}. Volume (~1,350 req/s over 26 min) is far beyond a simple missing-domain retry and points to a retry-storm bug with no backoff, compounding the missing-allowlist root cause.
3. Code Scanning Fixer — recurring 0-tok agent_logic failures, MEDIUM severity, RECURRING (since 2026-06-25)
4 more failures today (§32679408985, §32700683740, §32731261007, §32766280367), same signature as the known open issue — still unfixed.
Trend charts (30-day)
Workflow Health — daily success/failure counts and success-rate trend:
Success rate has held in the 78–86% band across the tracked history, with today (81.9%) squarely in that range — this is not a fleet-wide degradation, it's concentrated in codex. Note the 49-day gap (07-06 → 08-24) where no audit ran; it's shaded rather than interpolated since there's no data for that stretch.
Token Usage — daily tokens with 7-day moving average:
Today's ~34.9M tokens is elevated relative to the pre-gap baseline (~15-20M/day), consistent with the codex context-rebuild failures burning multi-million-token runs before crashing — the token-usage anomaly and the codex reliability finding are the same root cause.
Full engine breakdown (excl. intentional failures)
Engine
Runs
Success
Rate
copilot
243
207
85.2%
pi
125
121
96.8%
claude
98
81
82.7%
codex
83
41
49.4%
null
2
1
50.0%
aider
2
2
100%
goose
2
2
100%
crush
1
1
100%
gemini
1
0
0%
opencode
1
1
100%
Repo-memory update — blocked by sandbox environment
/tmp/gh-aw/repo-memory/default/ mounted read-only in this run despite the standard read/write contract (confirmed via git status showing a clean, correctly-checked-out memory/audit-workflows branch, and push_repo_memory reporting 0 bytes of patch diff after all write attempts failed with Read-only file system). Only /tmp/gh-aw/agent/ and /tmp/gh-aw/cache-memory/ were writable this run. This is the same class of sandbox quirk observed earlier in this run for /tmp/gh-aw/python/{data,charts}/ (also nominally part of a read/write mount but host-side read-only) — not something a workaround should attempt to bypass.
Net effect: today's findings could not be persisted to known-issues.json, recommendations.json, workflow-trends.json, anomalies.json, audit-history.jsonl, or metrics-summary.json. The three findings above (codex collapse, npm firewall block, code-scanning-fixer recurrence) are reported here only — please re-run the audit tomorrow to pick these up in repo memory if the mount issue resolves itself.
Next actions
File an upstream codex driver issue covering both crash modes (0-turn startup crash; unbounded context-rebuild before driver_exit). Consider a temporary rebuild_factor circuit-breaker to bound token burn until fixed.
Add registry.npmjs.org to the Daily Documentation Diagram firewall allowlist, and cap/backoff the retry loop so a blocked domain fails fast instead of retrying at ~1,350 req/s.
Re-investigate Code Scanning Fixer's 0-tok agent_logic failures — open since 2026-06-25, still unfixed, 4 more occurrences today.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Overview
24h audit window: 2026-08-23T21:19Z → 2026-08-24T21:04Z, 560 total runs (558 excl. 2 intentional credit-limit tests).
main(scheduled/prod): 77.4% (233/301) vs non-main(PR): 87.2% (224/257) — the gap is driven almost entirely by one engine (see below)driver_exitfailures, 11agent_logicfailures fleet-wideCritical action needed: codex engine is failing fleet-wide (~49% success), not tied to one workflow — see Finding 1.
Key findings
1. Codex engine driver_exit collapse — 49.4% success (41/83 runs), HIGH severity, NEW
42/83 codex runs died with
driver_exit(engine crashed, not agent logic). Spans multiple models (gpt-5.4,copilot/mai-code-1-flash-picker,copilot/auto) and both branch types (main42.9%, non-main56.1%) — this rules out a single-workflow or single-model cause and points at the codex driver itself. Two distinct crash signatures:rebuild_factor42.98x, burning ~5.07M tokens before dying2. Daily Documentation Diagram — 100% npm registry firewall block + retry storm, MEDIUM severity, NEW
Run §32692387113: all 2,102,274 outbound requests to
registry.npmjs.orgblocked — the workflow's firewall allowlist only has{defaults, github}. Volume (~1,350 req/s over 26 min) is far beyond a simple missing-domain retry and points to a retry-storm bug with no backoff, compounding the missing-allowlist root cause.3. Code Scanning Fixer — recurring 0-tok agent_logic failures, MEDIUM severity, RECURRING (since 2026-06-25)
4 more failures today (§32679408985, §32700683740, §32731261007, §32766280367), same signature as the known open issue — still unfixed.
Trend charts (30-day)
Workflow Health — daily success/failure counts and success-rate trend:
Success rate has held in the 78–86% band across the tracked history, with today (81.9%) squarely in that range — this is not a fleet-wide degradation, it's concentrated in codex. Note the 49-day gap (07-06 → 08-24) where no audit ran; it's shaded rather than interpolated since there's no data for that stretch.
Token Usage — daily tokens with 7-day moving average:
Today's ~34.9M tokens is elevated relative to the pre-gap baseline (~15-20M/day), consistent with the codex context-rebuild failures burning multi-million-token runs before crashing — the token-usage anomaly and the codex reliability finding are the same root cause.
Full engine breakdown (excl. intentional failures)
Repo-memory update — blocked by sandbox environment
/tmp/gh-aw/repo-memory/default/mounted read-only in this run despite the standard read/write contract (confirmed viagit statusshowing a clean, correctly-checked-outmemory/audit-workflowsbranch, andpush_repo_memoryreporting 0 bytes of patch diff after all write attempts failed withRead-only file system). Only/tmp/gh-aw/agent/and/tmp/gh-aw/cache-memory/were writable this run. This is the same class of sandbox quirk observed earlier in this run for/tmp/gh-aw/python/{data,charts}/(also nominally part of a read/write mount but host-side read-only) — not something a workaround should attempt to bypass.Net effect: today's findings could not be persisted to
known-issues.json,recommendations.json,workflow-trends.json,anomalies.json,audit-history.jsonl, ormetrics-summary.json. The three findings above (codex collapse, npm firewall block, code-scanning-fixer recurrence) are reported here only — please re-run the audit tomorrow to pick these up in repo memory if the mount issue resolves itself.Next actions
driver_exit). Consider a temporaryrebuild_factorcircuit-breaker to bound token burn until fixed.registry.npmjs.orgto the Daily Documentation Diagram firewall allowlist, and cap/backoff the retry loop so a blocked domain fails fast instead of retrying at ~1,350 req/s.References:
All reactions