Summary
A long-running Codex session completed clean source-unit intake for a 396-page controlled scanned standard, then spent extreme compute validating a false strict-closeout premise.
The primary defect was in our local project code: locator/reference occurrences were projected as physical tables or continuations, and object-level holds spread to reference-only pages. Codex did not originate that classifier bug. It materially amplified the incident because the run lacked token-versus-outcome visibility, repeated-input/result detection, a no-progress budget, and an automatic context-pressure handoff.
Session/thread ID: 019f6643-b589-75d3-ae45-7da1f5550351
Measured impact
A source-safe streaming audit of rollout metadata (no prompt, message, tool-body, licensed text, or private DB content retained) found:
- 5,345,347,029 cumulative token-events
- 98.29% cached-input ratio
- 191 context compactions
- approximately 3.98B token-events after clean-source intake was already 371/371 complete
- hottest six-hour zero-progress window: 248,568,594 token-events and 1,558 tool calls
- other zero-progress six-hour windows: roughly 196M–213M token-events
“Token-events” are usage metadata including cached traffic; this is not asserted as billed usage or cost.
Reliability near-miss
The first corrective terminal closeout was also stopped before commit: it proved that all 34 held pages had audit coverage but did not bind every page to explicit physical table/continuation dispositions. It mechanically zeroed genuine continuations. A later adversarial review also proved that a hand-shaped, hash-recomputed candidate could bypass the disposition evidence.
The corrected local contract now:
- requires exact per-page machine-readable dispositions
- derives relationship-review scope dynamically
- rebuilds the entire candidate from immutable evidence at validation/acceptance/replay and requires byte equality
- binds every page-decision hash, hierarchy review, raw hold-set count/hash, and continuation relationship review
- rejects forged table, continuation, hierarchy, and hold-ID mutations even after recomputing candidate hashes/IDs
- stops before a third expensive deterministic cycle unless an input, method, evidence item, or hypothesis changes
The corrected run ultimately reached 396/396 with the source DB unchanged.
Product requests
- Show cumulative total/cached/fresh/output/reasoning tokens plus tokens since the last task-owned outcome change.
- Detect identical normalized tool inputs with unchanged output/state hashes; after two expensive unchanged cycles, require an explicit changed-condition statement.
- Provide a configurable no-progress budget based on wall time, token traffic, tool calls, and task-owned counters.
- Warn when cached-input share is high while completion counters and state hashes are unchanged.
- Automatically checkpoint and propose a fresh-agent handoff after sustained context pressure or repeated compactions, verifying takeover.
- Reuse identical deterministic results inside a stable-input epoch, or at minimum flag the duplicate before another expensive orchestration cycle.
- Add evals for false-premise loops: reference mention vs physical target, adjacent citations vs continuation, and internally consistent gates built on a bad classification.
- Let
/feedback attach a source-safe incident summary while associating backend telemetry by session ID.
Full public-safe report and local corrective design:
https://github.com/Demigodofa/codex-agent-ops/blob/d808559/knowledge/codex_long_run_no_progress_incident_2026-07-29.md
Please use the session ID for server-side telemetry. The raw ~937MB rollout should not be requested publicly because it may contain private tool output and controlled-source evidence.
Summary
A long-running Codex session completed clean source-unit intake for a 396-page controlled scanned standard, then spent extreme compute validating a false strict-closeout premise.
The primary defect was in our local project code: locator/reference occurrences were projected as physical tables or continuations, and object-level holds spread to reference-only pages. Codex did not originate that classifier bug. It materially amplified the incident because the run lacked token-versus-outcome visibility, repeated-input/result detection, a no-progress budget, and an automatic context-pressure handoff.
Session/thread ID:
019f6643-b589-75d3-ae45-7da1f5550351Measured impact
A source-safe streaming audit of rollout metadata (no prompt, message, tool-body, licensed text, or private DB content retained) found:
“Token-events” are usage metadata including cached traffic; this is not asserted as billed usage or cost.
Reliability near-miss
The first corrective terminal closeout was also stopped before commit: it proved that all 34 held pages had audit coverage but did not bind every page to explicit physical table/continuation dispositions. It mechanically zeroed genuine continuations. A later adversarial review also proved that a hand-shaped, hash-recomputed candidate could bypass the disposition evidence.
The corrected local contract now:
The corrected run ultimately reached 396/396 with the source DB unchanged.
Product requests
/feedbackattach a source-safe incident summary while associating backend telemetry by session ID.Full public-safe report and local corrective design:
https://github.com/Demigodofa/codex-agent-ops/blob/d808559/knowledge/codex_long_run_no_progress_incident_2026-07-29.md
Please use the session ID for server-side telemetry. The raw ~937MB rollout should not be requested publicly because it may contain private tool output and controlled-source evidence.