Skip to content

fix(session): preserve codex's real error when systemError precedes the terminal#694

Merged
kaizhou-lab merged 1 commit into
mainfrom
fix/codex-systemerror-preserve-detail
Jul 27, 2026
Merged

fix(session): preserve codex's real error when systemError precedes the terminal#694
kaizhou-lab merged 1 commit into
mainfrom
fix/codex-systemerror-preserve-detail

Conversation

@kaizhou-lab

Copy link
Copy Markdown
Contributor

Summary

Fixes the opaque UNKNOWN_UPSTREAM_ERROR / "codex reported a system error" that hides the real cause of a failed codex turn.

On thread/status/changed → {type:"systemError"} the reader used to synthesize a terminal immediately with hard-coded text and set terminated — so the rich follow-ups arriving milliseconds later (error{willRetry:false} and turn/completed{failed, turn.error.message}) were absorbed by the terminated guard and dropped. The UI then showed only the generic code while the actual error (e.g. failed to load AWS credentials, a gateway 403) was thrown away.

This PR defers systemError exactly like the existing idle deferral: set system_error_pending, emit nothing, and let the fatal error{willRetry:false} or the authoritative turn/completed produce the terminal carrying the real message.

Closes #693

Why the old "synthesize immediately" assumption was wrong

The #609 protocol-audit comment said codex "does NOT reliably follow systemError with a turn/completed", so the terminal had to be synthesized on the spot. Evidence says otherwise:

  1. Real user session (codex 0.145.0, rollout 019f1215-96fd-7003-a5e2-192c410b924a, 2026-07-27): the conversation failed 6 times, and all 6 of 6 turns recorded a task_complete carrying the full error messagestream disconnected before completion: failed to load AWS credentials: an error occurred while loading credentials (and, on the first turn, unexpected status 403 Forbidden: user not allowed to access model …). The user saw only UNKNOWN_UPSTREAM_ERROR for every one of them and spent the session suspecting CA certificates.
  2. Live wire capture (codex 0.145.0 app-server, bedrock provider, empty AWS default credential chain; archived at ~/aion/protocols/samples/codex-cli/0.145.0/systemerror_bedrock_no_creds.jsonl, probe script alongside):
    error{willRetry:true}   ×5  "Reconnecting... N/5"          (transient retries)
    thread/status/changed       {"type":"systemError"}
    error{willRetry:false}      message = <rich cause>          (same millisecond)
    turn/completed              status=failed, turn.error.message = <rich cause>  (+5 ms)
    
  3. Schema (codex 0.145.0 app-server generate-json-schema, v2/ThreadStatusChangedNotification.json): SystemErrorThreadStatus carries only type — the status can never hold the detail itself, so deferring to the follow-up is the only way to surface the real cause.

Safety: the FSM still cannot hang Running

Two bounded fallbacks preserve the original guarantee for a hypothetical unfollowed systemError:

  • SYSTEM_ERROR_GRACE (2 s): a deadline on the reader's next-line wait, armed strictly after codex has declared the thread fatally errored. This is not a mid-turn watchdog (per the no-auto-timeout design) — it cannot false-kill a healthy turn, and intervening frames do not extend it.
  • EOF flush: if the process dies before any follow-up, the deferred systemError is flushed as the previous opaque error terminal.

Tests (TDD — both new behavior tests watched fail first with the opaque text)

  • codex_system_error_then_turn_completed_preserves_rich_error — deferred systemError resolved by turn/completed, terminal keeps the rich message, exactly one terminal.
  • codex_system_error_then_fatal_error_preserves_rich_error — resolved by error{willRetry:false} (in-flight), rich message wins.
  • codex_system_error_with_no_followup_times_out_with_generic_terminal — still-open stream, no follow-up: grace timer emits the generic terminal (gated segments keep stdout open so this exercises the timer, not the EOF flush; runs ≈2 s).
  • codex_system_error_status_synthesizes_terminal (existing) — now covers the EOF-flush path, assertion unchanged.

cargo fmt --check, cargo clippy -p aionui-session -- -D warnings, cargo test -p aionui-session (387 passed), cargo nextest run --workspace all green.

…he terminal

thread/status/changed→systemError used to synthesize the terminal
immediately with the opaque "codex reported a system error", so the rich
follow-ups codex sends milliseconds later — error{willRetry:false} and
turn/completed{failed, turn.error.message} — were absorbed by the
terminated guard and the real cause (e.g. "failed to load AWS
credentials", a gateway 403) never reached the user, who only saw
UNKNOWN_UPSTREAM_ERROR.

Defer systemError like the existing idle deferral (system_error_pending):
emit nothing and let the fatal error or the authoritative turn/completed
produce the terminal carrying the real message. Two bounded fallbacks
keep the FSM from hanging Running if the follow-up is missing:
SYSTEM_ERROR_GRACE (2s, armed only after codex declared the thread
fatally errored — not a mid-turn watchdog) and the EOF flush (process
death) which emits the previous opaque terminal.

Evidence (per AGENTS.md approved sources):
- schema: SystemErrorThreadStatus carries only `type` (codex 0.145.0
  app-server generate-json-schema, v2/ThreadStatusChangedNotification.json)
- live capture (0.145.0, bedrock provider, empty AWS credential chain;
  archived at ~/aion/protocols/samples/codex-cli/0.145.0/
  systemerror_bedrock_no_creds.jsonl): systemError →
  error{willRetry:false} same ms → turn/completed{failed} +5ms,
  both carrying the rich message
- real user session (rollout 019f1215-…, 2026-07-27): 6/6 failed turns
  recorded task_complete with the full error while the UI showed only
  the opaque code
@kaizhou-lab
kaizhou-lab merged commit 994e3fe into main Jul 27, 2026
6 checks passed
@kaizhou-lab
kaizhou-lab deleted the fix/codex-systemerror-preserve-detail branch July 27, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codex systemError masks the real turn error as opaque UNKNOWN_UPSTREAM_ERROR

1 participant