fix(session): preserve codex's real error when systemError precedes the terminal#694
Merged
Merged
Conversation
…he terminal
thread/status/changed→systemError used to synthesize the terminal
immediately with the opaque "codex reported a system error", so the rich
follow-ups codex sends milliseconds later — error{willRetry:false} and
turn/completed{failed, turn.error.message} — were absorbed by the
terminated guard and the real cause (e.g. "failed to load AWS
credentials", a gateway 403) never reached the user, who only saw
UNKNOWN_UPSTREAM_ERROR.
Defer systemError like the existing idle deferral (system_error_pending):
emit nothing and let the fatal error or the authoritative turn/completed
produce the terminal carrying the real message. Two bounded fallbacks
keep the FSM from hanging Running if the follow-up is missing:
SYSTEM_ERROR_GRACE (2s, armed only after codex declared the thread
fatally errored — not a mid-turn watchdog) and the EOF flush (process
death) which emits the previous opaque terminal.
Evidence (per AGENTS.md approved sources):
- schema: SystemErrorThreadStatus carries only `type` (codex 0.145.0
app-server generate-json-schema, v2/ThreadStatusChangedNotification.json)
- live capture (0.145.0, bedrock provider, empty AWS credential chain;
archived at ~/aion/protocols/samples/codex-cli/0.145.0/
systemerror_bedrock_no_creds.jsonl): systemError →
error{willRetry:false} same ms → turn/completed{failed} +5ms,
both carrying the rich message
- real user session (rollout 019f1215-…, 2026-07-27): 6/6 failed turns
recorded task_complete with the full error while the UI showed only
the opaque code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the opaque
UNKNOWN_UPSTREAM_ERROR / "codex reported a system error"that hides the real cause of a failed codex turn.On
thread/status/changed → {type:"systemError"}the reader used to synthesize a terminal immediately with hard-coded text and setterminated— so the rich follow-ups arriving milliseconds later (error{willRetry:false}andturn/completed{failed, turn.error.message}) were absorbed by the terminated guard and dropped. The UI then showed only the generic code while the actual error (e.g.failed to load AWS credentials, a gateway 403) was thrown away.This PR defers
systemErrorexactly like the existingidledeferral: setsystem_error_pending, emit nothing, and let the fatalerror{willRetry:false}or the authoritativeturn/completedproduce the terminal carrying the real message.Closes #693
Why the old "synthesize immediately" assumption was wrong
The #609 protocol-audit comment said codex "does NOT reliably follow systemError with a turn/completed", so the terminal had to be synthesized on the spot. Evidence says otherwise:
019f1215-96fd-7003-a5e2-192c410b924a, 2026-07-27): the conversation failed 6 times, and all 6 of 6 turns recorded atask_completecarrying the full error message —stream disconnected before completion: failed to load AWS credentials: an error occurred while loading credentials(and, on the first turn,unexpected status 403 Forbidden: user not allowed to access model …). The user saw onlyUNKNOWN_UPSTREAM_ERRORfor every one of them and spent the session suspecting CA certificates.app-server, bedrock provider, empty AWS default credential chain; archived at~/aion/protocols/samples/codex-cli/0.145.0/systemerror_bedrock_no_creds.jsonl, probe script alongside):app-server generate-json-schema,v2/ThreadStatusChangedNotification.json):SystemErrorThreadStatuscarries onlytype— the status can never hold the detail itself, so deferring to the follow-up is the only way to surface the real cause.Safety: the FSM still cannot hang Running
Two bounded fallbacks preserve the original guarantee for a hypothetical unfollowed
systemError:SYSTEM_ERROR_GRACE(2 s): a deadline on the reader's next-line wait, armed strictly after codex has declared the thread fatally errored. This is not a mid-turn watchdog (per the no-auto-timeout design) — it cannot false-kill a healthy turn, and intervening frames do not extend it.systemErroris flushed as the previous opaque error terminal.Tests (TDD — both new behavior tests watched fail first with the opaque text)
codex_system_error_then_turn_completed_preserves_rich_error— deferred systemError resolved byturn/completed, terminal keeps the rich message, exactly one terminal.codex_system_error_then_fatal_error_preserves_rich_error— resolved byerror{willRetry:false}(in-flight), rich message wins.codex_system_error_with_no_followup_times_out_with_generic_terminal— still-open stream, no follow-up: grace timer emits the generic terminal (gated segments keep stdout open so this exercises the timer, not the EOF flush; runs ≈2 s).codex_system_error_status_synthesizes_terminal(existing) — now covers the EOF-flush path, assertion unchanged.cargo fmt --check,cargo clippy -p aionui-session -- -D warnings,cargo test -p aionui-session(387 passed),cargo nextest run --workspaceall green.