feat(threads): opt in to recovery after terminal stream failures

[coygeek]

Summary

Add environment-wide, bounded recovery for confirmed provider stream failures after the provider has exhausted its own retries. Keep the recovery decision outside the agent model so a failed main turn can recover without a person returning to send a continuation.

Problem to solve

A provider can terminate a turn while the conversation remains reusable. The agent cannot schedule its own next turn after its stream has stopped. Completed delegated results can remain available while the parent stays idle indefinitely, especially when the last child completion already triggered the failed parent turn.

In one inspected non-security workflow on T3 Code 0.0.46-nightly.20261010.2908 with Codex CLI 0.162.1, five retries carried responseStreamDisconnected and a retryable transport classification. The terminal error became other with the same stream-disconnection message, and the main run stayed idle for approximately five and a half hours. A user follow-up started a successful new turn on the same conversation. The request used a custom local Responses endpoint; retained records do not establish whether the upstream service or intermediary closed the streams. Cyber safety-buffering notices appeared during the successful later turn, so they do not establish that a classifier caused the original stop.

Proposed behavior

• Provide a preference scoped to the selected environment that defaults to off and applies across projects. Recovery runs on the server and does not require an attached client or a live supervisory model.
• Initially recover only confirmed, recoverable native Codex stream failures on reusable conversations. Record the supported boundary accurately, including custom routes when their structured provider events meet the same criteria. Preserve existing behavior for providers without sufficient failure and continuation capabilities.
• Wait for terminal failure and exhaustion of provider-owned retries. Retain the failure chain for the exact run and provider turn so an uninformative final code cannot erase preceding structured transport evidence. Neither a generic other code alone nor message-text matching qualifies a run.
• Schedule a finite number of delayed attempts with a documented total budget. Continue the existing conversation and unfinished authorized work; do not replay the original prompt, switch models or accounts, or duplicate completed tool actions.
• Evaluate the same eligibility rules for failed main turns and app-owned delegated work where the existing thread and task lifecycle permits continuation. A recovery must preserve task ownership and result-delivery semantics.
• Make pending recovery and its reason observable through existing thread records and supported environment operations. Stop, disabling the preference, newer work, completion, settling, snoozing, archival or deletion, and pending approvals or questions take precedence.
• Treat explicit policy errors as requiring review. Safety-buffering notices are informational and never trigger retries. Unknown or unsupported failures retain their existing error and manual continuation path.

Acceptance criteria

1. The preference defaults to off for fresh and existing environments, persists per environment, and is reversible. With it off, no recovery turns start. With it on, eligible threads across multiple projects use the same server-owned behavior without a connected client.
2. A controlled native Codex fixture reproduces exhausted stream retries followed by terminal failure and a reusable conversation. After the configured delay, recovery continues that conversation exactly once for that failed run and preserves saved work and delegated results.
3. Structured transport retries followed by an unchanged terminal stream failure whose code becomes other remain distinguishable from an unrelated unknown failure. A changed failure, explicit policy error, or missing authoritative evidence prevents recovery.
4. No continuation starts while provider retries or another turn remain active. Client-only disconnects, quiet tools, successful turns, and safety-buffering notifications create no recovery attempts.
5. Duplicate terminal events, multiple clients, concurrent recovery paths, and server restart cannot create overlapping or duplicate continuation dispatches. The attempt budget survives the chosen restart behavior and cannot reset into an unbounded loop.
6. Stop, disabling recovery, a newer user instruction, completion, settling, snoozing, archive or delete, and pending approvals or questions invalidate a pending attempt before dispatch. Recovery never unsnoozes a thread, bypasses approval, or overrides user intent.
7. Cover root turns and app-owned delegated task lifecycle with focused tests. Recovering a task must not publish a premature terminal result or create an unrelated conversation. Unsupported task forms stay stopped with an accurate explanation.
8. Authentication, usage or rate limits, context errors, overload, explicit cyber or misalignment policy errors, and arbitrary task failures stay outside this stream-recovery policy. No provider or account fallback retries a blocked workflow.
9. Focused integration tests observe persisted events and drain the effect worker to prove failure, delayed eligibility, continuation, cancellation, and budget exhaustion. State the fixture's simulation limits; do not claim a real upstream classifier failure was reproduced.

Affected area

Environment preferences, provider failure classification, server-owned continuation scheduling, and root and delegated run lifecycle. Existing thread records and environment operations should expose the outcome consistently to local and remote clients.

Non-goals

Automatic continuation after policy refusals, guardrail bypass, recovery inferred from inactivity, prompt replay, model or account fallback, changes to live user databases, general retries across every provider, and capacity or usage-limit recovery.

Alternatives considered

A small external watchdog could prototype the same deterministic rules but would require separate authentication and durable state. A scheduled supervisory agent adds model usage and can itself fail. Codex lifecycle hooks can help with normal completion checks, but fatal stream-error coverage is not established. Bounded subagents and saved checkpoints reduce repeated work without restarting a failed main agent.

Supporting context

The accepted connection-loss proposal defines a narrower initial boundary: native Codex on the built-in OpenAI connection. This proposal requests direction on the current V2 lifecycle, preserved error-chain evidence, custom routes, and root and delegated task ownership. The separate capacity-recovery discussion covers overload and is outside this proposal's scope. Neither related item establishes approval of this broader direction.

OpenAI's app-server documentation exposes terminal turn errors. Its recovery guidance distinguishes bounded transient retries from policy errors and requires checking completed actions before retrying. The proposal preserves that distinction.

This is a feature proposal for maintainer direction and scope. Posting it does not establish maintainer approval of an implementation.

Triage assessment

• Impact level: not-applicable
• Assessment status: not-applicable
• Impact basis: This proposes a new opt-in recovery workflow; the existing runtime does not promise automatic recovery after terminal stream retries.
• Workaround status: not-applicable
• Workaround basis: Feature impact is not applicable. In the inspected incident, a manual follow-up resumed the existing conversation, but that does not provide unattended recovery.

[coygeek]

Implementation proposal: PR #17902, currently at 31ce98a918. It links this discussion as its source and implements the environment-wide opt-in for confirmed Codex stream retry exhaustion, with durable finite attempts, user-intent cancellation, and preserved delegated-task identity.

The current independent behavior run passed 87 controlled cases: 49 native protocol and persistence cases, 24 delegated lifecycle cases, and 14 effect-worker cases. Scoped server type checking, targeted lint, and cumulative AutoReview loop 6 passed. The three earlier inline findings remain resolved. Terminal cleanup now stays durable after dispatch faults without extending the continuation budget, and cancellation of an absent target thread is accepted as a typed, exact-target no-op. Regressions verify retained failure evidence, lease and Stop precedence, lost phase transitions, original task identity, one acknowledged result wake, and continued retries for transient or unrelated read failures. The fixtures simulate transport and command failures; they do not reproduce an upstream classifier incident.

CodeRabbit approved the current head, 31ce98a918. Its latest summary reports no new actionable comments and minimal merge risk; all four inline review threads are resolved. Workflow approval and explicit maintainer scope approval remain separate outstanding items.

The five Actions runs on the prior head, 9c64a0f8ad, awaited maintainer workflow approval before any jobs started. The evidence and approval request are posted on the PR. Local acceptance does not establish upstream CI or preview success.

Maintainer direction requested: the PR description review requires explicit approval of the proposal's direction and scope before merging. Please confirm whether the default-off, two-attempt Codex recovery described above is acceptable, including qualifying custom endpoints and tasks delegated through T3, or identify the scope changes required. No approval is implied by posting this proposal or its implementation.