Skip to content

[Bug]: Codex app-server session wedges indefinitely after a 429 rate-limit burst — terminal-idle watchdog still gated by activeAppServerTurnRequests > 0 (regression/incomplete fix of #82681) #89742

Closed

Description

@parthjayaram

Bug type

Behavior bug (incorrect state without crash) — session wedge / message loss.

Summary

On 2026.5.28, a Codex app-server session can wedge indefinitely with the agent silently not replying — the same activeAppServerTurnRequests > 0 watchdog-gating mechanism reported and "completed"-closed in #82681 (and related to #84076). The two trigger-agnostic mitigations proposed in #82681 are still not present in 2026.5.28, so the wedge still reproduces.

What's new here vs #82681: #82681 explicitly ruled out rate-limiting as its trigger (its account/rateLimits/updated was a benign periodic notification). This report is the rate-limit-triggered variant — a genuine HTTP 429 burst against codex/gpt-5.5 leaves the shared app-server client unresponsive, after which in-flight turns never settle, the turn counter never returns to zero, and the gated idle watchdogs never fire. Because the client is shared, one poisoned app-server stalls multiple unrelated sessions at once (we observed an interactive Telegram turn and a scheduled cron turn wedged simultaneously, both only released when the client was force-closed by a process restart).

Environment

  • openclaw: 2026.5.28 (e932160), npm i -g openclaw
  • Node: v22.22.2
  • OS: Linux aarch64 (kernel 6.17), systemd user service (openclaw-gateway.service)
  • Agent backend: codex/gpt-5.5 via ACP/codex app-server (OAuth); codex sidecar @openai/codex-linux-arm64 0.134.0
  • agents.defaults.model.fallback: none configured (so a Codex failure is not masked by a fallback)
  • acp.maxConcurrentSessions: 4; shared Codex app-server client across cron + channel sessions
  • Dist file for this build: dist/run-attempt-QNNU1VbX.js; source region extensions/codex/src/app-server/attempt-turn-watches.ts

Symptom (observed)

  1. A burst of HTTP 429 Too Many Requests from codex/gpt-5.5 (145 over ~1h in our logs; embedded run failover decision … reason=rate_limit … next=none).
  2. Shortly after, an interactive Telegram turn is dispatched, emits embedded_run:started, and then makes zero further progress. The bot shows a brief "typing…" then never replies.
  3. Diagnostics: stalled session … state=processing … reason=active_work_without_progress … lastProgress=embedded_run:started lastProgressAge=169s … recovery=none. No idle timeout ever fires.
  4. A cron turn on the same app-server is simultaneously stuck (durationMs=118722 when finally killed) — confirming the wedge is at the shared client / process level, not per-session.
  5. Recovery: only a gateway restart (which force-closes the client → both stuck turns reject with codex app-server client closed before turn completed) or, for the sticky-across-restart case from Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681, gateway call sessions.delete. There is no automatic self-recovery.

Root cause — confirmed from 2026.5.28 source

extensions/codex/src/app-server/attempt-turn-watches.ts:

  • The completion- and terminal-idle watchdogs both bail out when a request is nominally in flight. In 2026.5.28 this guard is present on both the schedule and the fire paths of the terminal-idle watchdog:
function scheduleTerminalIdleWatch() {
  clearTerminalIdleTimer();
  if (params.isCompleted() || params.signal.aborted || !terminalIdleWatchArmed
      || params.getActiveAppServerTurnRequests() > 0) return;   // <-- gate
  ...
}
function fireTerminalIdleTimeout() {
  if (params.isCompleted() || params.isTerminalTurnNotificationQueued() || params.signal.aborted
      || !terminalIdleWatchArmed
      || params.getActiveAppServerTurnRequests() > 0) return;    // <-- gate
  ...
}

Consequence: while a turn request is in flight, terminal-idle is never even scheduled — so it cannot act as the last-resort hard cap precisely when an in-flight request hangs.

const CODEX_TURN_COMPLETION_IDLE_TIMEOUT_MS = 6e4;            // 60s
const CODEX_TURN_ASSISTANT_COMPLETION_IDLE_TIMEOUT_MS = 1e4;  // 10s
const CODEX_TURN_TERMINAL_IDLE_TIMEOUT_MS = 30 * 6e4;         // 30 min
  • turnTerminalIdleTimeoutMs is still not exposed in the config schema (only turnCompletionIdleTimeoutMs is), so operators cannot lower the 30-minute worst-case cap.

Status of #82681's proposed fixes in 2026.5.28

#82681 proposal Present in 2026.5.28?
Expose turnTerminalIdleTimeoutMs + lower the 30-min default No — still 30 * 6e4, still absent from config schema
Drop the activeAppServerTurnRequests > 0 bailout on terminal-idle No — still present on both scheduleTerminalIdleWatch and fireTerminalIdleTimeout
Wrap non-tool-call request handlers in Promise.race Targets a different (hung-handler) path; does not cover this 429-poisoned main-turn path
Restart drops in-flight turn state Partial — restart cleared the run-level hang here, but #82681's sticky-session case persists

Hypothesized (not yet confirmed from source)

  • That the main app-server turn request is dispatched without a hard timeoutMs (the transport applies a timeout only when options.timeoutMs is set). This would explain why a never-responding app-server produces a never-settling await. Supporting observable: the documented ~60s/30-min idle defaults exist, yet nothing fired for 118–169s.
  • That the one un-gated watchdog (attempt-idle) is not armed for the pre-first-notification window, so it can't cover a turn that hangs before producing any output. (armAttemptIdleWatch() appears to have a single, activity-tied call site.)

Proposed fix (in priority order)

  1. Relax the activeAppServerTurnRequests > 0 gate on the terminal-idle watchdog (both scheduleTerminalIdleWatch and fireTerminalIdleTimeout). Terminal-idle is the last-resort hard cap; a hung in-flight request is exactly what it must catch. Because it is idleness-based (resets on any activity notification), relaxing the gate aborts only turns that have been completely silent for the full window — it does not kill long-but-active turns. The resulting abort rejects the pending request → finally runs → the counter resets → the session self-heals without a restart or sessions.delete. This is trigger-agnostic (covers 429 poisoning, hung handlers, app-server desync alike).
  2. Expose turnTerminalIdleTimeoutMs in the config schema and lower the default (e.g. 3–5 min). On its own this is ineffective (the gate suppresses the watchdog regardless of value); combined with (1) it turns a silent multi-minute/30-min hang into prompt, tunable self-recovery.

Optionally, as defense-in-depth: a hard ceiling timeoutMs on the main turn request, and/or arming the un-gated attempt-idle watch for the in-flight pre-output window.

Reproduction

  1. Single Codex app-server (codex/gpt-5.5 via ACP), no model fallback, shared across a chat channel + scheduled jobs.
  2. Drive enough concurrent Codex turns to hit the account's 429 rate limit (e.g. several scheduled jobs firing together).
  3. During/just after the 429 burst, send an interactive message. Observe embedded_run:started with no further progress, recovery=none, and no idle timeout firing.
  4. Confirm a second session (e.g. a cron turn) on the same app-server is also stuck. Only a restart (or sessions.delete for the sticky case) recovers.

Happy to provide full redacted journald traces and to test a patched build.

Activity

  1. clawsweeper commented on Jun 3, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Close: current main arms liveness watches immediately after a Codex turn starts, and a regression test verifies that an accepted turn with no subsequent progress is interrupted and its session released. The reported outbound-turn counter mechanism is not present in current source; that counter tracks app-server-to-OpenClaw request handlers.

    Possible regression — probable (reproduction; failure trace; reviewed change). No predecessor PR is attributed.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the current turn-start liveness and session-release path as the canonical recovery mechanism; reopen only with a current-main trace showing silence beyond the configured watchdog window.

    Do we have a high-confidence way to reproduce the issue?

    Yes, from source: the current integration test starts an accepted turn, sends no progress, and verifies timeout, interruption, and session release. A live 429 burst was not rerun in this read-only review.

    Is this the best way to solve the issue?

    Yes: current main fixes the silent-turn owner boundary directly, rather than relying on a terminal-idle configuration knob to compensate for it.

    Security review:

    Security review: This is an issue review without a proposed patch or security-sensitive report.

    AGENTS.md: found and applied where relevant.

    What I checked:

    Likely related people:

    • Peter Steinberger: Current-main blame assigns the central watchdog implementation and its regression coverage to this refactor commit. (role: recent area contributor; confidence: high; commits: 4eb6aa375f51; files: extensions/codex/src/app-server/attempt-turn-watches.ts, extensions/codex/src/app-server/run-attempt.turn-watches.test.ts)
    • obviyus: The linked lifecycle work and prior liveness commit are directly associated with the shared-client recovery path discussed by this issue. (role: related liveness contributor; confidence: medium; commits: ca3f247eb215; files: extensions/codex/src/app-server/shared-client.ts, extensions/codex/src/app-server/attempt-client-cleanup.ts)

    Codex review notes: model internal, reasoning high; reviewed against 57e862980be1; fix evidence: commit 4eb6aa375f51, main fix timestamp 2026-08-17T05:13:35-07:00.

  2. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.
    impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    on Jun 3, 2026
  3. added
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    and removed
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    on Jun 20, 2026
  4. obviyus commented on Jul 9, 2026

    @obviyus
    Collaborator

    Status after the run-lifecycle stack (PR #102160, commits ca3f247/80b2b709ca8/bb1da73b144/85897598186): the reported failure mode should be structurally retired, pending a live 429 confirmation. A shared Codex app-server client that stops responding is now torn down as a physical-client liveness event — when the terminal-idle backstop fires, retireSharedCodexAppServerClientIfCurrent(client, { failActiveLeases: true }) closes the process and fails every co-lease (extensions/codex/src/app-server/shared-client.ts:428-465), so unrelated sessions leased to the same poisoned client immediately reject and recover without a gateway restart. The per-turn completion-idle watch (60s) already aborts the individual silent turn, and users get a retryable notice with salvaged assistant text preferred.

    On the two literal asks here: the activeAppServerTurnRequests > 0 gate is intentionally retained because that counter tracks inbound server→client requests (approval/elicitation/tool waits where silence is expected), not the outbound turn — a purely 429-silent turn has counter 0 and the watches fire. turnTerminalIdleTimeoutMs remains internal. Keeping this open until someone confirms on a release containing PR #102160; reopen-strength evidence would be journald traces showing a shared client still wedging multiple sessions past the completion-idle window.

  5. removed
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    on Jul 9, 2026
  6. 14 remaining items

  7. added
    issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.
    issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    and removed
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-infoClawSweeper needs more reporter information before it can verify this issue.
    issue-rating: 🦐 gold shrimpDecent issue quality, but reproduction details are still incomplete.
    on Aug 1, 2026
  8. jinngimk-lang commented on Aug 17, 2026

    @jinngimk-lang
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions