Repository navigation
Description
Bug type
Behavior bug (incorrect state without crash) — session wedge / message loss.
Summary
On 2026.5.28, a Codex app-server session can wedge indefinitely with the agent silently not replying — the same activeAppServerTurnRequests > 0 watchdog-gating mechanism reported and "completed"-closed in #82681 (and related to #84076). The two trigger-agnostic mitigations proposed in #82681 are still not present in 2026.5.28, so the wedge still reproduces.
What's new here vs #82681: #82681 explicitly ruled out rate-limiting as its trigger (its account/rateLimits/updated was a benign periodic notification). This report is the rate-limit-triggered variant — a genuine HTTP 429 burst against codex/gpt-5.5 leaves the shared app-server client unresponsive, after which in-flight turns never settle, the turn counter never returns to zero, and the gated idle watchdogs never fire. Because the client is shared, one poisoned app-server stalls multiple unrelated sessions at once (we observed an interactive Telegram turn and a scheduled cron turn wedged simultaneously, both only released when the client was force-closed by a process restart).
Environment
- openclaw: 2026.5.28 (e932160),
npm i -g openclaw - Node: v22.22.2
- OS: Linux aarch64 (kernel 6.17), systemd user service (
openclaw-gateway.service) - Agent backend:
codex/gpt-5.5via ACP/codex app-server (OAuth); codex sidecar@openai/codex-linux-arm640.134.0 agents.defaults.model.fallback: none configured (so a Codex failure is not masked by a fallback)acp.maxConcurrentSessions: 4; shared Codex app-server client across cron + channel sessions- Dist file for this build:
dist/run-attempt-QNNU1VbX.js; source regionextensions/codex/src/app-server/attempt-turn-watches.ts
Symptom (observed)
- A burst of HTTP 429
Too Many Requestsfromcodex/gpt-5.5(145 over ~1h in our logs;embedded run failover decision … reason=rate_limit … next=none). - Shortly after, an interactive Telegram turn is dispatched, emits
embedded_run:started, and then makes zero further progress. The bot shows a brief "typing…" then never replies. - Diagnostics:
stalled session … state=processing … reason=active_work_without_progress … lastProgress=embedded_run:started lastProgressAge=169s … recovery=none. No idle timeout ever fires. - A cron turn on the same app-server is simultaneously stuck (
durationMs=118722when finally killed) — confirming the wedge is at the shared client / process level, not per-session. - Recovery: only a gateway restart (which force-closes the client → both stuck turns reject with
codex app-server client closed before turn completed) or, for the sticky-across-restart case from Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681,gateway call sessions.delete. There is no automatic self-recovery.
Root cause — confirmed from 2026.5.28 source
extensions/codex/src/app-server/attempt-turn-watches.ts:
- The completion- and terminal-idle watchdogs both bail out when a request is nominally in flight. In 2026.5.28 this guard is present on both the schedule and the fire paths of the terminal-idle watchdog:
function scheduleTerminalIdleWatch() {
clearTerminalIdleTimer();
if (params.isCompleted() || params.signal.aborted || !terminalIdleWatchArmed
|| params.getActiveAppServerTurnRequests() > 0) return; // <-- gate
...
}
function fireTerminalIdleTimeout() {
if (params.isCompleted() || params.isTerminalTurnNotificationQueued() || params.signal.aborted
|| !terminalIdleWatchArmed
|| params.getActiveAppServerTurnRequests() > 0) return; // <-- gate
...
}Consequence: while a turn request is in flight, terminal-idle is never even scheduled — so it cannot act as the last-resort hard cap precisely when an in-flight request hangs.
-
activeAppServerTurnRequestsis incremented around the app-server turn request and decremented in afinally. That balances correctly only if the awaited request settles. When the app-server stops responding (post-429), the awaited promise never settles →finallynever runs → the counter is stuck> 0→ all the counter-gated watchdogs are suppressed forever. -
Idle-timeout constants are still hardcoded (unchanged from Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681):
const CODEX_TURN_COMPLETION_IDLE_TIMEOUT_MS = 6e4; // 60s
const CODEX_TURN_ASSISTANT_COMPLETION_IDLE_TIMEOUT_MS = 1e4; // 10s
const CODEX_TURN_TERMINAL_IDLE_TIMEOUT_MS = 30 * 6e4; // 30 minturnTerminalIdleTimeoutMsis still not exposed in the config schema (onlyturnCompletionIdleTimeoutMsis), so operators cannot lower the 30-minute worst-case cap.
Status of #82681's proposed fixes in 2026.5.28
| #82681 proposal | Present in 2026.5.28? |
|---|---|
Expose turnTerminalIdleTimeoutMs + lower the 30-min default |
No — still 30 * 6e4, still absent from config schema |
Drop the activeAppServerTurnRequests > 0 bailout on terminal-idle |
No — still present on both scheduleTerminalIdleWatch and fireTerminalIdleTimeout |
Wrap non-tool-call request handlers in Promise.race |
Targets a different (hung-handler) path; does not cover this 429-poisoned main-turn path |
| Restart drops in-flight turn state | Partial — restart cleared the run-level hang here, but #82681's sticky-session case persists |
Hypothesized (not yet confirmed from source)
- That the main app-server turn request is dispatched without a hard
timeoutMs(the transport applies a timeout only whenoptions.timeoutMsis set). This would explain why a never-responding app-server produces a never-settling await. Supporting observable: the documented ~60s/30-min idle defaults exist, yet nothing fired for 118–169s. - That the one un-gated watchdog (attempt-idle) is not armed for the pre-first-notification window, so it can't cover a turn that hangs before producing any output. (
armAttemptIdleWatch()appears to have a single, activity-tied call site.)
Proposed fix (in priority order)
- Relax the
activeAppServerTurnRequests > 0gate on the terminal-idle watchdog (bothscheduleTerminalIdleWatchandfireTerminalIdleTimeout). Terminal-idle is the last-resort hard cap; a hung in-flight request is exactly what it must catch. Because it is idleness-based (resets on any activity notification), relaxing the gate aborts only turns that have been completely silent for the full window — it does not kill long-but-active turns. The resulting abort rejects the pending request →finallyruns → the counter resets → the session self-heals without a restart orsessions.delete. This is trigger-agnostic (covers 429 poisoning, hung handlers, app-server desync alike). - Expose
turnTerminalIdleTimeoutMsin the config schema and lower the default (e.g. 3–5 min). On its own this is ineffective (the gate suppresses the watchdog regardless of value); combined with (1) it turns a silent multi-minute/30-min hang into prompt, tunable self-recovery.
Optionally, as defense-in-depth: a hard ceiling timeoutMs on the main turn request, and/or arming the un-gated attempt-idle watch for the in-flight pre-output window.
Reproduction
- Single Codex app-server (
codex/gpt-5.5via ACP), no model fallback, shared across a chat channel + scheduled jobs. - Drive enough concurrent Codex turns to hit the account's 429 rate limit (e.g. several scheduled jobs firing together).
- During/just after the 429 burst, send an interactive message. Observe
embedded_run:startedwith no further progress,recovery=none, and no idle timeout firing. - Confirm a second session (e.g. a cron turn) on the same app-server is also stuck. Only a restart (or
sessions.deletefor the sticky case) recovers.
Happy to provide full redacted journald traces and to test a patched build.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
clawsweeper commentedon Jun 3, 2026
Thanks for the context here. I did a careful shell check against current
main, and this is already implemented.Close: current main arms liveness watches immediately after a Codex turn starts, and a regression test verifies that an accepted turn with no subsequent progress is interrupted and its session released. The reported outbound-turn counter mechanism is not present in current source; that counter tracks app-server-to-OpenClaw request handlers.
Possible regression — probable (reproduction; failure trace; reviewed change). No predecessor PR is attributed.
So I’m closing this as already implemented rather than keeping a duplicate issue open.
Review details
Best possible solution:
Keep the current turn-start liveness and session-release path as the canonical recovery mechanism; reopen only with a current-main trace showing silence beyond the configured watchdog window.
Do we have a high-confidence way to reproduce the issue?
Yes, from source: the current integration test starts an accepted turn, sends no progress, and verifies timeout, interruption, and session release. A live 429 burst was not rerun in this read-only review.
Is this the best way to solve the issue?
Yes: current main fixes the silent-turn owner boundary directly, rather than relying on a terminal-idle configuration knob to compensate for it.
Security review:
Security review: This is an issue review without a proposed patch or security-sensitive report.
AGENTS.md: found and applied where relevant.
What I checked:
extensions/codex/src/app-server/run-attempt-active-turn.ts:175, 4eb6aa375f51)extensions/codex/src/app-server/run-attempt-server-requests.ts:80, 4eb6aa375f51)extensions/codex/src/app-server/shared-client.ts:1200, 4eb6aa375f51)extensions/codex/src/app-server/attempt-turn-watches.ts:344, 4eb6aa375f51)Likely related people:
extensions/codex/src/app-server/attempt-turn-watches.ts,extensions/codex/src/app-server/run-attempt.turn-watches.test.ts)extensions/codex/src/app-server/shared-client.ts,extensions/codex/src/app-server/attempt-client-cleanup.ts)Codex review notes: model internal, reasoning high; reviewed against 57e862980be1; fix evidence: commit 4eb6aa375f51, main fix timestamp 2026-08-17T05:13:35-07:00.
obviyus commentedon Jul 9, 2026
Status after the run-lifecycle stack (PR #102160, commits ca3f247/80b2b709ca8/bb1da73b144/85897598186): the reported failure mode should be structurally retired, pending a live 429 confirmation. A shared Codex app-server client that stops responding is now torn down as a physical-client liveness event — when the terminal-idle backstop fires,
retireSharedCodexAppServerClientIfCurrent(client, { failActiveLeases: true })closes the process and fails every co-lease (extensions/codex/src/app-server/shared-client.ts:428-465), so unrelated sessions leased to the same poisoned client immediately reject and recover without a gateway restart. The per-turn completion-idle watch (60s) already aborts the individual silent turn, and users get a retryable notice with salvaged assistant text preferred.On the two literal asks here: the
activeAppServerTurnRequests > 0gate is intentionally retained because that counter tracks inbound server→client requests (approval/elicitation/tool waits where silence is expected), not the outbound turn — a purely 429-silent turn has counter 0 and the watches fire.turnTerminalIdleTimeoutMsremains internal. Keeping this open until someone confirms on a release containing PR #102160; reopen-strength evidence would be journald traces showing a shared client still wedging multiple sessions past the completion-idle window.14 remaining items
jinngimk-lang commentedon Aug 17, 2026