Skip to content

Codex app-server stalls after inter-session sessions_send timeout #90673

Closed as not planned

Description

@LibraHo

Summary

OpenClaw 2026.6.1 / Codex app-server can stall after an inter-session sessions_send delegation times out. In the observed incident, the target session had already received a completed tool result, then remained stuck in model_call with no further assistant output or error. The caller then blocked on sessions_history. Other unrelated Codex sessions also stopped making progress.

Environment

  • OpenClaw: 2026.6.1
  • Runtime: OpenAI Codex
  • Model: openai/gpt-5.5
  • Model API: openai-chatgpt-responses
  • Deployment: Docker container on NAS
  • Node runtime in logs: 24.16.0

What happened

An agent delegated a task to another configured agent via sessions_send. The target agent did useful work and completed local tool calls, but after the final bash tool result the Codex app-server did not produce any next assistant item. Later diagnostics showed the target stuck in model_call. The requester also became stuck while waiting on sessions_history.

After this, multiple independent sessions also showed stale active work.

Timeline / Evidence

Times below are Asia/Shanghai on 2026-06-05.

  • 18:18:24 gateway log:

    • subsystem: agent/embedded
    • message: codex dynamic tool call timed out
    • tool: sessions_send
    • timeout: 90000ms
  • Target session trajectory:

    • session key: target agent main session
    • 18:18:19: bash tool result completed successfully after reading trip index
    • 18:18:28: next bash tool call started
    • 18:18:32: bash tool result completed successfully after reading a memory file
    • after that: no assistant output, no error, no final result
  • 18:28:59 diagnostic:

    • requester session stalled
    • classification: blocked_tool_call
    • active tool: sessions_history
    • active tool age: ~622s
  • 18:30:20 diagnostic:

    • target session stalled
    • classification: stalled_agent_run
    • active work: model_call
    • last progress: codex_app_server:request:item/tool/call:response
    • last progress age: ~604s
  • 18:30:20 diagnostic also reported unrelated sessions stalled:

    • another agent session stalled in model_call
    • a cron-backed main session stalled in bash

This suggests the failure is broader than the original sessions_send call and may involve Codex app-server/runtime progress handling.

Expected behavior

  • sessions_send timeout should cleanly fail the caller.
  • If the target run is still active after the caller times out, it should either continue independently or be cancelled/marked failed, but it should not leave the caller blocked on sessions_history.
  • model_call should have a hard timeout or recovery path.
  • Stalled-session diagnostics should be able to abort/recover stuck runs, not only warn.
  • One stuck inter-session delegation should not stall unrelated sessions.

Actual behavior

  • sessions_send timed out.
  • The requester later remained stuck in sessions_history.
  • The target remained stuck in model_call after a completed tool result.
  • Other sessions also stopped making progress.
  • Recovery required gateway/container restart.

Mitigation applied locally

As a temporary workaround:

  • disabled gateway-level sessions_send
  • reduced default run timeout from 86400s to 900s
  • reduced default agent concurrency from 3 to 2
  • avoided cross-agent delegation and handled lookups in the current session directly

Requested fixes

  1. Add/enforce a hard timeout and cancellation path for Codex model_call.
  2. Ensure sessions_send timeout cancels or detaches the target safely and releases the caller.
  3. Avoid synchronous caller blocking on target sessions_history after delegation timeout.
  4. Add recovery/abort behavior to stalled-session detector.
  5. Ensure a stalled run cannot block unrelated sessions in the app-server/runtime dispatcher.

Activity

  1. clawsweeper commented on Jun 5, 2026

    @clawsweeper
    Contributor

    Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed June 21, 2026, 2:27 PM ET / 18:27 UTC.

    Summary
    Keep open: current main has partial Codex idle-timeout and stuck-recovery mitigations, but the reported inter-session timeout, caller release, target recovery, sessions_history behavior, and unrelated-session isolation are not fully proven fixed.

    Reproducibility: no. there is credible source and issue-comment evidence for the remaining failure family, but no high-confidence live current-main reproduction of the full process-wide Codex stall was established.

    Ways to help us reproduce this

    • Add a screenshot or short recording showing the behavior.
    • Add expected vs actual behavior.
    • Share version, platform, channel/provider, and relevant config details.

    Next step
    Manual review is needed because the remaining work spans Codex protocol/runtime behavior, sessions_send timeout semantics, diagnostic recovery policy, and live repro requirements.

    Review details

    Best possible solution:

    Keep the umbrella open until maintainers reconcile the focused timeout, Codex interrupt, and diagnostic-recovery fixes, then verify a live Codex inter-session timeout no longer wedges caller, target, or unrelated sessions.

    Do we have a high-confidence way to reproduce the issue?

    No; there is credible source and issue-comment evidence for the remaining failure family, but no high-confidence live current-main reproduction of the full process-wide Codex stall was established.

    Is this the best way to solve the issue?

    Unclear as one narrow fix; the best path needs maintainer agreement on sessions_send timeout semantics plus Codex interrupt and diagnostic recovery behavior before closure.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • No live current-main Codex inter-session reproduction was run, so process-wide unrelated-session isolation remains unproven end to end.
    • Changing sessions_send timeout behavior from delayed A2A acceptance to cancellation, detachment, or inline compensation is compatibility-sensitive because current code and tests preserve delayed delivery behavior.
    • Open related PRs cover narrower slices and may change the correct repair shape before the umbrella can be closed.

    Codex review notes: model internal, reasoning high; reviewed against bda05dbc2f15.

    Label changes

    Label justifications:

    • P1: The issue describes a shipped Codex runtime stall that blocked agent work across sessions and required gateway or container restart.
    • impact:session-state: The requester, target, and unrelated sessions can remain in stale active-work or recovery states after the timeout.
    • impact:message-loss: The target produced no final assistant output after completed tool results, and a related comment reports an interrupted command result.
    • impact:crash-loop: The reported recovery required restarting the gateway/container and affected process-level session availability.
    Evidence reviewed

    Acceptance criteria:

    • Run a live packaged Codex-backed inter-session sessions_send timeout reproduction covering caller timeout, target continuation or cancellation, sessions_history, and an unrelated concurrent session.
    • Add or confirm focused regression coverage for sessions_send timeout semantics and wait compensation.
    • Verify Codex post-tool/model_call idle timeout and diagnostic recovery abort/drain behavior against current main and the latest shipped release.

    What I checked:

    • Live issue remains open with a fresh related reproduction comment: this issue is still open; the body reports OpenClaw 2026.6.1 Codex app-server stalls after sessions_send, and a June 18 comment reports a related 2026.6.8 sessions_send native Codex tool.call recovery failure.
    • Current sessions_send still treats non-terminal wait timeout as accepted A2A follow-up: sessions_send starts the target run, waits, and on non-terminal timeout starts A2A flow and returns accepted rather than proving cancellation, detachment, or inline compensation for the reported stall. (src/agents/tools/sessions-send-tool.ts:705, bda05dbc2f15)
    • Run-wait helper does not read latest reply after non-ok wait: waitForAgentRunAndReadUpdatedAssistantReply returns immediately for non-ok wait results, so current main still lacks source-level proof that just-after-timeout replies are adopted on this path. (src/agents/run-wait.ts:263, bda05dbc2f15)
    • A2A delayed delivery remains a preserved behavior: The A2A helper waits again and only reads the latest assistant snapshot when that wait is ok; adjacent tests preserve delayed same-session reply delivery, making timeout cancellation/detach semantics a compatibility-sensitive decision. (src/agents/tools/sessions-send-tool.a2a.ts:91, bda05dbc2f15)
    • Codex completion idle timeout mitigation exists: Current main records turn.completion_idle_timeout, logs a Codex app-server completion idle timeout, and aborts with turn_completion_idle_timeout, which mitigates one stall mode but does not prove the full inter-session failure is closed. (extensions/codex/src/app-server/attempt-turn-watches.ts:331, bda05dbc2f15)
    • Codex retry path is intentionally narrow: Codex app-server recovery retries only stdio replay-safe completion failures before assistant/tool/item side effects; it rejects tool_activity and active_item cases, which matches why post-tool stalls remain a separate concern. (src/agents/embedded-agent-runner/run/codex-app-server-recovery.ts:19, bda05dbc2f15)

    Likely related people:

    • vincentkoc: Recent history shows work on sessions_send keying, message delivery evidence, and diagnostic recovery surfaces central to the remaining contract. (role: recent sessions and recovery area contributor; confidence: medium; commits: 0ea08076c3b5, 3659ff8bbf71, 22405223c24a; files: src/agents/tools/sessions-send-tool.ts, src/logging/diagnostic-stuck-session-recovery.runtime.ts)
    • steipete: Recent commits on run wait timeout handling and Codex app-server watch timers touch the wait and timeout behavior involved in this issue. (role: recent run-wait and Codex runtime adjacent contributor; confidence: medium; commits: 53d08d4aefec, 51cceaf70c32, 8d8f5a59e249; files: src/agents/run-wait.ts, extensions/codex/src/app-server/attempt-turn-watches.ts)
    • kevinslin: Recent Codex completion-timeout diagnostic work added part of the current mitigation for app-server completion stalls. (role: recent Codex app-server timeout contributor; confidence: medium; commits: ab7c92282522; files: extensions/codex/src/app-server/attempt-turn-watches.ts)
    • keshavbotagent: Recent Codex app-server completion-stall recovery work is adjacent to the replay-safe retry behavior checked for this issue. (role: adjacent Codex recovery contributor; confidence: medium; commits: 5f89fbe6699e; files: extensions/codex/src/app-server/attempt-turn-watches.ts, src/agents/embedded-agent-runner/run/codex-app-server-recovery.ts)
    • openperf: Recent diagnostic recovery commits handled wedged lanes and stale active-run recovery, directly adjacent to the abort/recovery side of the report. (role: recent diagnostic recovery contributor; confidence: medium; commits: f194be19e261; files: src/logging/diagnostic-stuck-session-recovery.runtime.ts)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
    • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
  2. carter-smalls commented on Jun 5, 2026

    @carter-smalls

    This looks like an OpenAI timeout situation. npx ai-doctor might catch missing timeout guards and unprotected requests.

  3. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:crash-loopCrash, hang, restart loop, or process-level availability failure.
    impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.
    impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    on Jun 15, 2026
  4. 100yenadmin commented on Jun 18, 2026

    @100yenadmin
    Contributor

    Fresh related local reproduction from OpenClaw 2026.6.8 (844f405) on macOS launchd, with active @martian-engineering/lossless-claw@0.13.1.

    Observed from a Telegram-bound agent:main:main session after sending /lossless doctor shortly after a gateway/service repair:

    TOOL ERROR
    OpenClaw recorded a native Codex tool.call without a matching tool.result before the turn completed.
    toolCallId=call_HqRkLKyUVY4X68GZFTdmSc9l
    toolName=sessions_send
    

    Immediately after that, the restored transcript received the system recovery prompt:

    [System] Your previous turn was interrupted by a gateway restart while OpenClaw was waiting on tool/model work. Continue from the existing transcript and finish the interrupted response.
    

    Follow-up state:

    gateway restarted and resumed agent:main:main
    lossless-claw later logged: sanitizeToolUseResultPairing dropped 10 duplicate assistant tool_use block(s)
    current repaired transcript has matching tool call/result counts after sanitization
    

    This looks like the same failure family as this issue: sessions_send can leave the native Codex transcript in a non-pairable tool-call state during timeout/restart recovery. The user-visible symptom was a dropped/interrupted turn rather than a clean command result.

  5. added
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    and removed
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    on Jun 19, 2026
  6. openclaw-barnacle commented on Jul 6, 2026

    @openclaw-barnacle

    This issue has been automatically marked as stale due to inactivity.
    Please add updates or it will be closed.

  7. openclaw-barnacle commented on Jul 14, 2026

    @openclaw-barnacle

    Closing due to inactivity.
    If this is still an issue, please retry on the latest OpenClaw release and share updated details.
    If you are absolutely sure it still happens on the latest release, open a new issue with fresh steps to reproduce.

  8. locked as resolved and limited conversation to collaborators on Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions