Skip to content

[Bug] sessions.list/send gateway timeout after event loop stall (v2026.5.2) #76421

Closed

Description

@ryswork1993

Bug Description

Gateway WebSocket calls (sessions.list, sessions.send, sessions_history, etc.) timeout after 10s when the event loop is blocked by compaction.

Environment

  • Version: OpenClaw 2026.5.2 (8b2a6e5)
  • Node: v22.22.2
  • Host: Linux 6.17, 10GB RAM, 4-core
  • Gateway: ws://127.0.0.1:18902, bind=lan, mode=local
  • Agents configured: 9 agents (main, pm1, director1, frontend1, backend1, qa1, clerk, devops1, reviewer)

Symptoms

  1. sessions.list / sessions_send timeout — Tool calls to sessions_list, sessions_send, sessions_history all fail with gateway timeout after 10000ms
  2. Event loop severely blocked — Log shows repeated liveness warnings with eventLoopDelayP99Ms up to 15000ms+, eventLoopUtilization=1
  3. sessions.list normal response time: ~800ms → spikes to 72,826ms+ during compaction stall, eventually times out
  4. Root cause: Compaction of a large transcript (main session with extensive history) blocks the event loop for 10-15 seconds, causing all WebSocket requests to queue up and timeout

Log Evidence

[2026-05-03T11:06:40.759+08:00] liveness warning: reasons=event_loop_delay,cpu eventLoopDelayP99Ms=1756.4 eventLoopDelayMaxMs=2115 eventLoopUtilization=0.858
[2026-05-03T11:06:54.565+08:00] [tools] sessions_list failed: gateway timeout after 10000ms
[2026-05-03T11:11:56.319+08:00] agent cleanup timed out: runId=... sessionId=...
[2026-05-03T11:18:43.947+08:00] sessions.list 101231ms
[2026-05-03T11:18:49.607+08:00] sessions.usage 89006ms

Current Compaction Config

agents.defaults.compaction.maxActiveTranscriptBytes: "15mb"
agents.defaults.compaction.truncateAfterCompaction: true

Impact

  • Inter-agent communication (main → pm1, main → director1) completely broken during compaction
  • Heartbeat / cron tasks fail when they depend on sessions_list
  • Gateway probe/status commands fail (openclaw status hangs)

Questions / Requests

  1. Can compaction be made non-blocking (run in background thread/worker)?
  2. Is there a way to limit compaction CPU impact so it does not freeze the gateway?
  3. Should sessions.list/usage have their own timeout/queue management separate from compaction stalls?

Activity

  1. clawsweeper commented on May 3, 2026

    @clawsweeper
    Contributor

    Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

    Close: current main and the v2026.6.6 release contain worker-backed compaction planning plus cooperative sessions.list row building, which addresses the reported event-loop starvation and 10s gateway timeout symptom.

    So I’m closing this as already implemented rather than keeping a duplicate issue open.

    Review details

    Best possible solution:

    Keep the shipped worker-backed compaction planning and cooperative sessions.list implementation; use a fresh focused report only if the same stall reproduces on v2026.6.6 or later.

    Do we have a high-confidence way to reproduce the issue?

    Yes for the original shipped-version failure: the report includes concrete v2026.5.2 timeout and liveness logs, and the current source shows the fixed compaction/listing paths. I did not run a live v2026.5.2 workload in this read-only review.

    Is this the best way to solve the issue?

    Yes: moving large compaction planning to a worker and making sessions.list cooperative is the maintainable fix shape; raising gateway call timeouts alone would only mask event-loop starvation.

    Security review:

    Security review: This is an issue cleanup review with no patch or security-sensitive report to inspect.

    AGENTS.md: found and applied where relevant.

    What I checked:

    • Repository policy applied: Root AGENTS.md, VISION.md, and the relevant scoped agent/gateway AGENTS.md files were read; the review applied the repository guidance for source, history, release, and hot-path checks. (AGENTS.md:1, 7e12a3326dd2)
    • Compaction planning moved off the main event loop: The current compaction planning worker runs CPU-heavy planning in a worker thread for histories large enough to risk event-loop starvation, with a 64-message threshold and a bounded worker timeout. (src/agents/compaction-planning-worker.ts:1, 7e12a3326dd2)
    • Compaction code uses worker-backed planning: Current compaction summary chunking and staged summarization call buildSummaryChunksWithWorker and buildStageSplitPlanWithWorker, so large transcript planning no longer stays purely on the gateway event loop. (src/agents/compaction.ts:163, 7e12a3326dd2)
    • Safeguard path uses worker-backed pruning: The compaction safeguard now builds history-prune plans and adaptive chunk ratios through worker-backed helpers, covering the large-history planning path implicated by the report. (src/agents/agent-hooks/compaction-safeguard.ts:1078, 7e12a3326dd2)
    • sessions.list is cooperative on current main: listSessionsFromStoreAsync builds lightweight rows, skips transcript-heavy fallback for bulk rows, and yields with setImmediate every 10 rows so concurrent RPCs and heartbeats are not starved. (src/gateway/session-utils.ts:2803, 7e12a3326dd2)
    • Gateway handler calls the async sessions list path: The sessions.list gateway method now awaits listSessionsFromStoreAsync inside the measured row-building span rather than using a fully synchronous row builder. (src/gateway/server-methods/sessions.ts:977, 7e12a3326dd2)

    Likely related people:

    • Vincent Koc: Current blame for the compaction planning worker, async session-list implementation, responsiveness test, and v2026.6.6 release tag points to this contributor. (role: recent area contributor; confidence: high; commits: 37e3e895b020, 8c802aa68351; files: src/agents/compaction-planning-worker.ts, src/agents/compaction-planning.worker.ts, src/agents/compaction-planning-worker.test.ts)
    • JC: Commit fff1934 recently adjusted session-list row metadata on the same hot path. (role: recent adjacent session-list contributor; confidence: medium; commits: fff193402e8e; files: src/gateway/session-utils.ts)
    • Josh Lehman: Commit ef47dd6 recently added a session accessor seam with a gateway consumer in the same sessions surface. (role: recent adjacent gateway sessions contributor; confidence: medium; commits: ef47dd610c87; files: src/gateway/session-utils.ts, src/gateway/server-methods/sessions.ts)

    Codex review notes: model internal, reasoning high; reviewed against 7e12a3326dd2; fix evidence: release v2026.6.6, commit 8c802aa68351.

  2. barnacle-openclaw commented on May 30, 2026

    @barnacle-openclaw

    This issue has been automatically marked as stale due to inactivity.
    Please add updates or it will be closed.

  3. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:crash-loopCrash, hang, restart loop, or process-level availability failure.
    impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.
    on Jun 14, 2026
  4. locked as resolved and limited conversation to collaborators on Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions