Repository navigation
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Bug Description
Gateway WebSocket calls (sessions.list, sessions.send, sessions_history, etc.) timeout after 10s when the event loop is blocked by compaction.
Environment
- Version: OpenClaw 2026.5.2 (8b2a6e5)
- Node: v22.22.2
- Host: Linux 6.17, 10GB RAM, 4-core
- Gateway: ws://127.0.0.1:18902, bind=lan, mode=local
- Agents configured: 9 agents (main, pm1, director1, frontend1, backend1, qa1, clerk, devops1, reviewer)
Symptoms
- sessions.list / sessions_send timeout — Tool calls to
sessions_list,sessions_send,sessions_historyall fail withgateway timeout after 10000ms - Event loop severely blocked — Log shows repeated liveness warnings with
eventLoopDelayP99Msup to 15000ms+, eventLoopUtilization=1 - sessions.list normal response time: ~800ms → spikes to 72,826ms+ during compaction stall, eventually times out
- Root cause: Compaction of a large transcript (main session with extensive history) blocks the event loop for 10-15 seconds, causing all WebSocket requests to queue up and timeout
Log Evidence
[2026-05-03T11:06:40.759+08:00] liveness warning: reasons=event_loop_delay,cpu eventLoopDelayP99Ms=1756.4 eventLoopDelayMaxMs=2115 eventLoopUtilization=0.858
[2026-05-03T11:06:54.565+08:00] [tools] sessions_list failed: gateway timeout after 10000ms
[2026-05-03T11:11:56.319+08:00] agent cleanup timed out: runId=... sessionId=...
[2026-05-03T11:18:43.947+08:00] sessions.list 101231ms
[2026-05-03T11:18:49.607+08:00] sessions.usage 89006ms
Current Compaction Config
agents.defaults.compaction.maxActiveTranscriptBytes: "15mb"
agents.defaults.compaction.truncateAfterCompaction: trueImpact
- Inter-agent communication (main → pm1, main → director1) completely broken during compaction
- Heartbeat / cron tasks fail when they depend on sessions_list
- Gateway probe/status commands fail (openclaw status hangs)
Questions / Requests
- Can compaction be made non-blocking (run in background thread/worker)?
- Is there a way to limit compaction CPU impact so it does not freeze the gateway?
- Should sessions.list/usage have their own timeout/queue management separate from compaction stalls?
Reacted by clawsweeper
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Activity
clawsweeper commentedon May 3, 2026
Thanks for the context here. I did a careful shell check against current
main, and this is already implemented.Close: current main and the v2026.6.6 release contain worker-backed compaction planning plus cooperative
sessions.listrow building, which addresses the reported event-loop starvation and 10s gateway timeout symptom.So I’m closing this as already implemented rather than keeping a duplicate issue open.
Review details
Best possible solution:
Keep the shipped worker-backed compaction planning and cooperative
sessions.listimplementation; use a fresh focused report only if the same stall reproduces on v2026.6.6 or later.Do we have a high-confidence way to reproduce the issue?
Yes for the original shipped-version failure: the report includes concrete v2026.5.2 timeout and liveness logs, and the current source shows the fixed compaction/listing paths. I did not run a live v2026.5.2 workload in this read-only review.
Is this the best way to solve the issue?
Yes: moving large compaction planning to a worker and making
sessions.listcooperative is the maintainable fix shape; raising gateway call timeouts alone would only mask event-loop starvation.Security review:
Security review: This is an issue cleanup review with no patch or security-sensitive report to inspect.
AGENTS.md: found and applied where relevant.
What I checked:
AGENTS.md:1, 7e12a3326dd2)src/agents/compaction-planning-worker.ts:1, 7e12a3326dd2)buildSummaryChunksWithWorkerandbuildStageSplitPlanWithWorker, so large transcript planning no longer stays purely on the gateway event loop. (src/agents/compaction.ts:163, 7e12a3326dd2)src/agents/agent-hooks/compaction-safeguard.ts:1078, 7e12a3326dd2)listSessionsFromStoreAsyncbuilds lightweight rows, skips transcript-heavy fallback for bulk rows, and yields withsetImmediateevery 10 rows so concurrent RPCs and heartbeats are not starved. (src/gateway/session-utils.ts:2803, 7e12a3326dd2)sessions.listgateway method now awaitslistSessionsFromStoreAsyncinside the measured row-building span rather than using a fully synchronous row builder. (src/gateway/server-methods/sessions.ts:977, 7e12a3326dd2)Likely related people:
src/agents/compaction-planning-worker.ts,src/agents/compaction-planning.worker.ts,src/agents/compaction-planning-worker.test.ts)src/gateway/session-utils.ts)src/gateway/session-utils.ts,src/gateway/server-methods/sessions.ts)Codex review notes: model internal, reasoning high; reviewed against 7e12a3326dd2; fix evidence: release v2026.6.6, commit 8c802aa68351.
barnacle-openclaw commentedon May 30, 2026
This issue has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.