Repository navigation
Description
Summary
Feishu WebSocket connection silently drops and never recovers. The auto-reconnect mechanism gives up after a single retry attempt, causing hours of silent message loss with no visible error.
This is the same underlying issue as #68766 (which is locked), but I wanted to provide additional data points from a different environment.
Environment
- OpenClaw Version: 2026.6.5
- OS: macOS Darwin 25.2.0 (arm64)
- Node Version: 25.9.0
- Gateway Mode: Local (loopback, port 18789)
- Feishu Connection Mode: WebSocket (长连接)
Timeline (2026-06-10)
- 08:15 — User reported Feishu not responding. Gateway restarted, Feishu recovered.
- 10:07 — Health check cron detected Feishu was down, triggered automatic gateway restart.
- 10:13 — Last Feishu message received successfully.
- 10:13–13:23 — WebSocket silently dropped again. No error logs.
openclaw status --deepreported Feishu: OK. - 13:23 — User manually reported the issue via DingTalk. Gateway restarted again.
- ~3 hours of silent message loss in a single day.
Key Problems
1. Reconnect logic gives up too easily
The auto-reconnect mechanism (auto-restart attempt 1/10) fires once, and if that single retry fails (likely because the upstream issue is still ongoing), it stops entirely. There is no exponential backoff with persistent retries.
2. openclaw status cannot detect dead WebSocket
Both openclaw status and openclaw status --deep report Feishu as "OK" even when the WebSocket connection is completely dead. The status check only verifies that the gateway process is running and the channel is configured — it does NOT verify that the WebSocket connection is actually alive (e.g., via ping/pong).
3. No error logging or alerting
When the WebSocket drops, there are zero log entries. No [error], no [warn], nothing. The failure is completely silent.
Workaround
Added a cron health check that monitors the Feishu session's lastActive timestamp via sessions_list. If no Feishu messages have been received for >2 hours (or >4 hours during 00:00-08:00), the gateway is automatically restarted.
This is a band-aid, not a fix. The root cause needs to be addressed in the Feishu plugin's reconnection logic.
Suggested Fixes
- Exponential backoff with persistent retries: 1s → 2s → 4s → ... → max 5min, retry indefinitely until reconnected
- WebSocket liveness check: Periodic ping/pong to detect dead connections
- Status detection:
openclaw statusshould verify WebSocket is actually alive, not just configured - Auto-restart on repeated failure: After N consecutive reconnection failures, restart the gateway automatically
- Clear logging: Log
[feishu] WebSocket disconnectedand[feishu] reconnectedevents
Impact
This is a critical reliability issue for any production Feishu deployment. Silent message loss for hours is unacceptable.
Related: #68766
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Activity
clawsweeper commentedon Jun 10, 2026
Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 4, 2026, 2:14 AM ET / 06:14 UTC.
Summary
A prior repair fixed retry exhaustion and SDK-signalled false-healthy states, but current Feishu WebSocket mode still has no recorded transport-activity fact for a callback-free stall; the gateway consequently keeps that account healthy and does not restart it.
Reproducibility: yes. from source: a connected Feishu WebSocket without lastTransportActivityAt is evaluated healthy, while the WebSocket monitor deliberately never records that fact. Live confirmation of a callback-free SDK stall still requires the pinned SDK contract and Feishu credentials.
Root-cause cluster
Relationship:
canonicalCanonical: #91899
Summary: This issue is the canonical remaining callback-free WebSocket-liveness gap; prior merged work fixed related retry exhaustion and SDK-signalled health reporting.
Members:
partial_overlap: fix(feishu): report transport activity (connected/lastEventAt) for health monitor #90966 - Merged status reporting makes SDK-signalled reconnects visible but does not create transport activity for a callback-free stall.partial_overlap: [Bug]: Feishu WebSocket connection does not recover after transient token refresh failure #68766 - Its SDK retry-exhaustion mechanism is now handled by the monitor retry loop, while this issue retains the distinct no-callback liveness gap.Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.
Maintainer decision needed
Why: Current source proves the Gateway needs an authoritative transport fact, but the available checkout cannot establish whether the pinned SDK exposes a heartbeat or pong signal that supplies one.
Next step
The source-proven gap needs the pinned SDK liveness contract before a safe plugin repair can be scoped.
Review details
Best possible solution:
Use an SDK-supported heartbeat, pong, or equivalent authoritative WebSocket liveness signal in the Feishu plugin to refresh lastTransportActivityAt, then prove the Gateway reports and recovers a callback-free stall without penalizing an idle healthy channel.
Do we have a high-confidence way to reproduce the issue?
Yes, from source: a connected Feishu WebSocket without lastTransportActivityAt is evaluated healthy, while the WebSocket monitor deliberately never records that fact. Live confirmation of a callback-free SDK stall still requires the pinned SDK contract and Feishu credentials.
Is this the best way to solve the issue?
No; persistent retries and lifecycle status solve SDK-signalled failures, but cannot detect a connection that emits neither callback nor transport fact. The plugin should record an authoritative liveness signal rather than infer it from application messages or startup.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning high; reviewed against 3f2fb7e36589.
Label changes
Label justifications:
P1: A callback-free Feishu transport stall can still suppress inbound delivery until an operator intervenes.impact:message-loss: The remaining behavior is silent loss of incoming Feishu messages.Evidence reviewed
What I checked:
extensions/feishu/src/monitor.transport.ts:234, 3f2fb7e36589)extensions/feishu/src/monitor.cleanup.test.ts:198, 3f2fb7e36589)extensions/feishu/src/monitor.transport.ts:256, 3f2fb7e36589)src/gateway/channel-health-policy.ts:174, 3f2fb7e36589)extensions/feishu/src/monitor.transport.ts:234, f9b47ad2a169)docs/releases/2026.7.1.md. (docs/releases/2026.7.1.md:499, e7d6566b8fd1)Likely related people:
extensions/feishu/src/monitor.transport.ts,extensions/feishu/src/monitor.cleanup.test.ts)extensions/feishu/src/channel.ts,extensions/feishu/src/monitor.transport.ts)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper autofix,@clawsweeper automerge,@clawsweeper fix ci, or@clawsweeper address review.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.8 remaining items
vincentkoc commentedon Aug 12, 2026
Closing as already fixed.
#103763 landed as
a0c4c31a179b523108e91506f00033fb61d5b79a, configuring the SDK'spingTimeoutwatchdog. The Feishu SDK heartbeat now owns socket liveness: after each ping, no pong or inbound frame within the timeout terminates the dead socket and enters the normal reconnect flow.#90966 landed as
e7d6566b8fd19b25467301087fc5b37e6979eda3, publishing Feishu WebSocket lifecycle status, and #73739 landed asf9b47ad2a169442c09f062372bee5f9276b1a5fa, retaining monitor-owned client recreation after SDK reconnect exhaustion.Current
mainretains the watchdog, status reporting, and client recreation paths. The first release containing the complete fix chain is https://github.com/openclaw/openclaw/releases/tag/v2026.7.2-beta.1, published July 15, 2026.