Skip to content

[Bug]: Feishu WebSocket silently disconnects — reconnect gives up after single retry (related to #68766) #91899

Closed

Description

@richardataxai-lab

Summary

Feishu WebSocket connection silently drops and never recovers. The auto-reconnect mechanism gives up after a single retry attempt, causing hours of silent message loss with no visible error.

This is the same underlying issue as #68766 (which is locked), but I wanted to provide additional data points from a different environment.

Environment

  • OpenClaw Version: 2026.6.5
  • OS: macOS Darwin 25.2.0 (arm64)
  • Node Version: 25.9.0
  • Gateway Mode: Local (loopback, port 18789)
  • Feishu Connection Mode: WebSocket (长连接)

Timeline (2026-06-10)

  1. 08:15 — User reported Feishu not responding. Gateway restarted, Feishu recovered.
  2. 10:07 — Health check cron detected Feishu was down, triggered automatic gateway restart.
  3. 10:13 — Last Feishu message received successfully.
  4. 10:13–13:23 — WebSocket silently dropped again. No error logs. openclaw status --deep reported Feishu: OK.
  5. 13:23 — User manually reported the issue via DingTalk. Gateway restarted again.
  6. ~3 hours of silent message loss in a single day.

Key Problems

1. Reconnect logic gives up too easily

The auto-reconnect mechanism (auto-restart attempt 1/10) fires once, and if that single retry fails (likely because the upstream issue is still ongoing), it stops entirely. There is no exponential backoff with persistent retries.

2. openclaw status cannot detect dead WebSocket

Both openclaw status and openclaw status --deep report Feishu as "OK" even when the WebSocket connection is completely dead. The status check only verifies that the gateway process is running and the channel is configured — it does NOT verify that the WebSocket connection is actually alive (e.g., via ping/pong).

3. No error logging or alerting

When the WebSocket drops, there are zero log entries. No [error], no [warn], nothing. The failure is completely silent.

Workaround

Added a cron health check that monitors the Feishu session's lastActive timestamp via sessions_list. If no Feishu messages have been received for >2 hours (or >4 hours during 00:00-08:00), the gateway is automatically restarted.

This is a band-aid, not a fix. The root cause needs to be addressed in the Feishu plugin's reconnection logic.

Suggested Fixes

  1. Exponential backoff with persistent retries: 1s → 2s → 4s → ... → max 5min, retry indefinitely until reconnected
  2. WebSocket liveness check: Periodic ping/pong to detect dead connections
  3. Status detection: openclaw status should verify WebSocket is actually alive, not just configured
  4. Auto-restart on repeated failure: After N consecutive reconnection failures, restart the gateway automatically
  5. Clear logging: Log [feishu] WebSocket disconnected and [feishu] reconnected events

Impact

This is a critical reliability issue for any production Feishu deployment. Silent message loss for hours is unacceptable.


Related: #68766

Activity

  1. clawsweeper commented on Jun 10, 2026

    @clawsweeper
    Contributor

    Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 4, 2026, 2:14 AM ET / 06:14 UTC.

    Summary
    A prior repair fixed retry exhaustion and SDK-signalled false-healthy states, but current Feishu WebSocket mode still has no recorded transport-activity fact for a callback-free stall; the gateway consequently keeps that account healthy and does not restart it.

    Reproducibility: yes. from source: a connected Feishu WebSocket without lastTransportActivityAt is evaluated healthy, while the WebSocket monitor deliberately never records that fact. Live confirmation of a callback-free SDK stall still requires the pinned SDK contract and Feishu credentials.

    Root-cause cluster
    Relationship: canonical
    Canonical: #91899
    Summary: This issue is the canonical remaining callback-free WebSocket-liveness gap; prior merged work fixed related retry exhaustion and SDK-signalled health reporting.

    Members:

    Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

    Maintainer decision needed

    Question Recommendation
    Which documented SDK liveness signal can safely populate Feishu lastTransportActivityAt for WebSocket mode? Verify an SDK liveness signal: Inspect the pinned SDK contract and wire its actual heartbeat or pong signal into the plugin-owned status sink with focused regression coverage.

    Why: Current source proves the Gateway needs an authoritative transport fact, but the available checkout cannot establish whether the pinned SDK exposes a heartbeat or pong signal that supplies one.

    Next step
    The source-proven gap needs the pinned SDK liveness contract before a safe plugin repair can be scoped.

    Review details

    Best possible solution:

    Use an SDK-supported heartbeat, pong, or equivalent authoritative WebSocket liveness signal in the Feishu plugin to refresh lastTransportActivityAt, then prove the Gateway reports and recovers a callback-free stall without penalizing an idle healthy channel.

    Do we have a high-confidence way to reproduce the issue?

    Yes, from source: a connected Feishu WebSocket without lastTransportActivityAt is evaluated healthy, while the WebSocket monitor deliberately never records that fact. Live confirmation of a callback-free SDK stall still requires the pinned SDK contract and Feishu credentials.

    Is this the best way to solve the issue?

    No; persistent retries and lifecycle status solve SDK-signalled failures, but cannot detect a connection that emits neither callback nor transport fact. The plugin should record an authoritative liveness signal rather than infer it from application messages or startup.

    AGENTS.md: found and applied where relevant.

    Remaining risk / open question:

    • The exact @larksuiteoapi/node-sdk heartbeat/callback contract could not be inspected: local dependency source is absent and network DNS was unavailable. A repair must not treat startup or ordinary messages as transport activity, because that would restart quiet healthy channels.

    Codex review notes: model internal, reasoning high; reviewed against 3f2fb7e36589.

    Label changes

    Label justifications:

    • P1: A callback-free Feishu transport stall can still suppress inbound delivery until an operator intervenes.
    • impact:message-loss: The remaining behavior is silent loss of incoming Feishu messages.
    Evidence reviewed

    What I checked:

    Likely related people:

    • vincentkoc: Authored the commit that added monitor-owned recovery after SDK reconnect exhaustion. (role: introduced retry-exhaustion recovery; confidence: high; commits: f9b47ad2a169; files: extensions/feishu/src/monitor.transport.ts, extensions/feishu/src/monitor.cleanup.test.ts)
    • Acache: Authored the merged Feishu account-status repair that made SDK-signalled disconnects visible to Gateway health. (role: introduced transport-status reporting; confidence: high; commits: e7d6566b8fd1; files: extensions/feishu/src/channel.ts, extensions/feishu/src/monitor.transport.ts)
    How this review workflow works
    • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
    • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
    • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
    • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
    • Maintainers can also comment @clawsweeper review to request a fresh review only.
    • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
    • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
    • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
  2. added
    P1High-priority user-facing bug, regression, or broken workflow.
    impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    and removed
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.
    clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.
    on Jun 10, 2026
  3. 8 remaining items

  4. added
    issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.
    clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.
    and removed
    clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.
    clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.
    issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.
    on Aug 2, 2026
  5. self-assigned this
    on Aug 12, 2026
  6. vincentkoc commented on Aug 12, 2026

    @vincentkoc
    Member

    Closing as already fixed.

    #103763 landed as a0c4c31a179b523108e91506f00033fb61d5b79a, configuring the SDK's pingTimeout watchdog. The Feishu SDK heartbeat now owns socket liveness: after each ping, no pong or inbound frame within the timeout terminates the dead socket and enters the normal reconnect flow.

    #90966 landed as e7d6566b8fd19b25467301087fc5b37e6979eda3, publishing Feishu WebSocket lifecycle status, and #73739 landed as f9b47ad2a169442c09f062372bee5f9276b1a5fa, retaining monitor-owned client recreation after SDK reconnect exhaustion.

    Current main retains the watchdog, status reporting, and client recreation paths. The first release containing the complete fix chain is https://github.com/openclaw/openclaw/releases/tag/v2026.7.2-beta.1, published July 15, 2026.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:not-repro-on-mainClawSweeper found high-confidence evidence that this issue no longer reproduces on main.close:already-fixedclaw-marshal labelimpact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions