Skip to content

fix(host-agent): daemon heartbeat + watchdog to recover stale channel subscription - #56

Merged
Pher217 merged 1 commit into
mainfrom
claude/2026-06-14-daemon-heartbeat-durability
Jun 15, 2026
Merged

fix(host-agent): daemon heartbeat + watchdog to recover stale channel subscription#56
Pher217 merged 1 commit into
mainfrom
claude/2026-06-14-daemon-heartbeat-durability

Conversation

@Pher217

@Pher217 Pher217 commented Jun 14, 2026

Copy link
Copy Markdown
Owner

Goal

Fix the "worked last week, broken now" reverse-input failure: a user's Telegram keystrokes (`pty.inject`) silently stopped reaching tmux after the daemon had been connected for hours.

Root cause: the backend `HostDaemonConsumer`'s Redis channel-receive stalls silently (after Redis reconnect churn). The WebSocket stays `ESTABLISHED`, daphne logs no disconnect, but `group_send` messages are never delivered to the daemon. The daemon can't detect this at the ws level — the connection looks healthy, it just receives nothing. Restarting the daemon (fresh consumer + fresh Redis receive) fixes it, proving the connection was "deaf but alive".

Changes

  • host-agent/agent_host/wsclient.py — per-connection liveness marker + two new coroutines in the existing gather:
    • _heartbeat: sends host_heartbeat up every HEARTBEAT_INTERVAL (30s).
    • _watchdog: if no ping returns within HEARTBEAT_TIMEOUT (90s), raises to tear down the connection so the existing outer loop reconnects with a fresh signed URL.
    • _receiver: a ping host_command updates the liveness marker.
  • backend/apps/hostlink/consumers.pyreceive_json handles host_heartbeat by group_send-ing a ping host_command back to the host's group. This deliberately round-trips through group_send → Redis → consumer.host_command → ws — the exact path pty.inject uses — so a stalled consumer fails the heartbeat and triggers recovery.

Approach

Heartbeat is echoed through the group path (not just the raw ws) so it tests the thing that actually decays. Watchdog reconnects only on timeout, so healthy connections are never disrupted. Degrades gracefully: if the backend never echoes, the watchdog becomes a ~90s periodic reconnect — still fixing the decay.

Test Summary

  • Daemon (host-agent/tests/test_wsclient.py): 3 new tests — heartbeat sent periodically, ping resets liveness (no reconnect), watchdog forces reconnect on silence. 24 passed.
  • Backend (backend/apps/hostlink/tests/test_consumers.py): 1 new test — host_heartbeat echoes a ping via the group. 8 passed.
  • New tests pass isolated in 3.2s; reuse existing fake-ws/injectable-connect helpers. No auth/signing/send_keys/offline-queue code touched.

Notes

  • Implemented by a delegated worker to a fixed spec (tasks/specs/daemon-heartbeat-durability.md), reviewed line-by-line.
  • Pre-existing ruff import warnings (lines 27/196) and an asyncio teardown warning in older tests are unrelated and left untouched (surgical scope).

Files Changed

  • host-agent/agent_host/wsclient.py — heartbeat + watchdog
  • backend/apps/hostlink/consumers.py — heartbeat echo
  • host-agent/tests/test_wsclient.py, backend/apps/hostlink/tests/test_consumers.py — tests
  • tasks/specs/daemon-heartbeat-durability.md — spec

… subscription

The backend HostConsumer's Redis channel-receive can stall silently while the
WebSocket stays ESTABLISHED, so pty.inject (Telegram keystrokes) stops reaching
tmux with no error and no disconnect. The daemon cannot detect this at the ws
level — the connection looks healthy, it just receives nothing.

Add an application-level heartbeat: the daemon sends host_heartbeat up; the
backend echoes a ping host_command back THROUGH the channel-layer group path
(group_send -> Redis -> consumer.host_command -> ws), exercising the exact path
that decays. A daemon watchdog forces a reconnect (fresh consumer, fresh Redis
receive) when no echo returns within HEARTBEAT_TIMEOUT. Healthy connections are
never disrupted; degrades to periodic reconnect if the backend never echoes.
@Pher217
Pher217 merged commit 6215c12 into main Jun 15, 2026
1 check passed
@Pher217
Pher217 deleted the claude/2026-06-14-daemon-heartbeat-durability branch June 15, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant