fix(host-agent): daemon heartbeat + watchdog to recover stale channel subscription - #56
Merged
Merged
Conversation
… subscription The backend HostConsumer's Redis channel-receive can stall silently while the WebSocket stays ESTABLISHED, so pty.inject (Telegram keystrokes) stops reaching tmux with no error and no disconnect. The daemon cannot detect this at the ws level — the connection looks healthy, it just receives nothing. Add an application-level heartbeat: the daemon sends host_heartbeat up; the backend echoes a ping host_command back THROUGH the channel-layer group path (group_send -> Redis -> consumer.host_command -> ws), exercising the exact path that decays. A daemon watchdog forces a reconnect (fresh consumer, fresh Redis receive) when no echo returns within HEARTBEAT_TIMEOUT. Healthy connections are never disrupted; degrades to periodic reconnect if the backend never echoes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goal
Fix the "worked last week, broken now" reverse-input failure: a user's Telegram keystrokes (`pty.inject`) silently stopped reaching tmux after the daemon had been connected for hours.
Root cause: the backend `HostDaemonConsumer`'s Redis channel-receive stalls silently (after Redis reconnect churn). The WebSocket stays `ESTABLISHED`, daphne logs no disconnect, but `group_send` messages are never delivered to the daemon. The daemon can't detect this at the ws level — the connection looks healthy, it just receives nothing. Restarting the daemon (fresh consumer + fresh Redis receive) fixes it, proving the connection was "deaf but alive".
Changes
host-agent/agent_host/wsclient.py— per-connection liveness marker + two new coroutines in the existinggather:_heartbeat: sendshost_heartbeatup everyHEARTBEAT_INTERVAL(30s)._watchdog: if nopingreturns withinHEARTBEAT_TIMEOUT(90s), raises to tear down the connection so the existing outer loop reconnects with a fresh signed URL._receiver: apinghost_command updates the liveness marker.backend/apps/hostlink/consumers.py—receive_jsonhandleshost_heartbeatbygroup_send-ing apinghost_command back to the host's group. This deliberately round-trips throughgroup_send → Redis → consumer.host_command → ws— the exact pathpty.injectuses — so a stalled consumer fails the heartbeat and triggers recovery.Approach
Heartbeat is echoed through the group path (not just the raw ws) so it tests the thing that actually decays. Watchdog reconnects only on timeout, so healthy connections are never disrupted. Degrades gracefully: if the backend never echoes, the watchdog becomes a ~90s periodic reconnect — still fixing the decay.
Test Summary
host-agent/tests/test_wsclient.py): 3 new tests — heartbeat sent periodically, ping resets liveness (no reconnect), watchdog forces reconnect on silence. 24 passed.backend/apps/hostlink/tests/test_consumers.py): 1 new test —host_heartbeatechoes apingvia the group. 8 passed.send_keys/offline-queue code touched.Notes
tasks/specs/daemon-heartbeat-durability.md), reviewed line-by-line.Files Changed
host-agent/agent_host/wsclient.py— heartbeat + watchdogbackend/apps/hostlink/consumers.py— heartbeat echohost-agent/tests/test_wsclient.py,backend/apps/hostlink/tests/test_consumers.py— teststasks/specs/daemon-heartbeat-durability.md— spec