gateway: 2s websocket heartbeat default disconnects relayed mobile clients (remote.mux / $events)

[MatunSh]

Summary

The gateway's WebSocket heartbeat default (websocketHeartbeatIntervalMs, default 2000 ms) combined with a fixed 2-missed-ping termination kills any client that goes silent for ~6 s. This kills mobile clients connected through dsh-relay: Android freezes backgrounded apps, so the phone's mux socket stays open at the TCP level while the app stops answering pongs — and the gateway terminates its $events stream seconds after the app is backgrounded.

Reproduction

1. Harness 0.1.2-rc.1 with dsh-relay 0.2.1 in front (relay binds :3443, proxies to the loopback gateway).
2. Connect DSH Mobile 0.11.0 from a phone over the relay. /api/remote.mux upgrade and the $events stream open succeed (relay log: OPEN → WS-READY clientId=...).
3. Background the app (or let Android freeze it after ~30–60 s).
4. The gateway heartbeat sweep terminates the connection. Relay lifecycle log shows CLOSED upstream-side after Nms with N ≈ 6–11 s under the default interval.

Impact

The phone app cannot hold its event stream while backgrounded — which is exactly when it needs it (to be notified when a turn completes). Completion notifications on mobile are unusable without a config workaround.

Workaround (verified)

Set websocketHeartbeatIntervalMs: 60000 on the typert-gateway composition row. At a 60 s interval the same phone held its mux for 3+ hours; only truly frozen sockets were reaped (observed 159537 ms / 170215 ms kills, matching 60 s × 2-miss + sweep alignment).

Suggestion

Make the termination deadline an absolute silence time (e.g. terminate only after ≥ 120 s of no pong) instead of a fixed 2-missed-ping count at whatever interval is configured — a 2 s interval with 2 misses is far too aggressive for networked/mobile clients, while a 120 s absolute deadline still reaps genuinely dead sockets promptly. Alternatively (or additionally), raise the default interval for remote/mux streams to 30 s.

Happy to test a patch if that's useful.

[balooooba]

Adding packet-level evidence from a case that matches #6092 exactly, with the trigger narrowed to switching into a session that has a deep transcript.

Setup. dsh@0.1.7-rc.1; dsh web on loopback behind tailscale serve, client on macOS over a phone hotspot. I re-checked 0.2.0-rc.2 and MAX_MISSED_HEARTBEATS = 2 / DEFAULT_WEBSOCKET_HEARTBEAT_INTERVAL_MS = 2000 are unchanged there, so this is still present.

The link is slow but clean, which is why it read as a mystery for a while: ~340 ms average RTT to the internet, 0% ICMP loss over 513 packets, and 0 TCP retransmits / 0 retransmit timeouts on the client. The tailnet path stayed direct with no endpoint change throughout.

Capture. tcpdump on the host loopback between the gateway listener and tailscaled, headers only, 195 s while switching sessions. Four $events mux WebSockets; three were killed by the server.

WS	born	closed	last pong	pong silence	longest pong gap	server→client
A	0.0 s	45.7 s	32.2 s	13.4 s	30.8 s	4.5 MB
B	75.6 s	89.8 s	76.3 s	13.5 s	—	0.8 MB
C	91.6 s	97.9 s	92.3 s	5.6 s	5.8 s	1.0 MB
D	99.7 s	alive	—	—	4.4 s	5.8 MB

Every socket that died exceeded the ~4–6 s deadline; the survivor never went past 4.4 s.

The close is server-initiated, and the client was alive the whole time:

45.658  3080 -> client   [FP.] len=1861   <- gateway sends FIN
63.042  client -> 3080   [P.]  len=6      <- client pong, 17.4 s later
63.042  3080 -> client   [R]              <- gateway RSTs

Between its last pong and the FIN the client kept sending bare TCP ACKs ([.] len=0), so the uplink was working — it simply was not producing pongs while ~1 MB was in flight. Reconnect after the kill took 9.4 s, matching the ~10 s loop reported in #6092.

Traffic shape: ~3.4 MB in the 20 s before kill A, ~2.8 MB before B, ~2.3 MB before C. The user's own description is "loading the transcript takes ~30 s, then the dropouts happen immediately after, usually several."

Workaround. websocketHeartbeatIntervalMs: 30000 on the typert-gateway row in the profile patch (patchReload: live, so no restart needed). Verified on the wire: ping spacing went from 2.000 s to 29.999 s. Same conclusion as #6867.

This is a workaround, not a fix. Consistent with the suggestion in #6092, the durable fix looks like: do not judge the peer dead while its carrier still has bufferedAmount queued (or use an absolute silence deadline rather than a missed-ping count), and bound what the client replays on reconnect.