Fix: agents on maxed-out hosts dropped by the dashboard
On a heavily loaded host (e.g. 12 nodes on one server, ~90ms link to the dashboard) agents were dropped every ~60-90s and agent updates failed. The dashboard log pinned the cause:
failed to handle control frame opPing: failed to write control frame opPong:
failed to acquire lock: context deadline exceeded
The agent's keepalive Ping arrives, the dashboard must answer with a Pong, but the connection's write lock is held by a conn.Write that isn't draining — the overloaded host isn't reading fast enough, so TCP backpressure builds. The Pong misses its deadline, the dashboard's read loop dies, the socket closes, and the agent sees unexpected EOF.
Two backward-compatible changes:
1. Update binary no longer travels over the WebSocket
Pushing the ~15 MB binary base64-encoded (~20 MB) through a single conn.Write was the worst offender — it held the write lock for the whole transfer and starved Pongs. Agents now advertise an http-update capability; when the dashboard sees it, the update command carries a download_path and the agent pulls the binary over a separate HTTP request (new endpoint GET /api/agent/binary/{server_id}, server_id-validated, checksum still verified before install).
Fully backward compatible, no manual step for anyone: agents that don't advertise the capability keep getting the inline base64 payload; a new agent talking to an old dashboard still accepts inline. Rollout order doesn't matter.
2. Discovery interval adapts to load (agent)
When a discovery cycle is slow (≥5s — Docker-daemon contention) the agent backs the interval off (doubling, capped at 5 min) instead of burning a constant slice of an already-starved CPU every 60s; it snaps back to 60s once discovery is fast again. Less CPU on discovery → the read loop keeps up with the socket → the backpressure eases at the root.
Both sides change (agent + dashboard). Thanks to the capability negotiation, rollout order is irrelevant and no agent needs a manual update.
Full Changelog: v0.3.86...v0.3.87