Repository navigation
v0.21.0
This release replaces the broadcast dispatchers' idle-timeout heuristic with explicit per-target accounting tied to mesh-state signals. Long-running fleet commands (e.g. dnf install across thousands of hosts) no longer get cut off at 30 s of silence between fast and slow responders; unreachable agents get retired the moment the server learns they're gone instead of pinning the dispatcher to the hard timeout; users can now set --timeout arbitrarily long without artificial caps.
Changed
- Broadcast dispatchers (query, exec, deploy) no longer use idle timeouts. Completion is driven by per-target accounting: a session's loop runs until
pendingis empty. Real responses and synthetic disconnect failures pass through one shared first-terminal-wins gate (sessionAccounting.ClaimAgent) so each target is counted at most once. The hard timeout becomes a true safety net atcommand_timeout + 30s transport graceand rarely fires in practice — stream-loss notifications retire unreachable agents long before it would. - Stream loss now drives dispatcher completion via four signal paths:
AgentStreamclose (synthesizes failures for the whole subtree below the lost ZL);PeerDisconnectedfrom upstream relays (now also callsMeshTopology.MarkSubtreeOfflinerather than only touching the DB); the reaper (walks the in-memory topology, notifies dispatchers about any online agent whose ZL ancestor lost its stream); fanout failures at dispatch time (when a ZL's send buffer is full, the subtree it would have relayed to is immediately accounted as failed instead of waiting forever). /api/v1/debug/inflightnow includes deploy sessions with target/received/missing counts to match exec and query.dirq hosts graph --dotrenders left-to-right (rankdir=LR) so large fleet trees fit usefully on screen.
Fixed
- No more
1/2490 completedlies on long-running broadcasts. The old idle timeout fired at 30 s of silence — for adnf installtaking 30+ s per host, the dispatcher exited after the first local response landed, claiming completion while the broadcast was actually still in progress on 2,489 other hosts. - AWS test fleet stops losing ~0.4 % of VHs to ephemeral-port collisions. Linux userdata now reserves ports 50052-51051 via
net.ipv4.ip_local_reserved_portsbeforednf installruns. Sized for up to 1,000 VHs per VM (50,000-agent fleets).
Notes
The design was reviewed against codex's "first terminal event wins per agent" critique. Notable invariants the implementation preserves:
- One synchronized gate (
ClaimAgent) covers both real responses and synthetic disconnect injections so the dispatcher never counts an agent twice. - Lock hygiene: snapshot session pointers under each global session mutex, release, then call per-session
markGoneinline. Result channels are never written while holding the global session lock — so stream close handlers don't get blocked by a slow dispatcher. - Subtree walks in
MeshTopologyare iterative (not recursive) and snapshot under a single RLock pass. - Synthesized failures count toward completion but are omitted from returned results — surfaced via the existing
missingfield added in 0.20.3.