buzz-agent/buzz-acp: transient network outage paralyzes agent for 12+ min; retries noisy and recovery passive

[csh0101]

Summary

A brief network outage (DNS failure lasting ~30s) paralyzed a managed buzz-agent for 12+ minutes, and the user-visible fallout ("turn error" states in the desktop UI, silent message queueing) persisted long after the network recovered. The harness eventually self-healed via requeue, but the recovery path is slow, opaque, and noisy. Observed on a production multi-agent deployment.

Observed timeline (all times UTC, real logs)

Time	Event
04:00:03	User DMs the agent; message steered into an in-flight turn, deadline extended +7200s
(network)	macOS DNS for the LLM gateway domain breaks; route black-holes
04:02:49	llm: transport error, retrying attempt=1 max_attempts=3 — ~2.5 min after the request started hanging
04:06:49	attempt=2 — another ~4 min hang
04:10:50	cumulative stall 721s across 3 attempts → session_prompt error code -32000
04:10:50	queue: requeueing failed batch with backoff attempt=1 max=10
04:12	Network confirmed healthy again (gateway reachable, 200 OK)
04:14:56	Requeued batch hits transport error again (stale connection state / still-settling routes)
04:19:46	WebSocket error: Connection reset without closing handshake
04:26+	Agent finally drains the backlog and replies to all queued DMs

Problems

1. Each failed LLM attempt hangs minutes before erroring.
crates/buzz-agent/src/llm.rs retries with sensible backoff (MAX_RETRIES=3, 500ms–8s jitter), but the transport error itself takes ~4 minutes to surface per attempt (black-holed route → long TCP/TLS timeout). So a 30-second DNS blip costs 3 × ~4min ≈ 12 minutes of turn paralysis. There is no aggressive per-attempt connect/request timeout for the chat-completions call.

2. Queued messages pile up behind the doomed turn.
New messages arriving during the stall become non-cancelling steers that only extend the hard deadline (extending in-flight deadline by 7200s + 100s buffer) — from the user's perspective the agent simply ignores them.

3. The failure/recovery is user-visible and noisy.
agent_returned outcome="error" propagates to observer frames, so the desktop shows "turn error" in the conversation while the queue is still retrying (requeue max=10). The user concluded "DM is broken" and started restarting the agent — which actually made diagnosis harder. Transient transport failures that will be retried should not surface as terminal-looking errors.

4. No proactive recovery trigger.
After the network heals, nothing probes the LLM endpoint to fast-retry queued batches — recovery waits for whatever backoff/deadline timer fires next (in our case ~10 extra minutes).

Proposals

1. Bounded per-attempt timeout for the LLM HTTP call (e.g. 30–60s, configurable): transport failures should surface in seconds, not minutes.
2. Circuit breaker for transport errors: after N consecutive transport failures, pause new turns, probe the endpoint in the background, and resume + drain the queue immediately when it recovers — instead of riding out long backoff timers.
3. Suppress user-visible error state while retries remain: only surface "turn error" to observers when the requeue budget is actually exhausted; mark intermediate failures as transient (e.g. a "retrying" state distinct from "error").
4. Optional: fail-fast for queued steers — if the in-flight turn is already doomed (LLM unreachable), cancel it at the next retry boundary and re-batch the steered messages, rather than extending a 7200s deadline.

Happy to implement (1) and (2) if the direction sounds right — we have the repro environment and logs.

Environment

• buzz-acp + built-in buzz-agent (OpenAI-compatible provider, GLM), subscribe=all, turn_timeout=320s, max_turn=7200s
• macOS desktop (Tauri managed agents), relay behind an LB over WSS
• Trigger: client-side DNS outage of the LLM gateway domain (~30s), Clash TUN fake-ip involved