feat(acp): make event queue max-retries configurable via BUZZ_ACP_QUEUE_MAX_RETRIES

[obbax]

Summary

The dead-letter threshold for a failed event batch is a hardcoded constant (queue.rs MAX_RETRIES = 10). When the downstream agent is hard-down for its whole outage (we hit this with a quota-exhausted LLM provider: -32603 … session limit), every queued message burns all ten attempts over ~30 minutes of pure backoff noise before dead-lettering, and operators can't tune it without recompiling.

This exposes the cap as --queue-max-retries / BUZZ_ACP_QUEUE_MAX_RETRIES (clap flag with env fallback, same pattern as --context-message-limit / --max-turns-per-session), threaded through Config into a new EventQueue::with_max_retries builder mirroring the existing with_in_flight_deadline. The flag rejects 0 (range(1..)), and MAX_RETRIES remains the default — behavior is byte-for-byte unchanged unless the flag/env var is set. The startup config summary line now includes queue_max_retries=….

Related issue

N/A — none found (searched issues/PRs for queue retry configurability).

Testing

• Existing test_requeue_dead_letters_after_max_retries unchanged — still covers the default path (EventQueue::new defaults to MAX_RETRIES).
• New test_requeue_dead_letters_after_custom_max_retries covers a custom cap via .with_max_retries(2): requeues up to the cap, dead-letters on the following failure.
• cargo test -p buzz-acp queue: 159 passed, 0 failed (Linux x86_64, rust:1-bookworm).

Signed-off-by: obbax robinniclasandersson@gmail.com

[Chessing234]

wiring looks right — with_max_retries(config.queue_max_retries) reaches the queue, so it is not an inert flag.

one thing on the bound: range(1..) has a floor but no ceiling. dead-lettering exists to stop a stuck batch retrying forever, so an operator can now switch that guarantee off with a large enough value and get the unbounded behaviour back without it looking like a mistake. capping it the way MAX_HINT_SECONDS caps a relay-supplied hint would keep the shape consistent.

worth reading alongside #5557 (429s retried with no backoff, 20-42/sec) before picking the ceiling — with no backoff on that path, raising this multiplies the storm rather than giving the batch more time to succeed.

[Chessing234]

two things, both small.

the docs in queue.rs didn't follow the code: :102 ("dead-letter after MAX_RETRIES"), :134 ("if retry_counts[channel] > MAX_RETRIES"), and especially :422 — requeue's own contract doc, "After [MAX_RETRIES] attempts the batch is dead-lettered", an intra-doc link to the constant the function no longer honours. you did update the const doc at :30, so it's just those three.

and worth stating the reach explicitly: this only tunes a hand-run buzz-acp. the sibling knobs are plumbed twice — runtime.rs:672/676 emits BUZZ_ACP_IDLE_TIMEOUT / BUZZ_ACP_MAX_TURN_DURATION on local spawn, and agents_deploy.rs:89 puts them into policy_env for remote deploys — and BUZZ_ACP_QUEUE_MAX_RETRIES is in neither, so desktop-managed and remote agents keep the hardcoded 10. given the incident you describe (provider quota exhaustion) hits hosted agents just as hard, that's exactly the population still untunable.