fix(acp): pace relay observer frames (6/s + 90/min, zero burst)#2217
Conversation
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
|
Two correctness issues before approval:
The moved-main turn-timeout/stall work does not otherwise intersect this publisher path: those changes only add/alter observer producers, while the publisher remains a detached task launched before readiness presence. The two paused-time pacing tests pass, and I independently ran the full package suite at the exact PR head: Current score: Minimalness 9/10, Elegance 9/10, Correctness 7/10. The reported live baseline/patched evidence convincingly covers the clean 73-frame startup case, but it does not exercise a NOTICE after pacing begins, which is where issue 1 lives. |
Subscribe to the live observer feed BEFORE taking the replay snapshot, so an event emitted between the two calls can never miss both. The overlap this creates is deduped in the run loop via the snapshot's high-water seq (ObserverEvent.seq is monotonic, assigned at emit). The publisher body moves into run_relay_observer_publisher so the exactly-once property is testable without a spawned task; the new test emits an event inside the race window and asserts before/overlap/after each publish exactly once, in order. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
The generic PublishEvent arm silently drops every publish while the rate-limit gate (#2199) is armed. That is correct for typing indicators, but observer telemetry frames (kind 24200) are durable: one NOTICE from any traffic source erased ~30 frames of turn history over the default ~5s gate. Observer frames now park in a bounded drop-oldest FIFO on BgState (cap 256 ≈ 40s of upstream pacing at 6/s) whenever the gate is armed or earlier parked frames are still draining (order preserved). The existing main-loop drain machinery delivers them one frame per 125ms pacing tick once the gate clears, arming the pacing timer to the gate deadline so drains fire even with no other traffic. Overflow evicts oldest and is counted + logged — visible loss, never silent. The same parking now also covers observer frames hit by a live send failure or emitted while disconnected, so they survive reconnect via the post-reconnect drain. Typing indicators keep the existing drop-while-gated behavior. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co> Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
…cache * origin/main: feat(desktop): fuzzy emoji shortcode autocomplete (#1945) Revert "feat(relay): inventory unreachable Git objects" (#2275) feat(relay): inventory unreachable Git objects (#2264) Keep avatar preview visible during upload (#2237) fix(desktop): keep machine onboarding for unrecognized identities after reset (#2244) fix(managed-agents): stale harness pin shadows runtime edits; add Restart quick action (#2252) fix(desktop): surface model discovery failures in agent config UI (#2246) Hide bundled harnesses from onboarding (#2233) relay: add author_type label to buzz_events_stored_total (#2243) fix(justfile): use Hermit-pinned lefthook in hooks recipe (#2241) Refresh README screenshots (#2236) fix(acp): pace relay observer frames (6/s + 90/min, zero burst) (#2217) Restore npub copy option for private community joins (#2232) fix(desktop): align onboarding runtime auth (#2229) fix(desktop): survive degraded networks with rate-limit-aware relay client (#2197) fix(cli): retry transient relay failures and raise timeouts (#2196) feat(desktop): browse immutable repository tags (#2231) Update built-in agent avatars (#2215) cleanup old AI doc (#2227) chore(release): release Buzz Mobile version 0.4.11 (#2225)
* origin/main: (24 commits) feat(relay): log NIP-98 pubkey attribution on HTTP bridge requests (block#2206) fix(ci): don't rebuild relay tests on every run (block#2203) fix(desktop): refresh cached channel member names (block#2258) chore(release): release Buzz Desktop version 0.4.22 (block#2220) feat(desktop): fuzzy emoji shortcode autocomplete (block#1945) Revert "feat(relay): inventory unreachable Git objects" (block#2275) feat(relay): inventory unreachable Git objects (block#2264) Keep avatar preview visible during upload (block#2237) fix(desktop): keep machine onboarding for unrecognized identities after reset (block#2244) fix(managed-agents): stale harness pin shadows runtime edits; add Restart quick action (block#2252) fix(desktop): surface model discovery failures in agent config UI (block#2246) Hide bundled harnesses from onboarding (block#2233) relay: add author_type label to buzz_events_stored_total (block#2243) fix(justfile): use Hermit-pinned lefthook in hooks recipe (block#2241) Refresh README screenshots (block#2236) fix(acp): pace relay observer frames (6/s + 90/min, zero burst) (block#2217) Restore npub copy option for private community joins (block#2232) fix(desktop): align onboarding runtime auth (block#2229) fix(desktop): survive degraded networks with rate-limit-aware relay client (block#2197) fix(cli): retry transient relay failures and raise timeouts (block#2196) ...
Problem
Starting a harness with many ACP workers replays the whole observer snapshot as a burst of kind-24200 EVENT frames. With 24 workers the snapshot is 73 frames (1
harness_started+ 24×(acp_write initialize,acp_read response,agent_initialized)); plus 3 REQs and presence, that blows the relay's shared WS admission budget (50 events / 5 s,admission.rs×human_ws_events_per_sec = 10) and the relay answers with the countdown NOTICE Morgan reported:Worse than the NOTICE: the rejected observer frames were silently lost (fire-and-forget publish), and the burst consumed budget that REQs/presence/user-visible publishes need.
Fix (harness-only, no relay deploy)
Pace the already-async observer publisher (
spawn_relay_observer_publisher) through a dual-window gate, per the joint design debate in buzz-bugs:LimitType::Messagesbudget with real chat messages; 90/min leaves ≥30/min headroomLaggedwarningStartup cost: the 73-frame snapshot drains in ~12 s of telemetry backlog delay only — agent initialization and readiness are unchanged (publisher task was already decoupled).
Note: the deployed-relay NOTICE-based adaptive rate gate already exists in the harness since #2199 (
set_rate_limit_gatematches therate-limited:prefix and defers REQs / drops ephemeral publishes) — with pacing in place it simply never trips in this scenario.Validation
cargo test -p buzz-acp— 565 passed (includes 2 new paused-time pacer tests: no-burst spacing, rolling-minute cap).cargo clippy -p buzz-acp --all-targetsandcargo fmt --checkclean.Live local test (per TESTING.md): tip-of-main relay, fresh migrated DB, real Redis admission limiter, 24 stub ACP agents with
BUZZ_ACP_RELAY_OBSERVER=true, NIP-OA auth tag so observer frames route:rate-limitedNOTICEsbuzz_admission_rejections_totalThe baseline reproduces Morgan's exact failure (27 rejections, matching the arithmetic from the debate thread); the patched build eliminates it with zero frame loss.
Design debate + consensus spec: buzz-bugs thread (Wren + Eva, 2026-07-18). Implementation is Wren's commit from
wren/pace-observer-publisher(#2069) carried onto current main; #2069 is superseded by this PR.