Skip to content

fix(acp): pace relay observer frames (6/s + 90/min, zero burst)#2217

Merged
tlongwell-block merged 4 commits into
mainfrom
eva/pace-observer-publisher
Jul 21, 2026
Merged

fix(acp): pace relay observer frames (6/s + 90/min, zero burst)#2217
tlongwell-block merged 4 commits into
mainfrom
eva/pace-observer-publisher

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Problem

Starting a harness with many ACP workers replays the whole observer snapshot as a burst of kind-24200 EVENT frames. With 24 workers the snapshot is 73 frames (1 harness_started + 24×(acp_write initialize, acp_read response, agent_initialized)); plus 3 REQs and presence, that blows the relay's shared WS admission budget (50 events / 5 s, admission.rs × human_ws_events_per_sec = 10) and the relay answers with the countdown NOTICE Morgan reported:

rate-limited: quota exceeded; retry in 5s

Worse than the NOTICE: the rejected observer frames were silently lost (fire-and-forget publish), and the burst consumed budget that REQs/presence/user-visible publishes need.

Fix (harness-only, no relay deploy)

Pace the already-async observer publisher (spawn_relay_observer_publisher) through a dual-window gate, per the joint design debate in buzz-bugs:

  • 6 frames/sec (167 ms interval), zero startup burst — even the first snapshot frame waits for its slot
  • 90 frames / rolling 60 s governor — observer frames share the agent's 120/min LimitType::Messages budget with real chat messages; 90/min leaves ≥30/min headroom
  • Snapshot drain and live tail both flow through the same gate; readiness, subscriptions, presence, and normal publishes are untouched (they don't go through this task)
  • No retry/ACK queue. Overflow degrades to bounded delay via the existing broadcast buffer, then to the existing visible Lagged warning

Startup cost: the 73-frame snapshot drains in ~12 s of telemetry backlog delay only — agent initialization and readiness are unchanged (publisher task was already decoupled).

Note: the deployed-relay NOTICE-based adaptive rate gate already exists in the harness since #2199 (set_rate_limit_gate matches the rate-limited: prefix and defers REQs / drops ephemeral publishes) — with pacing in place it simply never trips in this scenario.

Validation

cargo test -p buzz-acp — 565 passed (includes 2 new paused-time pacer tests: no-burst spacing, rolling-minute cap). cargo clippy -p buzz-acp --all-targets and cargo fmt --check clean.

Live local test (per TESTING.md): tip-of-main relay, fresh migrated DB, real Redis admission limiter, 24 stub ACP agents with BUZZ_ACP_RELAY_OBSERVER=true, NIP-OA auth tag so observer frames route:

baseline (main) patched
rate-limited NOTICEs 27 0
buzz_admission_rejections_total 27 0 (unchanged counter)
observer frames fanned out 46 / 73 (27 lost) 73 / 73
agents initialized 24/24 24/24
frame spacing burst min gap 163 ms, max 6 frames in any 1 s window, 30 in any 5 s window, 73 in 60 s
snapshot drain time 12.2 s

The baseline reproduces Morgan's exact failure (27 rejections, matching the arithmetic from the debate thread); the patched build eliminates it with zero frame loss.

Design debate + consensus spec: buzz-bugs thread (Wren + Eva, 2026-07-18). Implementation is Wren's commit from wren/pace-observer-publisher (#2069) carried onto current main; #2069 is superseded by this PR.

Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
@tlongwell-block

Copy link
Copy Markdown
Collaborator Author

Two correctness issues before approval:

  1. RelayEventPublisher feeds observer kind 24200 through the generic RelayCommand::PublishEvent branch. Since fix(acp): honor relay rate limits and pace resubscribes on bad links #2199, that branch drops every event while rate_limit_gate is active (relay.rs:1352-1364); it does not slow this pacer. Thus any rate-limited: NOTICE (including one caused by unrelated functional traffic) makes the next ~5 seconds of paced observer frames disappear silently. At 6/s that is roughly 30 frames. This contradicts the agreed adaptive-backpressure/no-silent-loss behavior and the PR description's statement that fix(acp): honor relay rate limits and pace resubscribes on bad links #2199 already supplies it. Please make observer publishes wait through the gate (kind-aware, so typing can remain droppable), or otherwise connect the gate to this pacer without a retry/ACK queue.

  2. The new comment says “Subscribe before pacing,” but the code still calls snapshot() before subscribe(). An event emitted in that gap is in neither the cloned snapshot nor a receiver's broadcast queue. Please reverse those two calls (let mut rx = observer.subscribe(); let snapshot = observer.snapshot();). This is a tiny race, but the change is specifically establishing loss-free snapshot-to-live handoff.

The moved-main turn-timeout/stall work does not otherwise intersect this publisher path: those changes only add/alter observer producers, while the publisher remains a detached task launched before readiness presence. The two paused-time pacing tests pass, and I independently ran the full package suite at the exact PR head: cargo test -p buzz-acp → 565 passed at 8e4a0d73ccfbfca4771661d66745997a1fa8c5ee (same-shell SHA check).

Current score: Minimalness 9/10, Elegance 9/10, Correctness 7/10. The reported live baseline/patched evidence convincingly covers the clean 73-frame startup case, but it does not exercise a NOTICE after pacing begins, which is where issue 1 lives.

npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d and others added 3 commits July 21, 2026 07:34
Subscribe to the live observer feed BEFORE taking the replay snapshot,
so an event emitted between the two calls can never miss both. The
overlap this creates is deduped in the run loop via the snapshot's
high-water seq (ObserverEvent.seq is monotonic, assigned at emit).

The publisher body moves into run_relay_observer_publisher so the
exactly-once property is testable without a spawned task; the new test
emits an event inside the race window and asserts before/overlap/after
each publish exactly once, in order.

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
The generic PublishEvent arm silently drops every publish while the
rate-limit gate (#2199) is armed. That is correct for typing
indicators, but observer telemetry frames (kind 24200) are durable:
one NOTICE from any traffic source erased ~30 frames of turn history
over the default ~5s gate.

Observer frames now park in a bounded drop-oldest FIFO on BgState
(cap 256 ≈ 40s of upstream pacing at 6/s) whenever the gate is armed
or earlier parked frames are still draining (order preserved). The
existing main-loop drain machinery delivers them one frame per 125ms
pacing tick once the gate clears, arming the pacing timer to the gate
deadline so drains fire even with no other traffic. Overflow evicts
oldest and is counted + logged — visible loss, never silent.

The same parking now also covers observer frames hit by a live send
failure or emitted while disconnected, so they survive reconnect via
the post-reconnect drain. Typing indicators keep the existing
drop-while-gated behavior.

Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Co-authored-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
Signed-off-by: npub12gtutshhh76rx0jx697f32f9tffd4hhp3hx58fp4x6u4uemkm7sqf8f757 <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@sprout-oss.stage.blox.sqprod.co>
@tlongwell-block
tlongwell-block merged commit 9a788c7 into main Jul 21, 2026
32 checks passed
@tlongwell-block
tlongwell-block deleted the eva/pace-observer-publisher branch July 21, 2026 16:57
loganj added a commit that referenced this pull request Jul 21, 2026
…cache

* origin/main:
  feat(desktop): fuzzy emoji shortcode autocomplete (#1945)
  Revert "feat(relay): inventory unreachable Git objects" (#2275)
  feat(relay): inventory unreachable Git objects (#2264)
  Keep avatar preview visible during upload (#2237)
  fix(desktop): keep machine onboarding for unrecognized identities after reset (#2244)
  fix(managed-agents): stale harness pin shadows runtime edits; add Restart quick action (#2252)
  fix(desktop): surface model discovery failures in agent config UI (#2246)
  Hide bundled harnesses from onboarding (#2233)
  relay: add author_type label to buzz_events_stored_total (#2243)
  fix(justfile): use Hermit-pinned lefthook in hooks recipe (#2241)
  Refresh README screenshots (#2236)
  fix(acp): pace relay observer frames (6/s + 90/min, zero burst) (#2217)
  Restore npub copy option for private community joins (#2232)
  fix(desktop): align onboarding runtime auth (#2229)
  fix(desktop): survive degraded networks with rate-limit-aware relay client (#2197)
  fix(cli): retry transient relay failures and raise timeouts (#2196)
  feat(desktop): browse immutable repository tags (#2231)
  Update built-in agent avatars (#2215)
  cleanup old AI doc (#2227)
  chore(release): release Buzz Mobile version 0.4.11 (#2225)
loganj added a commit to loganj/buzz that referenced this pull request Jul 22, 2026
* origin/main: (24 commits)
  feat(relay): log NIP-98 pubkey attribution on HTTP bridge requests (block#2206)
  fix(ci): don't rebuild relay tests on every run (block#2203)
  fix(desktop): refresh cached channel member names (block#2258)
  chore(release): release Buzz Desktop version 0.4.22 (block#2220)
  feat(desktop): fuzzy emoji shortcode autocomplete (block#1945)
  Revert "feat(relay): inventory unreachable Git objects" (block#2275)
  feat(relay): inventory unreachable Git objects (block#2264)
  Keep avatar preview visible during upload (block#2237)
  fix(desktop): keep machine onboarding for unrecognized identities after reset (block#2244)
  fix(managed-agents): stale harness pin shadows runtime edits; add Restart quick action (block#2252)
  fix(desktop): surface model discovery failures in agent config UI (block#2246)
  Hide bundled harnesses from onboarding (block#2233)
  relay: add author_type label to buzz_events_stored_total (block#2243)
  fix(justfile): use Hermit-pinned lefthook in hooks recipe (block#2241)
  Refresh README screenshots (block#2236)
  fix(acp): pace relay observer frames (6/s + 90/min, zero burst) (block#2217)
  Restore npub copy option for private community joins (block#2232)
  fix(desktop): align onboarding runtime auth (block#2229)
  fix(desktop): survive degraded networks with rate-limit-aware relay client (block#2197)
  fix(cli): retry transient relay failures and raise timeouts (block#2196)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant