fix(api): bound journal scans and seen-sets — API RSS no longer grows with the journal

[brownjuly2003-code]

Closes #183.

Root cause (reproduced at unit scale)

WebhookDispatcher.dispatch_new_events() called fetch_pipeline_events() with no limit every 2 s — SELECT … FROM pipeline_events ORDER BY processed_at ASC over the whole journal. The S11 soak (docs/perf/soak-s11-2026-07-10.md) appended ~683 k journal rows over 4 h, so each poll materialized the entire table (on ClickHouse: the full HTTP JSON response string + a list of ~683 k dicts), and the API's RSS high-water ratcheted to 1.67 GB. The bridge on the same host stayed flat because it never scans the journal.

Unit-scale measurement (DuckDB backend, same mechanism — the leak is journal-size-driven, not ClickHouse-specific):

journal rows	one scan allocates (before)	after
50 000	35.5 MB	≤ 0.8 MB
100 000	71.0 MB	≤ 0.8 MB
200 000	141.9 MB	≤ 0.8 MB
400 000	283.6 MB	≤ 0.8 MB

Extrapolated to the soak's 683 k rows: ~480 MB of transient allocation every 2 seconds by hour 4, plus two retained unbounded seen-sets (WebhookDispatcher.seen_event_ids, MetricCacheController._seen_event_ids — one entry per event forever, the push feed included). Matches the observed curve: monotonic ratcheting growth, flat bridge, and a partial drop (1.67 → 1.17 GB) once load stopped while scans continued against the now-static journal.

Fix

• Incremental bounded webhook scan — each pass fetches at most scan_batch_size (1000) rows at/after a processed_at cursor (new min_processed_at parameter on fetch_pipeline_events; strictly parsed → ValueError before any SQL, inclusive, second-floored — the seen-set dedups the re-fetched cursor second). Delivery semantics preserved: the cursor advances over the contiguous prefix of rows that end the pass seen and freezes at the first row whose durable enqueue failed, so that row is re-fetched and retried next pass, exactly like the full scan's retry-forever behavior. An all-seen full batch still advances the cursor (livelock guard). Startup seeding is O(batch), not O(journal).
• Bounded seen-sets — BoundedSeenSet (capped, FIFO-with-refresh eviction) in the dispatcher (50 k) and cache controller (10 k). Eviction is safe: webhook enqueue is idempotent on its primary key and inline delivery fires only for freshly inserted rows; a redundant metric-cache invalidate merely repopulates on the next read.
• Bonus bug found while fixing — the metric-cache journal-scan fallback was wired as an ascending limited scan: the oldest 200 rows, a window that stops changing once the journal outgrows it, silently disabling scan-driven invalidation (Redis push kept the soak drift-free, which masked it). journal_scan_fetch now reads the newest_first tail window; a regression test pins detection on a journal larger than the window.

Verification

• Repro before/after (table above): per-scan allocation flat at ×8 journal growth; seen capped; cursor reaches the tail.
• 1848 unit tests pass locally (2 unrelated locals cleared: one timing-flaky under full-suite load — passes in isolation; one stale editable-install of the SDK). ruff check/format --check clean over src/ tests/ scripts/ sdk/.
• New tests: cursor call shape, cursor advance, freeze-on-enqueue-failure then retry without duplicate POST, all-seen full-batch advance, seen-set caps (dispatcher + push path), tail-window fallback regression, min_processed_at parse/injection guards.
• Not claimed: live stand RSS re-verification — scheduled for the next stand window alongside S13; STATUS.md says exactly that.

Follow-up (not in scope)

The SSE stream's per-connection seen_event_ids set (routers/stream.py) grows for the lifetime of one connection — same class, but per-connection and not part of the soak's growth (no SSE clients were connected). Noted here for a future pass.

🤖 Generated with Claude Code

[github-actions]

DORA Metrics

• Window: last 30 days
• Branch: main
• Deployment frequency: 104 total / 24.27 per week
• Lead time for changes: avg 0.37h / median 0.0h
• Change failure rate: 59.62% (62/104)
• MTTR: 2.56h across 1 incident(s)