Route lag-tolerant reads to an optional Postgres read replica

[tlongwell-block]

What

When READ_DATABASE_URL is set (e.g. an Aurora cluster-ro- endpoint), three lag-tolerant read classes move off the writer — server-only, zero client changes, and unset env is byte-for-byte today's behavior:

1. FTS search → replica. The dedicated search pool prefers the replica URL. Search is lag-tolerant by construction and is the canary for replica traffic. Correctness backstop for free: replica FTS hits are re-fetched from the writer canonical store (get_events_by_ids) and re-authorized, so search can never surface an event the writer doesn't have.
2. Db::read() seam. Optional second PgPool with writer fallback when unconfigured. Zero call-site churn.
3. Cursor-bearing pages → replica. Channel scroll-back and thread pagination pass a keyset cursor over immutable history; cursor pages route to read(), head fetches (cursor: None) stay on the writer. Terminal-page writer verification for forward thread pagination: an under-limit replica page is a candidate EOF, so that one page re-runs on the writer — a lagged replica can never truncate a thread tail into a false EOF (per Perci's review of the design).

Scaling then becomes knobs we already have in bb-public (num_read_replicas, reader instance_class, Aurora reader autoscaling as follow-up) — capacity without touching the writer or the app.

Observability

buzz_db_read_pool_{size,idle,active,max} Prometheus gauges (emitted only when the replica pool exists), startup log markers (Postgres connected (writer + read replica), search replica=true).

Testing

Unit/integration — divergent writer/replica fixtures make routing observable: a writer-only "fresh" event and a replica-only "marker" prove which pool served each page (head→writer, cursor→replica, writer terminal-page re-run, writer-fallback-when-unset). buzz-db: 80 unit + 106 Postgres-backed integration, all green at this head. buzz-relay: 687 passed, 1 failed — mesh_demo::demo_join_forwarded_arm_round_trips_echo, network-dependent, reproduces on clean origin/main (ca384d0) in the same environment; pre-existing, not this change.

Live local (TESTING.md-style HARD pass, real data) — restored a production-scale dump (598 MB, 780K events, sha256-verified) into a writer + replica pair, then deliberately diverged them (fresh relay writes land only on the writer; a marker row injected only into the replica — a frozen replica is a maximally-lagged reader). Relay with READ_DATABASE_URL, verified at the wire via CLI + raw /query:

• Head fetch → writer (fresh present, marker absent). Cursor walk of full channel history (11 pages) → replica (marker surfaced, zero writer-only leakage).
• Search → replica, dispositively: a writer-only event with a confirmed writer-side FTS match is missed by relay search (query ran on the replica); replica-only FTS hits are dropped by the writer re-fetch.
• Thread forward pagination under maximal lag (replica has zero thread rows, every page under-fills): all 14/14 writer-only replies collected, correct EOF, no false truncation — terminal-page re-verification working live.
• Playwright GUI: fresh-built bundle, scroll-back perf spec against the live relay — cursor pages fired through the real IntersectionObserver → keyset path; pg_stat_database deltas during the run show the replica serving the cursor pages.
• Flag-off control: same binary, no READ_DATABASE_URL — no read-pool gauges, no replica reads, head + 10-page walk fully served by the writer.

Context

Part 1 of the read-scaling plan discussed in buzz-bb-block-deploy (Eva + Perci + Tyler). Head-fetch offload (overlap gate + SUBACK + cold-state writer bypass) stacks on the Db::read() seam later; nothing here blocks it.

Deployment & operational tradeoffs

• Enablement (Helm): the chart injects READ_DATABASE_URL as an optional secretKeyRef on the relay Deployment. Add the key to the release secret (chart-managed or secrets.existingSecret) to enable routing; omit it and the env is unset = routing disabled = exactly prior behavior. Render tests cover both secret modes. Documented in values.yaml, examples/secret-sample.yaml, and .env.example.
• No transparent mid-query failover: if the replica dies while serving a routed cursor/search query, that in-flight query errors rather than silently retrying on the writer. Subsequent cursor traffic returns to the writer via fence closure; search remains bound to its configured endpoint. Deliberate: retry-on-writer would double worst-case load exactly when the database tier is already unhealthy.
• Replica connection ceiling: with a reader configured, a pod can open up to 20 (read pool) + 10 (search pool, SQLx default) = 30 connections to the replica endpoint, alongside writer 20 + audit 5. Capacity-check max_connections on the reader instance class against pod count before rollout.

(Both tradeoffs surfaced in Max's independent red-team review; the chart gap it found is fixed in c27baf384.)

[tlongwell-block]

Blocking correctness issue in both cursor routes: Postgres replication preserves commit/WAL order, while these keysets paginate by client-signed created_at (plus id). Those orders are not monotonic (ordinary concurrent/offline clients can commit out of timestamp order; ingest permits ±15 minutes at handlers/ingest.rs:1405-1411). Therefore a lagged replica can contain a later-key row while missing an earlier-key row that is already on the writer.

Concrete thread case (lib.rs:1910-1922, ordered ASC in thread.rs:414,427): writer has r1@1, r2@2, r3@3, r4@4. r4 committed before r3, and the replica has replayed through r4 but not r3. Head on writer with limit=2 returns r1,r2. Cursor page on replica returns r4; with limit=1 it is full, so this code accepts it. The next cursor is >r4 and r3 is permanently skipped. The under-limit/EOF verification never runs.

The divergent test does not cover this: it models lag as a timestamp-ordered prefix (lib.rs:5368-5376), then uses a replica-only ghost (explicitly unphysical) to prove the full-page branch. Add the out-of-order-commit fixture above and it should fail.

The same assumption exists for channel scrollback (lib.rs:1961-1966, DESC keyset at thread.rs:605-612): a recently committed event with an older signed timestamp can be absent on the replica while an older WAL-prefix row beyond it fills the page, advancing the cursor past the missing event. In addition, history is not immutable here: events.deleted_at is mutable (thread.rs:596; soft deletes in event.rs:697+), so lag can resurrect deleted rows/summaries.

Safest minimal correction for this PR: keep thread cursor pages writer-bound, and either keep channel pages writer-bound too or introduce an authoritative overlap/fence protocol before routing them. Search + the optional pool seam remain sound wins. Terminal writer verification alone only proves EOF, not completeness of full pages.