feat(replays): manifest + bounded chunk ranges for playback

[Makisuo]

Stacked on #337 — base is claude/session-events-storage-0u2xp3, not main. Review that one first; the diff here is only the progressive-loading layer #337 names as its follow-up.

Why

Opening a replay often failed with 503 "Database is temporarily unavailable". Nothing was unavailable, and retrying could only fail the same way.

getReplayEvents selected Events for every chunk of a session with no LIMIT and buffered it into a 128 MB Worker. Ingest allows 1 GiB per session (p99 ~594 MB), so the abort matched the transient-upstream rule in execution/errors.ts, became a WarehouseUpstreamError, and was retried 3× before failing. Production getReplayEvents spans on Aug 3–4 ran 428ms → 12.6s against the list profile's 15s cap — the tail was walking into the wall.

#337 moves where those bytes live; it explicitly does not change how many are read at once. That's this PR.

What changed

Playback reads the way other replay products do (PostHog returns snapshot sources then fetches blobs; Sentry paginates segments): a cheap manifest of chunk positions and sizes, then payloads a bounded range at a time, seeded from a checkpoint.

• GET /v2/session_replays/:id/manifest — every chunk's position and size, no payloads. Pure index read; nothing to hydrate whatever the storage backend.
• GET /v2/session_replays/:id/events?from_chunk_seq=&to_chunk_seq= — bounded, cursor-paginated within the range, pushed into SQL.

Not streaming. On the managed Tinybird backend boundedResponseFetch reassembles the whole body before the SDK parses it, so HttpServerResponse.stream would stream a buffer the Worker already materialized — same OOM, more code, and it would leave the typed client path.

The v1 payload endpoint is deleted, not kept in sync. A second surface is a second way to reintroduce the unbounded read. Every other v1 replay endpoint is untouched. v1 no longer needs ReplayBlobStore at all, so #337's v1 hydration wiring comes back out.

The seam with #337

Worth reviewing closely, because combining these naively silently disables the guard this PR exists for.

compiledQueryBounded's responseLimits caps what the warehouse returns. Once payloads live in R2 that response is only an index — events is "" — so the ceiling stops measuring anything that matters, and the memory cost moves to blobs.hydrate(), which is unbounded.

Both ceilings are now in place, because there are two places a payload can come from:

• responseLimits — still the only guard for pre-cutover rows, which carry events inline.
• assertRangeFitsBudget — sums ByteSize from the index and refuses before hydrating. ByteSize is the uncompressed size feat(replays): store rrweb chunk payloads in R2, not ClickHouse #337 deliberately preserved, so an over-budget range is rejected without fetching a single object.

Over-budget is a typed 413 range_too_large, exempt from the v2 redaction: it carries no database diagnostics, only the range asked for, and it is the one error here where telling the caller what to do is the whole value.

Caught by end-to-end testing

Two things that unit tests would not have found, both fixed here:

1. A FirstEventMs column was added for exact seek anchoring, then reverted. The deployed cluster doesn't have it, so every read returned warehouse_schema_drift (502) — and the insert mapping named a column the table lacks, which would have broken replay ingestion. It also tripped the local-CLI append-only schema gate. Chunk position now comes from Timestamp; it trails by upload latency, which is well inside one chunk, and the exact offset comes from the loaded rrweb events. Net: this PR changes no schema.

2. Fixed-size ranges 413 on real data. A 16-chunk range assumed uniform chunks. Production sessions carry 838 KB chunks — 16 of those is ~13 MB against an 8 MB ceiling. Ranges are now planned from the manifest's byte sizes; walking greedily from the first chunk keeps boundaries stable as a live session appends, which is what the atom cache keys on. Verified live: the range came back 0–8, not 0–15.

Also fixed while verifying: the engine-mount effect keyed on the manifest total, so every manifest refetch rebuilt the Replayer — on a live session that resets the playhead every few seconds and reads as "playback frozen".

Compatibility

No schema change and no deploy ordering. The read path uses only columns every existing recording already has. Covered explicitly: checkpoint-less sessions (the SDK's over-cap guard drops the opening snapshot), single-chunk sessions, sessions whose chunk_seq doesn't start at 0, and JSON-quoted UInt64s from backends that refuse the unquote setting.

Testing

958 query-engine · 448 domain · 106 browser-session · 103 web replays · 325 api · 152 ingest (Rust) — all pass. Typecheck clean across five packages; clickhouse:schema:check passes in full, including the local-schema history check (still v1).

Verified live against the real stack (api + web, production-shaped data): manifest 200 → byte-planned range 200 → 47 KB of replayed DOM, scrubber at the manifest-derived duration, seeded from the first checkpoint rather than chunk 0.

One limit: wall-clock playback could not be driven in the verification browser — the pane renders hidden (document.hidden === true, a direct probe fired 0 rAF frames in 2s), and both the clock loop and rrweb's timer are rAF-driven. Transport and rendering are proven live; playback and seek behaviour are pinned in replay-progressive-load.test.ts as a simulated 100-chunk playthrough (each range fetched once, tail still unfetched mid-session, scrub-back hits cache, a seek past the loaded span rebuilds from the preceding checkpoint). Worth one manual click of play before merge.

Follow-ups, not here

• deriveMeta sees only loaded regions, so timeline markers fill in as you play. The right fix is sourcing them from sessionTranscriptQuery, which is already bounded, paginated, and prefetched by the route.
• TRANSIENT_RETRY_SCHEDULE retries a query that has already burned most of the request budget — a guard there would cap tail latency across every route.

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[github-actions]

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit c1f2173 · View workflow run

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
six-metric predicted-vs-observed comparison FAILED/ six-metric predicted-vs-observed comparison FAILED	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}