feat(replays): store rrweb chunk payloads in R2, not ClickHouse

[Makisuo]

Why

session_replay_events held the raw rrweb payload as an opaque Events String. Nothing ever queried inside it — the only reader (sessionReplayEventsQuery) selects the whole column for one (OrgId, SessionId) and orders by ChunkSeq — so it was a key/value blob fetch paying warehouse storage prices, and it's the bulk of a session's bytes (the per-session budget is sized against a ~594 MB p99).

The sibling session_events table deliberately stays in ClickHouse. It's queried across sessions — the event-match semi-join ("sessions where a 5xx happened") and the lagInFrame active/idle window function — which is what a column store is for. Only the blobs move.

What changed

The ingest gateway PUTs the browser's gzip verbatim to an S3-compatible bucket and enqueues a thin index row with an empty Events; the API refills the payload on read. The HTTP response shape is unchanged for both v1 and v2.

No schema change

An empty Events is already an unambiguous "blob-backed" marker: flush() returns early on an empty buffer, so the SDK never uploads a chunk with no events. The object key derives from (OrgId, SessionId, ChunkSeq), all of which the reader already has.

Adding a StorageKey column would have cost a migration across three schema systems plus an org-visible regression — a SCHEMA_VERSION bump gates clickhouse_ready, so every not-yet-migrated BYO-ClickHouse org would have had all signals (logs, traces, metrics) routed to Tinybird until they ran the schema-apply workflow, plus the first-ever versioned local-schema migration edge for a table local mode never reads. Verified this PR leaves SCHEMA_VERSION at 12 and the local structural manifest at v1; only the Tinybird-coupled PROJECT_REVISION moves, from the datasource description edits.

Invariants

• Ordering. The PUT happens before the row is enqueued, so a row can never point at a missing object. A failed PUT returns non-2xx and the SDK drops the chunk — which it already does without retrying. The reverse (orphan object, no row) is harmless and gets swept by lifecycle.
• Retention. The bucket expires objects at 32 days against the table's 30-day TTL, so the row always disappears first. A session that lists as recorded but plays back empty is the one failure with no good client-side handling.
• ByteSize stays uncompressed. It's a published v2 field and the input to ReplaySessionBudget's 1 GiB ceiling, both denominated in decompressed bytes. read_to_string is replaced with a streaming decode-and-discard counter — same number, same rejection of malformed gzip, without the multi-megabyte String alloc + JSON escape + WAL push that was the actual cost.
• Degrades by construction. An unset INGEST_REPLAY_R2_ENDPOINT keeps today's inline path (what self-hosted and BYO-ClickHouse run); a missing R2 binding makes API hydration a no-op. Rollback is unsetting one env var — no schema state to unwind.

Notable choices

• SigV4 hand-rolled (~100 lines) on the hmac/sha2/chrono/reqwest crates the binary already links, rather than pulling the smithy stack in for one verb. Pinned to AWS's published PUT Object test vector.
• PUT inline in the handler, not a new ExportDestination lane. The ExportWorker batches 5000 rows / 4 MiB into one NDJSON body; a blob PUT is one object, one request — a lane would be degenerate and would still push megabytes through the WAL, which is the cost being removed.
• v2 now paginates before hydrating, so a page request fetches only its own payloads instead of the whole session's. (paginateArray still slices post-materialization from the warehouse — that part is unchanged.)

Rollout

Ships dark: INGEST_REPLAY_R2_* is unset, so ingest behaves exactly as today until the Railway env vars are set. Watch maple.replay.storage (inline → r2) and the new ingest_replay_blob_put_failed_total, which should stay at zero.

Follow-ups, deliberately not in this PR: delete the events != "" dual-read branch at T+31 days; per-chunk URLs and progressive loading. Note the storage move does not fix "materialize the whole session in the Worker" — that's already the behavior today (sessionReplayEventsQuery has no LIMIT) and is what the progressive-loading follow-up addresses.

Cache-Control on the events response was scoped in but left out: in this framework it needs header declarations on the versioned v1 and v2 endpoint schemas, which isn't the cheap change it looked like.

Testing

• cargo test (ingest): 70 pass. New coverage for the key scheme, SigV4 against the AWS vector, the byte counter vs read_to_string, verbatim-gzip storage, the inline fallback, and the orphan-prevention invariant (R2 5xx → non-2xx and zero WAL growth).
• apps/api routes + platform: 24 files / 198 tests pass, including 8 new ReplayBlobStore tests. The key-scheme expectations are pinned in both the Rust and TS suites — nothing at runtime reconciles the writer and reader, and a divergence would read as "every recording is empty".
• @maple/query-engine (949), @maple/domain (448), @maple/web (1204) pass. bun typecheck 36/36.
• Caught and fixed mid-review: adding the service requirement to the route groups broke telemetry.http.test.ts, which builds its own v2 layer stack. The route layers now provide ReplayBlobStoreLive themselves. Verified against the parent commit that the failure was mine, not pre-existing.
• @maple/cli fails in this sandbox ("Failed to start server. Is port 0 in use?") — confirmed identical on a clean checkout of the parent commit.

Reviewer notes

• alchemy@2.0.0-beta.64 does expose Cloudflare.R2.Bucket with the lifecycleRules shape used here (verified against installed sources), but the bucket + binding have not been deployed to any stage — that's the first operational step.
• The R2 API token and Railway credentials still need to be minted; apps/ingest/src/r2.rs documents the six INGEST_REPLAY_R2_* vars.

🤖 Generated with Claude Code

https://claude.ai/code/session_019vF2QDhdj77aJrjm89fgiq

---

Generated by Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[Makisuo]

CI red on TypeScript (@maple/web#test), but the failure is on the base branch and predates this PR — 5 failures in apps/web/src/api/warehouse/query-builder-timeseries.test.ts, all bucket-ladder assertions landing one rung finer than expected (120→60, 3600→900, 900→300, 14400→3600).

Traced to b84e828 "Increase graph sample count" on main, which changed the ladder in packages/query-engine/src/datetime.ts and updated packages/query-engine/src/datetime.test.ts and apps/web/src/api/warehouse/timeseries-utils.test.ts — but query-builder-timeseries.test.ts also asserts bucket seconds and wasn't updated. That run's own CI on main failed for the same reason; it reaches this PR through the merge commit.

Verified by checking out b84e828 directly, with none of this branch's code:

Test Files  1 failed (1)
     Tests  5 failed | 12 passed (17)

On this branch alone that file passes 17/17, and nothing here touches the ladder — the only web change in this PR is a comment.

Leaving it to whoever owns b84e828, since the stale expectations should move to whatever the new sample count intends rather than to whatever makes them green. Happy to take it if that's preferred. I'll merge main in and re-run once it's fixed.

Nothing else is red: the pullfrog check is a Router balance/billing failure, unrelated to the diff.

---

Generated by Claude Code

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[pullfrog]

Your Pullfrog Router balance is exhausted.

You have a payment method on file but auto-reload is disabled, so runs paused once your balance went past the overdraft buffer.

Top up balance → · Enable auto-reload →

  ^{｜ Rerun failed job ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using Claude Opus ｜ 𝕏}

[Makisuo]

Pushed one commit here (bf9ebbaf5) while reviewing the stack in #338.

The lifecycle rule was pinned to prefix: "v1/", but replay_object_key versions its key scheme on purpose — the comment in r2.rs says a format change writes under v2/ while v1/ ages out. The first time that happens, the rule stops matching anything and the bucket grows forever, with no failing test to catch it.

The bucket is single-purpose (only the ingest gateway writes it, only the REPLAY_BLOBS binding reads it), so the rule is now unprefixed and covers whatever scheme is current. Expiry is unchanged at 32 days.

Everything else in this PR reviewed clean from #338's side.

[github-actions]

🍁 Maple PR preview

Note

Preview resources were removed when this pull request closed.

Final commit bf9ebba · View workflow run