feat(web-analytics): gate session replay tile behind removal experiment - #70048
Conversation
Gate the web analytics dashboard's Session Replay tile behind the `web-analytics-remove-replay-tile` experiment flag. Control keeps the tile (current behavior, the default when the flag is absent); the `test` variant hides it. The tile mounts sessionRecordingsPlaylistLogic on every dashboard load to render only its top 5 rows, so it runs a full session-recordings-list query each time. Usage data shows almost no one interacts with it, so this sets up an A/B test to measure whether removing it affects retention before dropping it outright. Generated-By: PostHog Code Task-Id: 98495d89-d4bb-423c-b2b3-8efd32e3f9c0
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Reviews (1): Last reviewed commit: "feat(web-analytics): gate session replay..." | Re-trigger Greptile |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cc220c1a7d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Note 🤖 stamphog reviewed Unresolved Codex comment on the current head correctly flags that the flag is read for bot-analytics tab users too, polluting the experiment's retention/guardrail metrics — a substantive, unaddressed concern that undermines the PR's stated purpose.
Gate mechanics and policy version
|
allTiles is also constructed for the bot-analytics tab (then discarded via an early return), so reading the removal experiment flag unconditionally enrolled bot-tab users the experiment can't affect, diluting the retention and replay guardrail metrics. Only read the flag when productTab is the analytics tab, where the tile actually renders. Generated-By: PostHog Code Task-Id: 98495d89-d4bb-423c-b2b3-8efd32e3f9c0
🤖 CI report✅ Bundle size — 🟢 -61.5 KiB (-0.1%)Uncompressed size of every built Total: 70.02 MiB · 🟢 -61.5 KiB (-0.1%)
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report ✅ Eager graph — within budgetHow much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy
🟢 Largest files eagerly shipped from
|
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 277.3 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/rrweb.js |
| 266.9 KiB | ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 224.6 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.2 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +203.3 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1438.40 MiB · 🔺 +203.3 KiB (+0.0%)
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
The replay tile is also hidden whenever a conversion goal is set (the !conversionGoal guard), so reading the removal experiment flag for analytics-tab viewers with a conversion goal enrolled users neither variant affects. Gate the flag read on !conversionGoal as well, so enrollment only happens on the code path where the tile is actually shown or removed. Generated-By: PostHog Code Task-Id: 98495d89-d4bb-423c-b2b3-8efd32e3f9c0
2 participants
Problem
The Web analytics dashboard renders a "Session replay" tile that mounts
sessionRecordingsPlaylistLogicon every dashboard load to show just its top 5 recordings, so it runs a full session-recordings-list query each time the dashboard opens. Usage analysis shows the tile gets almost no genuine engagement (people reach replays via the left nav and the contextual "View recordings" cross-sell button instead), so we want to test removing it — but validate retention isn't affected before dropping it for good.Changes
web-analytics-remove-replay-tilefeature flag (control/test) infrontend/src/lib/constants.tsx.TileId.REPLAYtile inwebAnalyticsLogic.tsxso it renders only when the flag is nottest. Default/absent flag keeps the tile (no behavior change for anyone not enrolled); thetestvariant removes it, which also stops the recordings-list query from running on those loads.This is scaffolding for experiment #382421:
/replaypageviews) — should not drop.How did you test this code?
Not manually run by the agent (no dev stack in this environment). The change mirrors the existing
WEB_ANALYTICS_TILE_HEADER_V2 === 'test'flag-gating pattern already used in the same selector, and the flag comparison is type-identical to it. CI typecheck/lint will cover the static checks.Automatic notifications
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Requested from a Slack thread investigating whether the Session Replay tile in Web analytics earns its query cost. The investigation found the tile is barely used, so the ask was to set up an A/B test for removing it and measure retention + whether anyone misses it.
/creating-experiments,/configuring-experiment-analytics.web-analytics-remove-replay-tileflag were created via MCP; this PR only wires the flag into the frontend.controlwas chosen as the tile-shown baseline so absent-flag behavior is unchanged.Created from a Slack thread.