feat(replay): add experiment context to the session replays player - #69942
Conversation
…eriment variants GET /api/projects/:id/experiments/session_context/?session_id=... returns which launched experiments (and variants) a session recording saw, merging $feature_flag_called exposure events with stamped $feature/<key> properties to cover the SDK dedupe gap. Includes generated TS/MCP client types. Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
Behind the replay-experiment-context flag: a kea logic fetches the session's experiment context from the new session_context endpoint once per recording, drives variant tags in the player top bar (with a multi-variant warning state) and an Experiments section in the overview sidebar with a jump-to-variant- assignment seek and an experiments cross-sell product intent. Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
Synthesizes an 'experiment-variant' inspector item per experiment the session saw, positioned at the flag-evaluation moment from the session context, and widens the seekbar tick types so the markers render as clickable ticks. Items without an in-session flag evaluation produce no marker. Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
exporter/_parent/node_modules/.pnpm/katex@0.16.45/node_modules/katex/dist/katex.js |
259.3 KiB | 🔺 +259.3 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/architectureDiagram-3BPJPVTR.js |
147.3 KiB | 🔺 +147.3 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/sequenceDiagram-3UESZ5HK.js |
114.4 KiB | 🔺 +114.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/cose-bilkent-S5V4N54A.js |
80.8 KiB | 🔺 +80.8 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/blockDiagram-GPEHLZMM.js |
70.4 KiB | 🔺 +70.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/c4Diagram-AAUBKEIU.js |
68.9 KiB | 🔺 +68.9 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/flowDiagram-I6XJVG4X.js |
59.9 KiB | 🔺 +59.9 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/ganttDiagram-6RSMTGT7.js |
49.4 KiB | 🔺 +49.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/vennDiagram-CIIHVFJN.js |
40.5 KiB | 🔺 +40.5 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/xychartDiagram-2RQKCTM6.js |
38.5 KiB | 🔺 +38.5 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/quadrantDiagram-W4KKPZXB.js |
33.5 KiB | 🔺 +33.5 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/requirementDiagram-4Y6WPE33.js |
30.9 KiB | 🔺 +30.9 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/timeline-definition-PNZ67QCA.js |
30.3 KiB | 🔺 +30.3 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/gitGraphDiagram-PVQCEYII.js |
29.2 KiB | 🔺 +29.2 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/erDiagram-TEJ5UH35.js |
26.8 KiB | 🔺 +26.8 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/mermaid.core.js |
26.7 KiB | 🔺 +26.7 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/wardleyDiagram-YWT4CUSO.js |
25.5 KiB | 🔺 +25.5 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/journeyDiagram-JHISSGLW.js |
23.2 KiB | 🔺 +23.2 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/sankeyDiagram-5OEKKPKP.js |
22.8 KiB | 🔺 +22.8 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/mindmap-definition-RKZ34NQL.js |
22.8 KiB | 🔺 +22.8 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/kanban-definition-UN3LZRKU.js |
20.5 KiB | 🔺 +20.5 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/ishikawaDiagram-YF4QCWOH.js |
17.4 KiB | 🔺 +17.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/diagram-OG6HWLK6.js |
11.4 KiB | 🔺 +11.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/dagre-BM42HDAG.js |
11.3 KiB | 🔺 +11.3 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/diagram-KO2AKTUF.js |
11.1 KiB | 🔺 +11.1 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/stateDiagram-AJRCARHV.js |
10.7 KiB | 🔺 +10.7 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/diagram-2AECGRRQ.js |
6.4 KiB | 🔺 +6.4 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/diagram-LMA3HP47.js |
4.8 KiB | 🔺 +4.8 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/pieDiagram-4H26LBE5.js |
4.7 KiB | 🔺 +4.7 KiB (new) |
render-query/src/render-query/render-query.js |
21.21 MiB | 🔺 +3.9 KiB (+0.0%) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/diagram-5GNKFQAL.js |
3.4 KiB | 🔺 +3.4 KiB (new) |
exporter/src/lib/lemon-ui/LemonMarkdown/MermaidDiagram.js |
2.2 KiB | 🔺 +2.2 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/infoDiagram-5YYISTIA.js |
1.2 KiB | 🔺 +1.2 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/classDiagram-v2-Q7XG4LA2.js |
1.0 KiB | 🔺 +1.0 KiB (new) |
exporter/_parent/node_modules/.pnpm/mermaid@11.15.0/node_modules/mermaid/dist/chunks/mermaid.core/classDiagram-4FO5ZUOK.js |
1.0 KiB | 🔺 +1.0 KiB (new) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.21 MiB · 22 files | 🔺 +76 B (+0.0%) | ███░░░░░░░ 28.2% of 4.29 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.10 MiB · 2,972 files | 🔺 +4.2 KiB (+0.1%) | █████████░ 87.5% of 9.25 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 278.6 KiB | ../node_modules/.pnpm/posthog-js@1.399.2/node_modules/posthog-js/dist/rrweb.js |
| 266.9 KiB | ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 234.9 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.2/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.3 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +13.74 MiB (+1.1%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1280.85 MiB · 🔺 +13.74 MiB (+1.1%)
ℹ️ MCP UI apps size — 31 app(s), 15651.9 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 577.8 KB | 177.4 KB |
| action | 429.2 KB | 177.4 KB |
| action-list | 542.6 KB | 177.4 KB |
| cohort | 428.2 KB | 177.4 KB |
| cohort-list | 541.6 KB | 177.4 KB |
| email-template | 428.1 KB | 177.4 KB |
| error-details | 443.9 KB | 177.4 KB |
| error-issue | 428.9 KB | 177.4 KB |
| error-issue-list | 542.5 KB | 177.4 KB |
| experiment | 539.9 KB | 177.4 KB |
| experiment-list | 543.4 KB | 177.4 KB |
| experiment-results | 541.6 KB | 177.4 KB |
| feature-flag | 545.4 KB | 177.4 KB |
| feature-flag-list | 549.1 KB | 177.4 KB |
| feature-flag-testing | 432.1 KB | 177.4 KB |
| insight-actors | 540.6 KB | 177.4 KB |
| invite-email-preview | 427.5 KB | 177.4 KB |
| llm-costs | 537.9 KB | 177.4 KB |
| session-recording | 429.9 KB | 177.4 KB |
| session-summary | 435.3 KB | 177.4 KB |
| survey | 429.7 KB | 177.4 KB |
| survey-global-stats | 540.7 KB | 177.4 KB |
| survey-list | 543.3 KB | 177.4 KB |
| survey-stats | 540.6 KB | 177.4 KB |
| trace-span | 428.6 KB | 177.4 KB |
| trace-span-list | 542.6 KB | 177.4 KB |
| workflow | 428.6 KB | 177.4 KB |
| workflow-list | 542.0 KB | 177.4 KB |
| query-results | 723.5 KB | 177.4 KB |
| render-ui | 613.8 KB | 177.4 KB |
| visual-review-snapshots | 433.0 KB | 177.4 KB |
|
👋 Visual changes detected for this PR. Review and approve in PostHog Visual Review If these changes are unexpected, they may be caused by a flaky test or a broken snapshot on master. Don't approve — rerun the job or wait for a fix. |
…text Review feedback fixes: - Filter observed variant values to the flag's defined multivariate variant keys, mirroring the variant IN filter in build_common_exposure_conditions. Without this, a non-enrolled user's `$feature_flag_called` event (`$feature_flag_response: false`) surfaced as a bogus variant named "false". Verified against production data. Regression test added. - Replace the test-tube emoji in the player meta chip with IconFlask, the experiments product icon (matches the inspector row icon). - Clarify that the inspector variant-assignment marker is synthesized client-side from the session_context response, not a backend event. Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
🦔 Hogbox preview · ❌ build failedThe preview didn't come up for commit |
|
Reviews (1): Last reviewed commit: "fix(experiments): only surface defined v..." | Re-trigger Greptile |
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 3 · PR risk: 0/10 |
Addresses two review-bot findings: - The candidate experiment cap (50) could silently and nondeterministically drop matching experiments — the unordered queryset slice happened before checking which flags the session saw. Now the exposure-events query runs unfiltered first, so any flag the session verifiably called rescues its experiment past the cap; the cap only bounds the stamped-property query's column width and slices newest-first for determinism. Verified against production data (the cap is exceeded on real projects today; the capped query performs fine at full width). - Feature flags can arrive after the player mounts, and the loader only checked the flag once in afterMount. A kea subscription on a derived enabled selector now retries the load when the gate flips on (same shape as maxWebAnalyticsNudgeLogic's subscription + afterMount evaluation). Regression tests for both. Generated-By: PostHog Code Task-Id: 06b4bcbe-818a-4d7b-bc64-e93038a62451
Address review findings on the session_context endpoint: - Filter candidate experiments through the caller's object-level access controls, so private experiments no longer leak into another user's session context (the queryset is passed from the view, matching how list-action ACL filtering behaves). - Require session_recording:read alongside experiment:read and gate the action on resource-level session replay access, following the replay_vision precedent for endpoints reading recording-derived data. - Apply the ClickHouse burst/sustained throttles the replay API uses, since unknown session ids fall back to an unbounded metadata scan. - Recompute the stamped-property flag keys after exposure-rescued experiments are added, so their stamped variants count toward variants_seen. Generated-By: PostHog Code Task-Id: 4d51e451-ecb1-408a-b405-19b4ec00f7dd
…config Filter the $feature_flag_called exposure query to the overlapping experiments' flag keys and their defined variant names, so its group-by cardinality is bounded by team configuration rather than event payloads. Sessions routinely call plenty of non-experiment flags, and previously those rows competed with real experiment exposures for HogQL's implicit LIMIT 100 (with no ORDER BY), which could silently and nondeterministically drop legitimate exposures in flag-heavy sessions. The key list is built from all overlapping experiments (before the candidate cap) so the beyond-cap rescue keeps working, and the limit is now an explicit generous backstop instead of the implicit 100. Generated-By: PostHog Code Task-Id: 4d51e451-ecb1-408a-b405-19b4ec00f7dd
There was a problem hiding this comment.
Great feature!
Read through the code. Well structured and easy to follow. Decisions looks sound. Focused mostly on the experiment side of things.
One behavior that could be discussed is how the exposure criteria is handled. Right now, it hard codes feature flag called and it does point out that this may differ from how the experiment handles it. We could alternatively here use the experiment exposure criteria (resolve it via the helper) and use that in the query instead.
That said, it is likely not often requested, so I think its fine to wait and rather see if its worth improving later as a follow up.
Problem
Replays can be opened from Experiments (per-variant "View recordings"), but once you land in the player all experiment context is gone
Changes
Everything user-visible is behind a new
replay-experiment-contextfeature flag - the player is unchanged until it's flipped.GET /api/projects/:id/experiments/session_context/?session_id=...resolves which launched experiments (and variants) a session recording saw. It merges$feature_flag_calledexposure events with stamped$feature/<key>properties (covers the posthog-js dedupe gap where a returning user's session has no exposure event), flags sessions that saw multiple variants of one flag, only counts the flag's defined variant keys (a non-enrolled user's$feature_flag_response: falsenever surfaces as a variant), and fast-paths to empty without touching ClickHouse when no launched experiment's run window overlaps the recording. Query/merge logic lives inproducts/experiments/backend/session_context.py. Requiresexperiment:read+session_recording:read, respects per-experiment access controls, is ClickHouse-throttled, and bounds the exposure query to real experiment flag keys/variants.experiment-variantinspector item per experiment at the flag-evaluation moment, rendered as a clickable seekbar tick.All copy deliberately says the session saw a variant, and the marker is named after flag evaluation, never "exposure" – flag evaluation only coincides with an experiment's exposure moment for default-exposure experiments (custom exposure configs, holdouts and the run window all differ). The variant itself always comes from the flag, so it's correct for custom-exposure experiments too.
(Known adjacent debt, deliberately untouched: the Max replay tools in
products/experiments/backend/max_tools.pyre-implement session→experiment filtering in Python with$feature_flag_calledhardcoded, ignoring custom exposure configs. They could be ported to this endpoint as a follow-up)Screenshots:
How did you test this code?
false-variant bug was caught).products/experiments/backend/test/test_session_context_api.py(14 tests, run against real Postgres + ClickHouse): endpoint contract (400 withoutsession_id, 404 for missing recordings), variant resolution from exposure events and from stamped properties (regression: dropping either source silently loses context for returning users), non-enrollment flag responses ignored (regression:falsesurfacing as a variant), multi-variant detection, non-experiment flags excluded from results, run-window overlap fast path, and team isolation (another team's recording 404s, another team's experiments never appear – the IDOR case), plus authorization (scope, replay access, private experiments) and non-experiment-flag noise cases.sessionRecordingExperimentContextLogic.test.ts(3 tests): selector derivation from the response, empty response ⇒ no context, and no fetch when the feature flag is off (pins the dark-launch guarantee).playerInspectorLogic.test.ts(2 new tests): a timestamped context item yields exactly oneexperiment-variantinspector item and one seekbar tick while a null-timestamp item yields none, and flag off yields no markers.typescript:check, frontend format, ruff,hogli ci:preflight --fix, and an OpenAPI drift check (regenerated output identical to committed) all clean. (Review-feedback commits verified with ruff, preflight, and CI.)Automatic notifications
Docs update
None yet – the feature is dark behind the
replay-experiment-contextflag; docs should land with the flag rollout.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/improving-drf-endpoints,/writing-tests,/adopting-generated-api-types,/writing-kea-logics.seekbarItemsfilters byitem.type), so the agent took the plan's explicitly-allowed alternative – widening theseekbarItemsfilter andPlayerSeekbarTicksprop types.Created with PostHog Code