feat: Refactor loading of snapshots #20632

benjackwhite · 2024-02-29T10:54:00Z

Problem

This is a beefier refactor than I initially intended... but i think its worth it.

Primarily I needed to be able to control the parsing better so that v3 realtime snapshots would be loaded just like blob ones.
This led me down a bit of a rabbit hole in the logic where I realised it is pretty fragile and we could make much better use of separating out loaders and selectors

Changes

Moves the initial list of sources to its own loader
Separate loader for loading specific sources which reduces to a cached obejct (no more in flight merging of data)
Selector to merge all the snapshots from the various sources (much easier to reason on and should perform better
Polling simplified as well
Changed the Player logic to react to changes in the state rather than listening to specific events (again should be easier to reason on later)
Fixed v3 realtime loading so that it parses it like blob data as it is streamed the same way
Added Etag management (so cool) which means once the browser has a cache of the realtime data, we will only transfer it again if the actual data changes

TODO

Confirm that there isn't any unneeded re computation going on for big recordings

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Still need to update the tests

posthog-bot · 2024-02-29T11:03:53Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-02-29T11:15:54Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

pauldambra

long overdue improvement...

(doesn't actually playback for me at the moment 🙈 but it's still draft so 🚀)

frontend/src/scenes/session-recordings/player/sessionRecordingPlayerLogic.ts

frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.ts

posthog-bot · 2024-02-29T11:34:28Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-02-29T11:43:43Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-02-29T11:52:27Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 0 modified, 0 deleted
webkit: 0 added, 1 modified, 0 deleted (diff for shard 2)

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-02-29T12:01:08Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 0 modified, 0 deleted
webkit: 0 added, 1 modified, 0 deleted (diff for shard 2)

Triggered by this commit.

👉 Review this PR's diff of snapshots.

github-actions · 2024-02-29T15:41:32Z

Size Change: 0 B

Total Size: 999 kB

ℹ️ View Unchanged

Filename	Size
`frontend/dist/toolbar.js`	999 kB

_{compressed-size-action}

posthog-bot · 2024-02-29T15:42:42Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-02-29T16:00:44Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-03-05T09:16:49Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 1 modified, 0 deleted (diff for shard 2)

Triggered by this commit.

👉 Review this PR's diff of snapshots.

pauldambra · 2024-03-07T17:46:53Z

@daibhin I suggested we should pick this up to get it over the line... I'll chop out a few pieces while I get my head around it to get started

e.g. #20767

daibhin · 2024-03-07T17:48:28Z

@pauldambra great idea! I was actually holding off some work on our performance timing until this went through to see if it fixed the weird P99 issues. Happy to help out

posthog-bot · 2024-03-15T07:30:57Z

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

daibhin

Did a thorough review of this and

daibhin · 2024-02-29T11:15:53Z

..._/scenes-app-insights-error-empty-states--estimated-query-execution-time-too-long--light.png

@PostHog/team-product-analytics this snapshot seems to be flapping on quite a few PRs this week

frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.ts

daibhin · 2024-04-02T15:02:26Z

frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.ts

+                // TODO: Do a proper check for all sources having been loaded
                return (


Could we check that the number of sources and the snapshots by source counts are the same e.g.

Suggested change

// TODO: Do a proper check for all sources having been loaded

return (

// TODO: Do a proper check for all sources having been loaded

return (

snapshotSources.length === Object.keys(snapshotsBySource).length

daibhin · 2024-04-02T15:10:40Z

frontend/src/scenes/session-recordings/__mocks__/recording_snapshots.ts

I've shipped some smaller parts of this change in previous PRs and did a thorough review today. It's pretty difficult to review 100% but I'm fairly confident it does what it's intended to. My plan is to test it locally but think it's probably worth shipping at this point and following up with any fixes forward we need.

@benjackwhite is there anything you'd like to do before we ship? There's still a few TODOs knocking about

I just don't have any spare capacity for it so its fully in your hands

frontend/src/scenes/session-recordings/__mocks__/recording_snapshots.ts

pauldambra · 2024-04-16T12:46:13Z

frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.test.ts

            ])
        })

-        it('can start polling for snapshots', async () => {
+        it('polls up to a max threshold', async () => {


🤯 I've just realised that the polling shouldn't trigger loading state in the logic - we don't want the loading affordances in the UI when we poll.

Totally separate to this, reading the code back made me realise

pauldambra · 2024-04-16T12:47:36Z

frontend/src/scenes/session-recordings/player/sessionRecordingDataLogic.ts

+                    const params = {
+                        source: source.source,
+                        blob_key: source.blob_key,
+                        version: values.featureFlags[FEATURE_FLAGS.SESSION_REPLAY_V3_INGESTION_PLAYBACK] ? '3' : '2',


and I think we can delete all the v3 stuff since that experiment is over

pauldambra

ran locally and I can load blob and realtime sources

daibhin · 2024-04-17T14:09:56Z

Haven't seen any reports of this causing issues 🥳 Thanks for taking it on @benjackwhite

feat: Refactor loading of snapshots

a633392

benjackwhite requested review from pauldambra and daibhin February 29, 2024 10:54

github-actions bot added 2 commits February 29, 2024 11:03

Update UI snapshots for chromium (2)

ab797d2

Update query snapshots

4d1c7a7

Update UI snapshots for chromium (2)

6543e07

pauldambra reviewed Feb 29, 2024

View reviewed changes

github-actions bot added 2 commits February 29, 2024 11:23

Update query snapshots

3a4c063

Update UI snapshots for chromium (2)

93353b5

Update UI snapshots for chromium (2)

1a7ed41

Update UI snapshots for webkit (2)

ce82531

github-actions bot and others added 2 commits February 29, 2024 11:59

Update UI snapshots for webkit (2)

f53b69c

Fixes

2e95887

benjackwhite removed the request for review from daibhin February 29, 2024 12:01

benjackwhite added 2 commits February 29, 2024 15:19

Fix tests

1c33af8

Fixes

d0cfc9b

Update UI snapshots for chromium (2)

aa37022

benjackwhite and others added 3 commits February 29, 2024 16:44

Fix

bdb7775

Fixes

76d4e5b

Update UI snapshots for chromium (2)

a73dc1b

Updated logs

b8583de

pauldambra mentioned this pull request Mar 7, 2024

chore: rename method #20767

Merged

posthog-bot added the stale label Mar 15, 2024

pauldambra added waiting Prevents stale-bot from marking the PR as stale. and removed stale labels Mar 15, 2024

daibhin mentioned this pull request Mar 28, 2024

chore: renaming and moving snapshot loading #21207

Merged

daibhin added 2 commits March 28, 2024 15:02

Merge branch 'master' into fix/v3-snapshots

6d126ff

minor fixes from review

7d75dc8

daibhin approved these changes Apr 2, 2024

View reviewed changes

daibhin mentioned this pull request Apr 2, 2024

chore: etag responses #21276

Closed

daibhin added 5 commits April 4, 2024 17:20

Merge branch 'master' into fix/v3-snapshots

64f0e01

Merge branch 'master' into fix/v3-snapshots

60447d4

remove etag changes

242fdd6

remove unnecessary line break

e3307f4

fix comment

713574c

daibhin requested a review from pauldambra April 12, 2024 10:19

pauldambra reviewed Apr 16, 2024

View reviewed changes

frontend/src/scenes/session-recordings/__mocks__/recording_snapshots.ts Show resolved Hide resolved

pauldambra reviewed Apr 16, 2024

View reviewed changes

pauldambra approved these changes Apr 16, 2024

View reviewed changes

Merge branch 'master' into fix/v3-snapshots

8dc8f36

daibhin marked this pull request as ready for review April 16, 2024 14:29

daibhin merged commit 4fd7a76 into master Apr 16, 2024
105 checks passed

daibhin deleted the fix/v3-snapshots branch April 16, 2024 15:01

This was referenced Apr 16, 2024

chore: remove ingestion experiment #21583

Merged

Sprint - Apr 15 to Apr 26, 2024 #21456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Refactor loading of snapshots #20632

feat: Refactor loading of snapshots #20632

benjackwhite commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

pauldambra left a comment

posthog-bot commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

github-actions bot commented Feb 29, 2024 •

edited

posthog-bot commented Feb 29, 2024

posthog-bot commented Feb 29, 2024

posthog-bot commented Mar 5, 2024

pauldambra commented Mar 7, 2024

daibhin commented Mar 7, 2024

posthog-bot commented Mar 15, 2024

daibhin left a comment

daibhin Feb 29, 2024

daibhin Apr 2, 2024

daibhin Apr 2, 2024

benjackwhite Apr 8, 2024

pauldambra Apr 16, 2024

pauldambra Apr 16, 2024

pauldambra left a comment

daibhin commented Apr 17, 2024

		// TODO: Do a proper check for all sources having been loaded
		return (

feat: Refactor loading of snapshots #20632

feat: Refactor loading of snapshots #20632

Conversation

benjackwhite commented Feb 29, 2024

Problem

Changes

TODO

How did you test this code?

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

pauldambra left a comment

Choose a reason for hiding this comment

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

github-actions bot commented Feb 29, 2024 • edited

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Feb 29, 2024

📸 UI snapshots have been updated

posthog-bot commented Mar 5, 2024

📸 UI snapshots have been updated

pauldambra commented Mar 7, 2024

daibhin commented Mar 7, 2024

posthog-bot commented Mar 15, 2024

daibhin left a comment

Choose a reason for hiding this comment

daibhin Feb 29, 2024

Choose a reason for hiding this comment

daibhin Apr 2, 2024

Choose a reason for hiding this comment

daibhin Apr 2, 2024

Choose a reason for hiding this comment

benjackwhite Apr 8, 2024

Choose a reason for hiding this comment

pauldambra Apr 16, 2024

Choose a reason for hiding this comment

pauldambra Apr 16, 2024

Choose a reason for hiding this comment

pauldambra left a comment

Choose a reason for hiding this comment

daibhin commented Apr 17, 2024

github-actions bot commented Feb 29, 2024 •

edited