Chunk session recording events

[macobo]

Closes #3632, Closes #3510 and replaces https://github.com/PostHog/posthog/pull/3566/files

This should make it possible to ingest large full snapshot events which currently get silently dropped due to kafka size limits.

Base64 is used to compress the data for serialization purposes.

pytest.mock is used for clean patching methods

Checklist

• All querysets/queries filter by Organization, by Team, and by User
• Django backend tests
• Jest frontend tests
• Cypress end-to-end tests

[mariusandra]

Hey, I didn't run it (yet), but got two related questions from just reading the code:

1. "assumption: all events within a request have same session_id". I'm not sure if I'd be brave enough to make this assumption. Right now, if a session recording event comes in through posthog-js, this assumption will be true. However it's not given that this is how we'll always get events into capture.py. What if there's someone who's using the bigquery plugin to export all their events... and later imports all of them via some bulk import or API? We can't guarantee any future developments will respect this assumption.

2. What if you re-capture a compressed snapshot? Similar case to the plugin scenario, what if these compressed snapshots are somehow re-ingested, either via a bulk export or some other plugin-based scenario. Will they all get compressed again and will the frontend know how to handle them?

3. Finally, should we add some postgres indices for the JSONExtractBool(snapshot_data, 'has_full_snapshot')), etc columns?

[macobo]

Re 1, Good note. I generally don't think it's worth being super defensive in the code here for hypothetical usecases, but this is easy enough to add.

Tangent: in general our event data is not symmetrical enough to allow for easy re-ingestion. Some fields are added in or removed from a valid request - the ip url parameter, token, project_id and others. IMO this is a small fail in our API design.

This applies even more so to session recordings - the data format here is more volatile and likely to change over time, and will look completely different for e.g. an android session recording library.

Re 2 - Not quite sure what you mean re What if you re-capture a compressed snapshot? Do you mean that someone manages to pollute the database with junk data by duplicating rows somehow? Note that it's not possible to send the payloads generated here to capture again - it would re-chunk them in a different way in all likelihood. Also, deduping the underlying snapshots is mostly handled in rrweb.

Re 3 - This requires some actual measurements to determine whether the index would do anything here. There's already a index "posthog_ses_team_id_0409c4_idx" btree (team_id, "timestamp") which I think will carry most of the weight here. Will add a follow-up task for this.

[macobo]

Re 3: I now think the index would do nothing since the query does require looking through all session recording events in timerange as it returns things like start/end/duration. :)

[mariusandra]

Re tangent: totally agreed. For added side effects (documenting on the fly here), the plugin server also adds $set_once: {'initial_utm_source': 'mailchimp'} if it detects a utm_source parameter for example. This is added during ingestion (captureEE), so the last plugin won't see it, but it'll show up in the database.

Re Re 2: I mean making a plugin that effectively does if event == $snapshot then posthog.capture(the event itself) (ignore the obvious recursive race condition)... and probably not directly, but through some import/export script. I think a block to prevent rechunking is easy enough to add and should be worth it. Being defensive again, but these usecases seem not that unlikely with a growing userbase and customisable open source code...

[mariusandra]

LGTM!