Add Sliding Sync /sync/e2ee endpoint for To-Device messages
#17167
Conversation
Based on: - MSC3575: Sliding Sync (aka Sync v3): matrix-org/matrix-spec-proposals#3575 - MSC3885: Sliding Sync Extension: To-Device messages: matrix-org/matrix-spec-proposals#3885 - MSC3884: Sliding Sync Extension: E2EE: matrix-org/matrix-spec-proposals#3884
|
|
MadLittleMods
changed the title
/sync/e2eeendpoint for To-Device messages/sync/e2ee endpoint for To-Device messages
We were using the enum just to distinguish /sync v2 vs Sliding Sync /sync/e2ee so we should just make an enum for that instead of trying to glom onto the existing `sync_type` (overloading it).
synapse/handlers/sync.py
Outdated
| # TODO: Do we need to worry about these? All of this info is | ||
| # normally calculated when we `_generate_sync_entry_for_rooms()` but we | ||
| # probably don't want to do all of that work for this endpoint. | ||
| newly_joined_rooms=frozenset(), | ||
| newly_joined_or_invited_or_knocked_users=frozenset(), | ||
| newly_left_rooms=frozenset(), | ||
| newly_left_users=frozenset(), |
There was a problem hiding this comment.
Since I'm not familiar exactly with how /sync should operate or exactly what we care about, I need some advice on whether this part matters.
Does /sync/e2ee need to care about the device changes that would come from doing all of that extra work with rooms?
Perhaps device_lists (device changes) should be in their own endpoint? It seems like a lot of work but I haven't profiled or seen how fast all of things go.
There was a problem hiding this comment.
Yeah, it's important that device_lists get told about those things, which sucks a bit. I think you're right that in future we might want to reconsider bundling device_lists and e2ee, but I think for now we should keep it bundled (as that is what the Rust SDK expects).
We can always change it later once we can try it out in practice.
There was a problem hiding this comment.
Updated to include all of the room derived information ✅
As far as I can tell, we don't have tests for the scenarios that this important information is necessary for.
It feels like all the benefits of this endpoint have fallen away though.
We're doing all the same work for this new endpoint as the old sync v2 so the performance is bound to be about the same. Which also means To-Device events will probably get backed up by other events above the same. I did add a filter to only work with membership events which might help.
| Work | Sync v2 | /sync/e2ee |
|---|---|---|
_generate_sync_entry_for_account_data() |
✅ | ❌ |
_generate_sync_entry_for_rooms() |
✅ | ✅ (sets up derived data for device_lists) |
_generate_sync_entry_for_presence() |
✅ (not run for matrix.org anyway) |
❌ |
_generate_sync_entry_for_device_list() |
✅ | ✅ |
_generate_sync_entry_for_to_device() |
✅ | ✅ |
device_one_time_keys_count |
✅ | ✅ |
device_unused_fallback_key_types |
✅ | ✅ |
And it doesn't seem to relieve the Sliding Sync proxy if we're going to use the same device for sync v2 and this new endpoint. This point is being tracked in this discussion -> #17167 (comment)
There was a problem hiding this comment.
Hmm, if we can split up _generate_sync_entry_for_rooms so that we can get just the membership changes then I think that would help a lot. Though equally happy to land it as is and do follow up in a separate PR
There was a problem hiding this comment.
They're pretty gnarled and coupled at the moment. In order to decouple, instead of trying to re-use the data we pull out in _generate_sync_entry_for_rooms(), it might be easier just to do our own dedicated lookup to figure out these membership changes in our own new function.
I would definitely prefer to land this and update in a follow-up 👍
There was a problem hiding this comment.
@erikjohnston Before trying to optimize /sync/e2ee, it would be good to see just how fast/slow it goes. Traces from matrix.org on good size account would probably give us the best indication. Not sure how hard it may be to get enough to_device/device_lists traffic for a good sample
| @@ -689,43 +689,177 @@ def test_noop_sync_does_not_tightloop(self) -> None: | |||
|
|
|||
|
|
|||
| class DeviceListSyncTestCase(unittest.HomeserverTestCase): | |||
There was a problem hiding this comment.
Consolidated all of these device_lists tests spread between test_devices.py and test_sync.py to here
tests/rest/client/test_sync.py
Outdated
| class DeviceOneTimeKeysSyncTestCase(unittest.HomeserverTestCase): | ||
| """Tests regarding device one time keys (`device_one_time_keys_count`) changes.""" |
There was a problem hiding this comment.
Added tests for device_one_time_keys_count in /sync because they didn't exist before
MadLittleMods
requested review from
erikjohnston
and removed request for
erikjohnston
Conflicts: synapse/handlers/sync.py
- Added `room.account_data` and `room.presence` to avoid extra work in `_generate_sync_entry_for_rooms()` - Added a comment to the top-level `account_data` and `presence` filters that `(This is just here for good measure)` See #17167 (comment)
As suggested in #17167 (comment)
|
Thanks for the review @erikjohnston and tips/context @anoadragon453 @kegsay @Hywan @MatMaul 🐧 |
Add Sliding Sync
/sync/e2eeendpoint for To-Device messages.This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (
/syncv2). And we can avoid encryption events being backed up by the main sync response or vice-versa.Splitting To-Device messages out to its own endpoint also helps when clients need to have 2 or more sync streams open at a time, e.g a push notification process and a main process. This can cause the two processes to race to fetch the To-Device events, resulting in the need for complex synchronisation rules to ensure the token is correctly and atomically exchanged between processes. (based on the words from MSC3885)
Part of the Sliding Sync simplification. See this discussion below for why it may not be as useful as we thought before implementing.
Based on:
Dev notes
Running relevant tests:
$ SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_sliding_sync # Or without running with poetry but in the poetry environment $ poetry shell SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.rest.client.test_sync tests.rest.client.test_sliding_sync tests.rest.client.test_sendtodevice $ SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.rest.client.test_sliding_syncNotes on sharing/inheriting Twisted trial unit tests: #17167 (comment)
Putting quotes around a type is called a forward reference. (mypy)
Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.(run the linters)