Performance of device list updates could be improved (aka /login is unusably slow)

[babolivier]

I was investigating why logging in with a new device was leading up to poor performances on my account. It seems like both sending the device list update to other homeservers and other homeservers in turn hitting GET /_matrix/federation/v1/user/devices/{userId}.

Looking a bit further, it looks like both code paths end up calling _get_e2e_device_keys_txn in the database code, which seems to be pretty expensive, and is called multiple time in a row, yet isn't cached. This results in quite bad performances:

On the bottom right graph the _get_e2e_device_keys_txn transaction (in blue) is originating from sending out the m.device_list_update EDUs, while the get_devices_with_keys_by_user one (in purple) is originating from responding to GET /_matrix/federation/v1/user/devices/{userId}. Both transactions involve calling the _get_e2e_device_keys_txn function (in fact, the first one does only that).
On the bottom left graph these two transactions are colored respectively in green and purple.

[erikjohnston]

I think the next steps here are to add a cache 👍

[hifi]

In #12048 I had an issue where logging in with a new device pretty much kills synapse for a while because of the device notify on login. Just batching up and slowing down sending the notifies would help a lot.

[richvdh]

It's worth mentioning that this effectively prevents logging in with Hydrogen (cf element-hq/hydrogen-web#699)

[richvdh]

oh, and fluffychat, which seems to do the same thing

[erikjohnston]

Note that there seems to be two related but distinct problems:

1. _get_e2e_device_keys_txn is slow as mentioned in the issue description
2. notify_device_updates is slow

@richvdh ooi what were you seeing? Or was it a combination of the two?

[richvdh]

I was seeing all requests to my HS responding with a 502, due to my reverse-proxy timing out. The mean reactor tick time is about 15 seconds, which means Synapse is essentially unable to respond to anything. I think it's very busy sending out federation updates.

[richvdh]

related: #5373

[hifi]

@erikjohnston In #12048 I disabled notify_device_updates temporarily and I could login just fine without delays and Synapse was chugging along without stalling.

[erikjohnston]

#12132 will hopefully help a bit here, as it smears out sending out device list updates a bit. However, it won't help for the other issues that we're seeing.

My current thinking is that we should try and get rid of the use of store.get_users_who_share_room_with_user in notify_device_update, as that is expensive for larger accounts (both in CPU and memory terms). We can instead pull out the rooms for the user, which is enough to wake up the notifier. Then, instead of recording the device update per remote server, we store the device update per room¹. When the federation sender pulls out the device updates it can find the hosts in the room.

This does lead to the same amount of work being done (as self.store.get_users_who_share_room_with_user effectively just does get_rooms_for_user followed by a loop of get_users_in_room), but has the advantage of:

1. a lot of the process happens asynchronously, so we can return the /login (or whatever) request sooner; and
2. the federation sender can prioritise which rooms and hosts to send out to first, e.g. it could handle rooms in order of when the last message was sent, so that we quickly get device updates to servers that we talk to frequently at the expense of servers that we haven't talked to for ages.

We can also do a vaguely similar thing for presence, i.e. get rid of store.get_users_who_share_room_with_user, which should help the perf overhead of presence.

---

¹. This can be done in a backwards compatible way be recording in the DB both device list updates by server and by room at the same time, then in a future release stop doing the former and bump the minimum DB schema version.

[ara4n]

I literally can't log in any more thanks to this. It would be very much appreciated if this could be prioritised...

[erikjohnston]

1.58.0rc1 should now include the fix described in #7721 (comment), so I think this is fixed from a user perspective. Logging in on a large account still takes a lot of CPU and DB though, so we may want to investigate that, but for now I think we can close this issue.

[chagai95]

I still can not log in with an account with many rooms, I have 8 GB RAM and 8 cores, is there anything in the logs that I can keep an eye out for? I tried playing around with get_users_who_share_room_with_user and get_users_in_room but had no success. Anything else I am missing perhaps?