Key upload seems slow on matrix.org

[jryans]

Uploading keys as part of E2E key backup seems slow to me. Here's a capture of my recent attempt on matrix.org with PII removed:

2019-01-15 20:15:48,131 - synapse.access.http.8080 - 302 - INFO - PUT-34926278 - <IP> - 8080 - {@jryans:matrix.org} Processed request: 5.185sec/0.003sec (0.100sec, 0.024sec) (0.968sec/2.317sec/403) 2B 200 "PUT /_matrix/client/r0/room_keys/keys?version=16 HTTP/1.1" <UA> [0 dbevts]
2019-01-15 20:15:51,011 - synapse.access.http.8080 - 302 - INFO - PUT-34926997 - <IP> - 8080 - {@jryans:matrix.org} Processed request: 1.987sec/0.001sec (0.080sec, 0.048sec) (0.407sec/0.925sec/403) 2B 200 "PUT /_matrix/client/r0/room_keys/keys?version=16 HTTP/1.1" <UA> [0 dbevts]
2019-01-15 20:15:54,246 - synapse.access.http.8080 - 302 - INFO - PUT-34927329 - <IP> - 8080 - {@jryans:matrix.org} Processed request: 2.315sec/0.000sec (0.084sec, 0.036sec) (0.390sec/1.173sec/403) 2B 200 "PUT /_matrix/client/r0/room_keys/keys?version=16 HTTP/1.1" <UA> [0 dbevts]
2019-01-15 20:15:54,989 - synapse.access.http.8080 - 302 - INFO - PUT-34927390 - <IP> - 8080 - {@jryans:matrix.org} Processed request: 2.736sec/0.001sec (0.052sec, 0.004sec) (0.172sec/0.339sec/202) 2B 200 "PUT /_matrix/client/r0/room_keys/keys?version=16 HTTP/1.1" <UA> [0 dbevts]
2019-01-15 20:15:55,022 - synapse.access.http.8080 - 302 - INFO - PUT-34927676 - <IP> - 8080 - {@jryans:matrix.org} Processed request: 0.195sec/0.004sec (0.000sec, 0.004sec) (0.007sec/0.013sec/3) 2B 200 "PUT /_matrix/client/r0/room_keys/keys?version=16 HTTP/1.1" <UA> [0 dbevts]

Request body size:

• First 4 requests: ~165 KiB
• Last request: 0.9 KiB

Looking at the first request for example, we have:

• 5.185 s total processing time
• 0.968 s DB scheduling time
• 2.317 s DB txn time
• 403 DB txns

[jryans]

Each request contains up to 200 sessions to store keys for.

Looking at upload_room_keys:

synapse/synapse/handlers/e2e_room_keys.py

Lines 164 to 169 in aa955f2

	# XXX: this should/could be done concurrently, given we're in a lock.
	for room_id, room in iteritems(room_keys['rooms']):
	for session_id, session in iteritems(room['sessions']):
	yield self._upload_room_key(
	user_id, version, room_id, session_id, session
	)

we may want to do what this comment suggests and add the whole batch at once, instead of doing separate DB inserts for each session.