Appservice /transactions requests are queued behind requests to the AS /users/ (and maybe other)

[Half-Shot]

I observed that after creating a room (generating a lot of transaction traffic for each created event) that the events trickled through to the appservice rather than being sent in quick succession. Having looked at the logs, I could see timeouts for the /users/ endpoint.

While I believe the AS timing out the response to the /users/ endpoint is a bug on our side, I don't think Synapse should be queuing transactions behind it.

I think this might be the result of a recent change to authenticate more requests to the AS endpoints, but that is pure speculation.

Logs:

2022-10-12 09:49:16,326 - synapse.appservice.api - 129 - WARNING - notify_interested_services-3- query_user to http://appservice-service:9995/users/%40server%3Aserver.host received 504
2022-10-12 09:49:16,344 - synapse.http.client - 446 - INFO - as-sender-5- Received response to PUT http://appservice-service:9995/transactions/8?access_token=<redacted>: 200
....
2022-10-12 09:50:16,330 - synapse.http.client - 455 - INFO - notify_interested_services-3- Error sending request to  GET http://appservice-service:9995/users/%40server%3Aserver.host?access_token=<redacted>: RequestTimedOutError 504: Timeout waiting for response from remote server
2022-10-12 09:50:16,330 - synapse.appservice.api - 129 - WARNING - notify_interested_services-3- query_user to http://appservice-service:9995/users/%40server%3Aserver.host received 504
2022-10-12 09:50:16,346 - synapse.http.client - 446 - INFO - as-sender-6- Received response to PUT http://appservice-service:9995/transactions/9?access_token=<redacted>: 200

[DMRobertson]

Any ideas when this was introduced? I guess this isn't something you can easily do a bisection on :(

[Half-Shot]

I can because it was most likely me! #12746