Add the ability to configure which workers will issue POST /_matrix/key/v2/query requests

[anoadragon453]

When an operation that involves verifying a remote homeserver's signature occurs on a worker, that worker will attempt to locate both the corresponding public key as well as the valid_until_ts attribute of that key. It will first check the database for a local copy, then if it can't find it will make an HTTP request to either a configured trusted_key_server, or the origin homeserver itself.

This puts a requirement for outbound federation requests from workers that would normally never otherwise need to contact other homeservers over federation. Ideally only a federation_sender (and potentially the main process) would need that ability. Therefore, it would be nice if a given set of worker names could be configured as those that should go out and fetch keys from remote locations, for which other workers not in the set would then make a HTTP replication request to perform the operation. This has implications for high-security network environments.

As a workaround, workers that syadmins do not wish to make these outbound requests can be individually configured with a trusted_key-servers entry of the local server_name. Those workers will then make a request to the load balancer, which will then be directed back to a worker that is authorised for outbound federation traffic. It is not an ideal solution however, as this would require both the original worker and the designated federation worker to perform a check against the local database, whereas an incoming replication request handler could assume that the local database had already been consulted.

[DMRobertson]

This puts a requirement for outbound federation requests from workers that would normally never otherwise need to contact other homeservers over federation. Ideally only a federation_sender (and potentially the main process) would need that ability. Therefore, it would be nice if a given set of worker names could be configured as those that should go out and fetch keys from remote locations, for which other workers not in the set would then make a HTTP replication request to perform the operation.

Synapse 1.89 added the ability to proxy outgoing federation traffic through a given set of workers, see https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#outbound_federation_restricted_to. (#15773, #15913).