Route key requests to federation senders #15121

DMRobertson · 2023-02-21T11:47:09Z

Fixes #14805, which says:

Ideally only a federation_sender (and potentially the main process) would need that ability. Therefore, it would be nice if a given set of worker names could be configured as those that should go out and fetch keys from remote locations, for which other workers not in the set would then make a HTTP replication request to perform the operation. This has implications for high-security network environments.

(emphasis mine).

I am loathe to add a config option for this; I have chosen to route all key requests via federation senders.

clokep · 2023-02-22T15:01:15Z

synapse/crypto/keyring.py

+    async def _fetch_keys(
+        self, keys_to_fetch: List[_FetchKeyRequest]
+    ) -> Dict[str, Dict[str, FetchKeyResult]]:
+        # For simplicity's sake, pick a random federation sender


I think we would want to choose the same federation sender that's talking to that instance? 🤷

That's what I thought, but federation_shard_config is a ShardedWorkerHandlingConfig rather than a class RoutableShardedWorkerHandlingConfig; only the latter defines get_instance publicly.

Personally I find this all incomprehensible.

My reading of it is that if all workers have the same federation_sender_instances config, then federation_shard_config can be upgraded to a RoutableShardedWorkerHandlingConfig, since all workers will route requests the same way.

clokep · 2023-02-22T15:02:01Z

changelog.d/15121.misc

@@ -0,0 +1 @@
+Route remote key requests via federation senders.


(As I've left this comment on #15133 and #15134 I should just copy it here...)

I think we also need to update:

docker/configure_workers_and_start.py so complement uses this

https://github.com/matrix-org/sytest/blob/d1c931af1179848526d323f3955a9f21f81d600f/lib/SyTest/Homeserver/Synapse.pm#L1257 so sytest uses this.

docs/workers.md

The upgrade notes will need amending too; I think federation senders will need to be declared in the instance_map.

This will help to break an import cycle

So I can call it from federation senders

DMRobertson · 2023-02-24T00:49:08Z

This should now be working well enough to start review.

I have one outstanding concern. If this change lands as-is, every worker that can possibly need to fetch keys will need an entry for the federation_sender in its instance map. That means adding upgrade notes, changing sytest and Synapse etc and is something to trip server admins up.

We could add a config option e.g. workers/make_key_requests_via_federation_sender, but

a) I am loathe to any new config option. Indeed, that's why I chose to exfiltrate these requests via federation senders rather than add some new complex worker class/role.

b) There is likely to be future traffic that we might want to route to the external internet via federation senders only; so we would probably end up adding similar config options to workers/make_profile_requests_via_federation_sender, resulting in further config option soup.
c)

Could use input from the team to make that call: @matrix-org/synapse-core

New config option, admins don't have to change their instance_maps
No new config option, admins likely have to change their instance_maps

MatMaul · 2023-02-24T10:00:52Z

I would favor an option, but it would be nice to make it generic. Something like workers/make_outgoing_requests_via_federation_sender.
But this only make sense when all the work is done.
Perhaps introduce an experimental option for now, and once we are confident that we cover all the requests, remove them for a generic one ?

H-Shay · 2023-02-24T16:12:27Z

My two cents is that it makes more sense to just do the second option: it means confusion and annoyance in the short term while everyone gets up to speed, rather than the endless confusion and annoyance that results from a continuously expanding and complex-ifying config.

DMRobertson · 2023-02-24T18:50:38Z

There are some unit tests that need fixing up. I would like us to decide how to proceed on this (config option or not) before fixing that up. I'll chase up on Monday.

erikjohnston · 2023-02-27T17:53:05Z

I'm in several minds about this. I'm not mad keen about the faff of ensuring that everyone has correctly set up their instance_map config. OTOH if we think that routing all federation traffic via a federation sender, then so be it.

OTOH, I'm not really that keen with this approach. It feels like we're going to play whack-a-mole with lots of the federation requests, rather than take a more systematic approach.

I wonder if we should take this opportunity to go down a "federation proxy" route, either as an external util or as a dedicated config option federation_proxy: <worker_name>. Though it still involves adding a new config option, it somehow seems less weird to me?

DMRobertson · 2023-02-27T18:56:29Z

I wonder if we should take this opportunity to go down a "federation proxy" route, either as an external util or as a dedicated config option federation_proxy: <worker_name>.

So, what, something like this?

config option to a worker/group of workers as "federation outbound proxies"
define a generic endpoint on federation outbound proxies which makes an HTTP request to another homeserver and returns the response.
change the MatrixFederationAgent in the code to use this new endpoint if the current worker is not a federation outbound proxy.
admins can configure the federation outbound proxies to be e.g. their federation senders

squahtx

Looks alright functionally.

Server admins having to update their config sounds like a lot of faff. The alternative of making it an option isn't great either. It sounded like we're exploring whether we can make a federation proxy as an alternative to this PR, is that right?

squahtx · 2023-03-02T15:15:55Z

docs/workers.md

+instance_map:
+    - federation_sender1:
+        - host: localhost
+        - port: 1001
+    - federation_sender2:
+         - host: localhost
+         - port: 1002


Suggested change

instance_map:

- federation_sender1:

- host: localhost

- port: 1001

- federation_sender2:

- host: localhost

- port: 1002

instance_map:

- federation_sender1:

- host: localhost

- port: 1001

- federation_sender2:

- host: localhost

- port: 1002

squahtx · 2023-03-02T15:38:50Z

synapse/crypto/keyring.py

+    async def _fetch_keys(
+        self, keys_to_fetch: List[_FetchKeyRequest]
+    ) -> Dict[str, Dict[str, FetchKeyResult]]:
+        # For simplicity's sake, pick a random federation sender


My reading of it is that if all workers have the same federation_sender_instances config, then federation_shard_config can be upgraded to a RoutableShardedWorkerHandlingConfig, since all workers will route requests the same way.

squahtx · 2023-03-02T15:45:38Z

synapse/replication/http/keys.py

+    We would normally return a group of FetchKeyResponse structs like the
+    normal code path does, but FetchKeyResponse holds a nacl.signing.VerifyKey
+    which is not JSON-serialisable. Instead, for each requested key we respond
+    with a boolean: `true` meaning we fetched this key, and `false` meaning we
+    didn't.
+
+    The response takes the form:
+
+        200 OK
+        {
+            "fetched_count": 1
+        }


Are these comments still accurate? I can't figure out what the endpoint returns just from the code below, but it doesn't seem to match the comment.

No, I think I had iterated since then and not updated the comment.

DMRobertson · 2023-03-02T15:58:18Z

It sounded like we're exploring whether we can make a federation proxy as an alternative to this PR, is that right?

That's right. Let me close this.

DMRobertson added the Z-Time-Tracked Element employees should track their time spent on this issue/PR. label Feb 21, 2023

clokep reviewed Feb 22, 2023

View reviewed changes

David Robertson added 2 commits February 23, 2023 23:55

Pull out _FetchKeyRequest

9a7f925

This will help to break an import cycle

Keyring: isolate the keyfetching mechanism

e0841c5

So I can call it from federation senders

DMRobertson force-pushed the dmr/key-requests-from-fed-senders branch from 37c633b to 615c55f Compare February 24, 2023 00:32

David Robertson added 4 commits February 24, 2023 00:33

Define a new replication endpoint

0919513

Add new KeyFetcher impl

cdec544

Use new KeyFetcher

8c50676

Comments

bab3b58

DMRobertson force-pushed the dmr/key-requests-from-fed-senders branch from 615c55f to 8f5f5f8 Compare February 24, 2023 00:33

David Robertson added 2 commits February 24, 2023 00:41

Test case

3a9aa53

Changelog

4e3d2e8

DMRobertson force-pushed the dmr/key-requests-from-fed-senders branch from 8f5f5f8 to 4e3d2e8 Compare February 24, 2023 00:41

Make new replication endpoint accessible in complement

15b357e

DMRobertson mentioned this pull request Feb 24, 2023

Add federation senders to instance map matrix-org/sytest#1333

Closed

DMRobertson changed the title ~~[WIP] Route key requests to federation senders~~ Route key requests to federation senders Feb 24, 2023

David Robertson added 2 commits February 24, 2023 17:18

Upgrade notes and docs

5fc4155

Complain at startup if we can't find fed senders

3452c2a

DMRobertson force-pushed the dmr/key-requests-from-fed-senders branch from e3ccb15 to 3452c2a Compare February 24, 2023 17:18

github-actions bot deployed to PR Documentation Preview February 24, 2023 17:19 View deployment

DMRobertson marked this pull request as ready for review February 24, 2023 17:20

DMRobertson requested a review from a team as a code owner February 24, 2023 17:20

Master is a fed sender if none are configured

b4517d7

github-actions bot deployed to PR Documentation Preview February 24, 2023 17:46 View deployment

squahtx reviewed Mar 2, 2023

View reviewed changes

DMRobertson closed this Mar 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Route key requests to federation senders #15121

Route key requests to federation senders #15121

DMRobertson commented Feb 21, 2023 •

edited

clokep Feb 22, 2023

DMRobertson Feb 22, 2023

squahtx Mar 2, 2023

clokep Feb 22, 2023

DMRobertson Feb 22, 2023

DMRobertson commented Feb 24, 2023

MatMaul commented Feb 24, 2023

H-Shay commented Feb 24, 2023

DMRobertson commented Feb 24, 2023

erikjohnston commented Feb 27, 2023

DMRobertson commented Feb 27, 2023

squahtx left a comment

squahtx Mar 2, 2023

squahtx Mar 2, 2023

squahtx Mar 2, 2023

DMRobertson Mar 2, 2023

DMRobertson commented Mar 2, 2023

		@@ -0,0 +1 @@
		Route remote key requests via federation senders.

Route key requests to federation senders #15121

Route key requests to federation senders #15121

Conversation

DMRobertson commented Feb 21, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson commented Feb 24, 2023

MatMaul commented Feb 24, 2023

H-Shay commented Feb 24, 2023

DMRobertson commented Feb 24, 2023

erikjohnston commented Feb 27, 2023

DMRobertson commented Feb 27, 2023

squahtx left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DMRobertson commented Mar 2, 2023

DMRobertson commented Feb 21, 2023 •

edited