Re-introduce the outbound federation proxy #15913

MadLittleMods · 2023-07-10T21:19:54Z

Originally merged in #15773, reverted in #15910. This PR re-introduces the change alongside some fixes

Allow configuring the set of workers to proxy outbound federation traffic through (outbound_federation_restricted_to).

This is useful when you have a worker setup with federation_sender instances responsible for sending outbound federation requests and want to make sure all outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the federation_sender's to contact the outside world.

The original code is from @erikjohnston's branches which I've gotten in-shape to merge.

Details of the additional fixes in the re-introduction PR:

Adds Proxy-Authorization for outbound federation proxying between workers. This fixes the security issue that caused us to revert last time where Synapse could be used as an open proxy. This is the main thing to make sure we're doing correctly.
Since Unix Sockets for HTTP Replication #15708 was merged in between us reverting and re-introducing now, we also need to support inter-worker communication via Unix sockets.

Fix https://github.com/matrix-org/internal-config/issues/1448

Todo

Fix open proxy security problem, https://github.com/matrix-org/internal-config/issues/1448
Incorporate Fix TLS in the federation outbound proxy #15886 directly in this PR
Once Unix Sockets for HTTP Replication #15708 merges, include cd1579b

Dev notes

SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.http.test_matrixfederationclient.FederationClientProxyTests

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

@erikjohnston

Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.

@realtyem

Same fix as #15746 Thanks to @realtyem for pointing it out! `creatorForNetloc(...)` doesn't come with typing and expects `host` to be `bytes` instead of a `str`. `ProxyAgent` was introduced with the federation outbound proxy: #15773

Originally from #15708

…another worker

MadLittleMods · 2023-07-12T08:14:27Z

synapse/config/workers.py

+            if not self.worker_replication_secret:
+                raise ConfigError(
+                    "`worker_replication_secret` must be configured when using `outbound_federation_restricted_to`."
+                )


Are we okay with re-using worker_replication_secret for use with Proxy-Authorization between workers? It's still inter-worker communication but it's not replication traffic.

MadLittleMods · 2023-07-12T22:17:56Z

changelog.d/15913.feature

For reviewers, here are all the new changes that are different from the original PR -> https://github.com/matrix-org/synapse/pull/15913/files/7ad4cfc6a056f42a884803072cd0440db07bdc69..4e49e661577ea4d0caf55a159df119d304121215. At the top in the PR description, there is a section that describes the "Details of the additional fixes in the re-introduction PR"

If you're looking for context/discussion on any previous code, #15773 might have some answers.

erikjohnston

Woo, looks good!

Introduced in #15913

@wrjlewis

No significant changes since 1.89.0rc1. - Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924)) - Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911)) - Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912)) - Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969)) - Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929)) - Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820)) - Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887)) - Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925)) - Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957)) - Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960)) - Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973)) - Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975)) - Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921)) - Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938)) - Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981)) - Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928)) - Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884)) - Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909)) - Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922)) - Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926)) - Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958)) - Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940)) - Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952)) - Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955)) - Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961)) - Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968)) - Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971)) * Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949)) * Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984)) * Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943)) * Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948)) * Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986)) * Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945)) * Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946)) * Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834)) * Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951)) * Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985)) * Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950)) * Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932)) * Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983)) * Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947))

@wrjlewis

No significant changes since 1.89.0rc1. - Add Unix Socket support for HTTP Replication Listeners. [Document and provide usage instructions](https://matrix-org.github.io/synapse/v1.89/usage/configuration/config_documentation.html#listeners) for utilizing Unix sockets in Synapse. Contributed by Jason Little. ([\matrix-org#15708](matrix-org#15708), [\matrix-org#15924](matrix-org#15924)) - Allow `+` in Matrix IDs, per [MSC4009](matrix-org/matrix-spec-proposals#4009). ([\matrix-org#15911](matrix-org#15911)) - Support room version 11 from [MSC3820](matrix-org/matrix-spec-proposals#3820). ([\matrix-org#15912](matrix-org#15912)) - Allow configuring the set of workers to proxy outbound federation traffic through via `outbound_federation_restricted_to`. ([\matrix-org#15913](matrix-org#15913), [\matrix-org#15969](matrix-org#15969)) - Implement [MSC3814](matrix-org/matrix-spec-proposals#3814), dehydrated devices v2/shrivelled sessions and move [MSC2697](matrix-org/matrix-spec-proposals#2697) behind a config flag. Contributed by Nico from Famedly, H-Shay and poljar. ([\matrix-org#15929](matrix-org#15929)) - Fix a long-standing bug where remote invites weren't correctly pushed. ([\matrix-org#15820](matrix-org#15820)) - Fix background schema updates failing over a large upgrade gap. ([\matrix-org#15887](matrix-org#15887)) - Fix a bug introduced in 1.86.0 where Synapse starting with an empty `experimental_features` configuration setting. ([\matrix-org#15925](matrix-org#15925)) - Fixed deploy annotations in the provided Grafana dashboard config, so that it shows for any homeserver and not just matrix.org. Contributed by @wrjlewis. ([\matrix-org#15957](matrix-org#15957)) - Ensure a long state res does not starve CPU by occasionally yielding to the reactor. ([\matrix-org#15960](matrix-org#15960)) - Properly handle redactions of creation events. ([\matrix-org#15973](matrix-org#15973)) - Fix a bug where resyncing stale device lists could block responding to federation transactions, and thus delay receiving new data from the remote server. ([\matrix-org#15975](matrix-org#15975)) - Better clarify how to run a worker instance (pass both configs). ([\matrix-org#15921](matrix-org#15921)) - Improve [the documentation](https://matrix-org.github.io/synapse/v1.89/admin_api/user_admin_api.html#login-as-a-user) for the login as a user admin API. ([\matrix-org#15938](matrix-org#15938)) - Fix broken Arch Linux package link. Contributed by @SnipeXandrej. ([\matrix-org#15981](matrix-org#15981)) - Remove support for calling the `/register` endpoint with an unspecced `user` property for application services. ([\matrix-org#15928](matrix-org#15928)) - Mark `get_user_in_directory` private since it is only used in tests. Also remove the cache from it. ([\matrix-org#15884](matrix-org#15884)) - Document which Python version runs on a given Linux distribution so we can more easily clean up later. ([\matrix-org#15909](matrix-org#15909)) - Add details to warning in log when we fail to fetch an alias. ([\matrix-org#15922](matrix-org#15922)) - Remove unneeded `__init__`. ([\matrix-org#15926](matrix-org#15926)) - Fix bug with read/write lock implementation. This is currently unused so has no observable effects. ([\matrix-org#15933](matrix-org#15933), [\matrix-org#15958](matrix-org#15958)) - Unbreak the nix development environment by pinning the Rust version to 1.70.0. ([\matrix-org#15940](matrix-org#15940)) - Update presence metrics to differentiate remote vs local users. ([\matrix-org#15952](matrix-org#15952)) - Stop reading from column `user_id` of table `profiles`. ([\matrix-org#15955](matrix-org#15955)) - Build packages for Debian Trixie. ([\matrix-org#15961](matrix-org#15961)) - Reduce the amount of state we pull out. ([\matrix-org#15968](matrix-org#15968)) - Speed up updating state in large rooms. ([\matrix-org#15971](matrix-org#15971)) * Bump anyhow from 1.0.71 to 1.0.72. ([\matrix-org#15949](matrix-org#15949)) * Bump click from 8.1.3 to 8.1.6. ([\matrix-org#15984](matrix-org#15984)) * Bump cryptography from 41.0.1 to 41.0.2. ([\matrix-org#15943](matrix-org#15943)) * Bump jsonschema from 4.17.3 to 4.18.3. ([\matrix-org#15948](matrix-org#15948)) * Bump pillow from 9.4.0 to 10.0.0. ([\matrix-org#15986](matrix-org#15986)) * Bump prometheus-client from 0.17.0 to 0.17.1. ([\matrix-org#15945](matrix-org#15945)) * Bump pydantic from 1.10.10 to 1.10.11. ([\matrix-org#15946](matrix-org#15946)) * Bump pygithub from 1.58.2 to 1.59.0. ([\matrix-org#15834](matrix-org#15834)) * Bump pyo3-log from 0.8.2 to 0.8.3. ([\matrix-org#15951](matrix-org#15951)) * Bump sentry-sdk from 1.26.0 to 1.28.1. ([\matrix-org#15985](matrix-org#15985)) * Bump serde_json from 1.0.100 to 1.0.103. ([\matrix-org#15950](matrix-org#15950)) * Bump types-pillow from 9.5.0.4 to 10.0.0.1. ([\matrix-org#15932](matrix-org#15932)) * Bump types-requests from 2.31.0.1 to 2.31.0.2. ([\matrix-org#15983](matrix-org#15983)) * Bump typing-extensions from 4.5.0 to 4.7.1. ([\matrix-org#15947](matrix-org#15947)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCAAdFiEE1508oLYUKainYFJakD7OEIo53t0FAmTI2e4ACgkQkD7OEIo5 # 3t2x1RAAohu1Rmjv0mOqFR4P1YZpA5RFbYajcyq77n/ciDKSM1dqBelONqKOq2A9 # uGbVNm6rC+EFwIl5MF5TrFdsDQHvGcRgW6NpQDZ+uIUOYizjZH1g37BoNPLlGYQx # fmKG7/XqdWhSc5tHN9HsRHyHKmsndebjXoUCPKmieGZa1GLXvGwrNkWQlEpwd9Qu # mj3uewJxLFGgIIAOiplJ4UO8FaCbMD+By27hSiWtVsLT6pyav4HC2P8RQD1iv0jW # OXNHvEWyqfBPlsPOkCD4nQZrmZqa5GWLYfBm8zFgIBxNy+e33C07L4bO+QdCE86v # /SUKug/0nsp66jSZst1fM/M2ssXvjU+LNO9fqonOCZ4TiJ4i/yoa8AvmcAg5hy7C # HR9IBp9cMrQ2u1y2/knxF657AGHxgXEltgw0PDvZHowqsqoSb+5HWl0zv1wnVjMa # 2QYLKWPBk/AdlHkmC3S4/+gfVZVsT2RSBP3JUCbFyOqug9vXFvSGTfH07Lk4PDI3 # o5idBzumvyonsuC2ypkzlj49FAj21l/8DInxEpY9JcHdVncLWvu9gmLd+H7GY7H7 # ODa2gOynrsSGVH7IpOl6dpw/GH6R8ZlfHl87bFslOqVObBxquL/ODIoFOgld+MpT # YYXp+0tW564mg+AYw3+eo44JTq0lKh7eyENP3SqKN/Z8ssQL97c= # =Ar/g # -----END PGP SIGNATURE----- # gpg: Signature made Tue Aug 1 11:09:50 2023 BST # gpg: using RSA key D79D3CA0B61429A8A760525A903ECE108A39DEDD # gpg: key 903ECE108A39DEDD: new key but contains no user ID - skipped # gpg: Total number processed: 1 # gpg: w/o user IDs: 1 # gpg: Can't check signature: No public key # Conflicts: # poetry.lock # synapse/http/site.py # synapse/storage/databases/main/roommember.py

Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.

Introduced in matrix-org#15913

Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.

Introduced in matrix-org#15913

MadLittleMods added A-Federation Z-Time-Tracked Element employees should track their time spent on this issue/PR. labels Jul 10, 2023

github-actions bot deployed to PR Documentation Preview July 10, 2023 21:20 View deployment

MadLittleMods changed the title ~~Re-introduce the outbound outbound federation proxy (#15773)~~ Re-introduce the outbound federation proxy (#15773) Jul 10, 2023

MadLittleMods added 2 commits July 10, 2023 17:19

Update changelog

06107b8

Fix TLS when using the ProxyAgent

f067e6a

Same fix as #15746 Thanks to @realtyem for pointing it out! `creatorForNetloc(...)` doesn't come with typing and expects `host` to be `bytes` instead of a `str`. `ProxyAgent` was introduced with the federation outbound proxy: #15773

MadLittleMods mentioned this pull request Jul 10, 2023

Fix TLS in the federation outbound proxy #15886

Closed

4 tasks

github-actions bot deployed to PR Documentation Preview July 10, 2023 22:20 View deployment

MadLittleMods and others added 3 commits July 11, 2023 14:29

Add slightly better representation than built-in

4c17080

Merge branch 'develop' into madlittlemods/outbound-federation-proxy

1ac93f3

Add Unix socket support to the proxyagent

e127596

Originally from #15708

github-actions bot deployed to PR Documentation Preview July 11, 2023 19:35 View deployment

MadLittleMods added 2 commits July 12, 2023 02:05

Add authentication when proxying outbound federation traffic through …

cb7c73e

…another worker

Add wrong authentication tests

cbb536f

github-actions bot deployed to PR Documentation Preview July 12, 2023 08:11 View deployment

Remove comment that was copy pasted from above around pool_key

a7131bf

github-actions bot deployed to PR Documentation Preview July 12, 2023 08:13 View deployment

MadLittleMods commented Jul 12, 2023

View reviewed changes

MadLittleMods changed the title ~~Re-introduce the outbound federation proxy (#15773)~~ Re-introduce the outbound federation proxy Jul 12, 2023

MadLittleMods added 2 commits July 12, 2023 17:01

Better error handling

6823d77

Rewrap docs

a41d9d9

github-actions bot deployed to PR Documentation Preview July 12, 2023 22:11 View deployment

MadLittleMods commented Jul 12, 2023

View reviewed changes

MadLittleMods added 2 commits July 12, 2023 17:24

Document worker_replication_secret requirement

2183969

Fix broken link in docs

4e49e66

github-actions bot deployed to PR Documentation Preview July 12, 2023 22:47 View deployment

MadLittleMods marked this pull request as ready for review July 12, 2023 23:44

MadLittleMods requested a review from a team as a code owner July 12, 2023 23:44

erikjohnston approved these changes Jul 17, 2023

View reviewed changes

erikjohnston merged commit 1c802de into develop Jul 18, 2023
39 checks passed

erikjohnston deleted the madlittlemods/outbound-federation-proxy branch July 18, 2023 08:49

erikjohnston mentioned this pull request Jul 20, 2023

Don't log exceptions for every non-200 response #15969

Merged

erikjohnston added a commit that referenced this pull request Jul 20, 2023

Don't log exceptions for every non-200 response (#15969)

fd44053

Introduced in #15913

Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 29, 2023

Don't log exceptions for every non-200 response (matrix-org#15969)

d296299

Introduced in matrix-org#15913

Fizzadar pushed a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 31, 2023

Don't log exceptions for every non-200 response (matrix-org#15969)

d5af4de

Introduced in matrix-org#15913

This was referenced Nov 13, 2023

Add the ability to configure which workers will issue POST /_matrix/key/v2/query requests #14805

Closed

Fix outbound_federation_restricted_to docs & note when added #16628

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Re-introduce the outbound federation proxy #15913

Re-introduce the outbound federation proxy #15913

MadLittleMods commented Jul 10, 2023 •

edited

MadLittleMods Jul 12, 2023

MadLittleMods Jul 12, 2023 •

edited

erikjohnston left a comment

Re-introduce the outbound federation proxy #15913

Re-introduce the outbound federation proxy #15913

Conversation

MadLittleMods commented Jul 10, 2023 • edited

Todo

Dev notes

Pull Request Checklist

MadLittleMods Jul 12, 2023

Choose a reason for hiding this comment

MadLittleMods Jul 12, 2023 • edited

Choose a reason for hiding this comment

erikjohnston left a comment

Choose a reason for hiding this comment

MadLittleMods commented Jul 10, 2023 •

edited

MadLittleMods Jul 12, 2023 •

edited