Reverse-proxy to Synapse via matrix-synapse-reverse-proxy-companion

Fixes #2090

CHANGELOG.md

@@ -1,3 +1,51 @@
+# 2022-11-20
+
+## (Backward Compatibility Break) Changing how reverse-proxying to Synapse works - now via a `matrix-synapse-reverse-proxy-companion` service
+
+**TLDR**: There's now a `matrix-synapse-reverse-proxy-companion` nginx service, which helps with reverse-proxying to Synapse and its various worker processes (if workers are enabled), so that `matrix-nginx-proxy` can be relieved of this role. `matrix-nginx-proxy` still remains as the public SSL-terminating reverse-proxy in the playbook. `matrix-synapse-reverse-proxy-companion` is just one more reverse-proxy thrown into the mix for convenience. People with a more custom reverse-proxying configuration may be affected - see [Webserver configuration](#webserver-configuration) below.
+
+### Background
+
+Previously, `matrix-nginx-proxy` forwarded requests to Synapse directly. When Synapse is running in worker mode, the reverse-proxying configuration is more complicated (different requests need to go to different Synapse worker processes). `matrix-nginx-proxy` had configuration for sending each URL endpoint to the correct Synapse worker responsible for handling it. However, sometimes people like to disable `matrix-nginx-proxy` (for whatever reason) as detailed in [Using your own webserver, instead of this playbook's nginx proxy](docs/configuring-playbook-own-webserver.md).
+
+Because `matrix-nginx-proxy` was so central to request forwarding, when it was disabled and Synapse was running with workers enabled, there was nothing which could forward requests to the correct place anymore.. which caused [problems such as this one affecting Dimension](https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/2090).
+
+### Solution
+
+From now on, `matrix-nginx-proxy` is relieved of its function of reverse-proxying to Synapse and its various worker processes.
+This role is now handled by the new `matrix-synapse-reverse-proxy-companion` nginx service and works even if `matrix-nginx-proxy` is disabled.
+The purpose of the new `matrix-synapse-reverse-proxy-companion` service is to:
+
+- serve as a companion to Synapse and know how to reverse-proxy to Synapse correctly (no matter if workers are enabled or not)
+
+- provide a unified container address for reaching Synapse (no matter if workers are enabled or not)
+  - `matrix-synapse-reverse-proxy-companion:8008` for Synapse Client-Server API traffic
+  - `matrix-synapse-reverse-proxy-companion:8048` for Synapse Server-Server (Federation) API traffic
+
+- simplify `matrix-nginx-proxy` configuration - it now only needs to send requests to `matrix-synapse-reverse-proxy-companion` or `matrix-dendrite`, etc., without having to worry about workers
+
+- allow reverse-proxying to Synapse, even if `matrix-nginx-proxy` is disabled
+
+`matrix-nginx-proxy` still remains as the public SSL-terminating reverse-proxy in the playbook. All traffic goes through it before reaching any of the services.
+It's just that now the Synapse traffic is routed through `matrix-synapse-reverse-proxy-companion` like this:
+
+(`matrix-nginx-proxy` -> `matrix-synapse-reverse-proxy-companion` -> (`matrix-synapse` or some Synapse worker)).
+
+Various services (like Dimension, etc.) still talk to Synapse via `matrix-nginx-proxy` (e.g. `http://matrix-nginx-proxy:12080`) preferentially. They only talk to Synapse via the reverse-proxy companion (e.g. `http://matrix-synapse-reverse-proxy-companion:8008`) if `matrix-nginx-proxy` is disabled. Services should not be talking to Synapse (e.g. `https://matrix-synapse:8008` directly anymore), because when workers are enabled, that's the Synapse `master` process and may not be serving all URL endpoints needed by the service.
+
+### Webserver configuration
+
+- if you're using `matrix-nginx-proxy` (`matrix_nginx_proxy_enabled: true`, which is the default for the playbook), you don't need to do anything
+
+- if you're using your own `nginx` webserver running on the server, you shouldn't be affected. The `/matrix/nginx/conf.d` configuration and exposed ports that you're relying on will automatically be updated in a way that should work
+
+- if you're using another local webserver (e.g. Apache, etc.) and haven't changed any ports (`matrix_*_host_bind_port` definitions), you shouldn't be affected. You're likely sending Matrix traffic to `127.0.0.1:8008` and `127.0.0.1:8048`. These ports (`8008` and `8048`) will still be exposed on `127.0.0.1` by default - just not by the `matrix-synapse` container from now on, but by the `matrix-synapse-reverse-proxy-companion` container instead
+
+- if you've been exposing `matrix-synapse` ports (`matrix_synapse_container_client_api_host_bind_port`, etc.) manually, you should consider exposing `matrix-synapse-reverse-proxy-companion` ports instead
+
+- if you're running Traefik and reverse-proxying directly to the `matrix-synapse` container, you should start reverse-proxying to the `matrix-synapse-reverse-proxy-companion` container instead. See [our updated Traefik example configuration](docs/configuring-playbook-own-webserver.md#sample-configuration-for-running-behind-traefik-20). Note: we now recommend calling the federation entry point `federation` (instead of `synapse`) and reverse-proxying the federation traffic via `matrix-nginx-proxy`, instead of sending it directly to Synapse (or `matrix-synapse-reverse-proxy-companion`). This makes the configuration simpler.
+
+
 # 2022-11-05
 
 ## (Backward Compatibility Break) A new default standalone mode for Etherpad

docs/configuring-playbook-email2matrix.md

@@ -70,7 +70,7 @@ matrix_email2matrix_matrix_mappings:
     SkipMarkdown: true
 ```
 
-You can also set `MatrixHomeserverUrl` to `http://matrix-synapse:8008`, instead of the public `https://matrix.DOMAIN`.
+You can also set `MatrixHomeserverUrl` to `http://matrix-synapse-reverse-proxy-companion:8008`, instead of the public `https://matrix.DOMAIN`.
 However, that's more likely to break in the future if you switch to another server implementation than Synapse.
 
 Re-run the playbook (`--tags=setup-email2matrix,start`) and try sending an email to `my-mailbox@matrix.DOMAIN`.

docs/configuring-playbook-federation.md

@@ -46,6 +46,9 @@ matrix_synapse_federation_port_enabled: false
 
 # This removes the `8448` virtual host from the matrix-nginx-proxy reverse-proxy server.
 matrix_nginx_proxy_proxy_matrix_federation_api_enabled: false
+
+# This stops the federation port on the synapse-reverse-proxy-companion side (normally `matrix-synapse-reverse-proxy-companion:8048` on the container network).
+matrix_synapse_reverse_proxy_companion_federation_api_enabled: false
 ```
 
 ## Changing the federation port from 8448 to a different port to use a CDN that only accepts 443/80 ports

docs/configuring-playbook-own-webserver.md

@@ -40,8 +40,8 @@ No matter which external webserver you decide to go with, you'll need to:
 
       Here are the variables required for the default configuration (Synapse and Element)
        ```
-        matrix_synapse_container_client_api_host_bind_port: '0.0.0.0:8008'
-        matrix_synapse_container_federation_api_plain_host_bind_port: '0.0.0.0:8048'
+		matrix_synapse_reverse_proxy_companion_container_client_api_host_bind_port: '0.0.0.0:8008'
+		matrix_synapse_reverse_proxy_companion_container_federation_api_host_bind_port: '0.0.0.0:8048'
         matrix_client_element_container_http_host_bind_port: "0.0.0.0:8765"
        ```
 
@@ -172,31 +172,24 @@ matrix_nginx_proxy_container_extra_arguments:
 
   # The Nginx proxy container will receive traffic from these subdomains
   - '--label "traefik.http.routers.matrix-nginx-proxy.rule=Host(`{{ matrix_server_fqn_matrix }}`,`{{ matrix_server_fqn_element }}`,`{{ matrix_server_fqn_dimension }}`,`{{ matrix_server_fqn_jitsi }}`)"'
-
   # (The 'web-secure' entrypoint must bind to port 443 in Traefik config)
   - '--label "traefik.http.routers.matrix-nginx-proxy.entrypoints=web-secure"'
-
   # (The 'default' certificate resolver must be defined in Traefik config)
   - '--label "traefik.http.routers.matrix-nginx-proxy.tls.certResolver=default"'
-
   # The Nginx proxy container uses port 8080 internally
   - '--label "traefik.http.services.matrix-nginx-proxy.loadbalancer.server.port=8080"'
 
-matrix_synapse_container_extra_arguments:
-  # May be unnecessary depending on Traefik config, but can't hurt
-  - '--label "traefik.enable=true"'
-
-  # The Synapse container will receive traffic from this subdomain
-  - '--label "traefik.http.routers.matrix-synapse.rule=Host(`{{ matrix_server_fqn_matrix }}`)"'
-
-  # (The 'synapse' entrypoint must bind to port 8448 in Traefik config)
-  - '--label "traefik.http.routers.matrix-synapse.entrypoints=synapse"'
-
+  # Federation
+  - '--label "traefik.http.routers.matrix-nginx-proxy-federation.rule=Host(`{{ matrix_server_fqn_matrix }}`)"'
+  # (The 'federation' entrypoint must bind to port 8448 in Traefik config)
+  - '--label "traefik.http.routers.matrix-nginx-proxy-federation.entrypoints=federation"'
   # (The 'default' certificate resolver must be defined in Traefik config)
-  - '--label "traefik.http.routers.matrix-synapse.tls.certResolver=default"'
+  - '--label "traefik.http.routers.matrix-nginx-proxy-federation.tls.certResolver=default"'
+  # The Nginx proxy container uses port `matrix_nginx_proxy_proxy_matrix_federation_port (8448) internally
+  - '--label "traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.port={{ matrix_nginx_proxy_proxy_matrix_federation_port }}"'
+  - '--label "traefik.http.services.matrix-nginx-proxy-federation.loadbalancer.server.scheme={{ 'https' if matrix_nginx_proxy_https_enabled else 'http' }}"'
 
-  # The Synapse container uses port 8048 internally
-  - '--label "traefik.http.services.matrix-synapse.loadbalancer.server.port=8048"'
+matrix_synapse_reverse_proxy_companion_container_labels_traefik_enabled: true
 ```
 
 This method uses labels attached to the Nginx and Synapse containers to provide the Traefik Docker provider with the information it needs to proxy `matrix.DOMAIN`, `element.DOMAIN`, `dimension.DOMAIN` and `jitsi.DOMAIN`. Some [static configuration](https://docs.traefik.io/v2.0/reference/static-configuration/file/) is required in Traefik; namely, having endpoints on ports 443 and 8448 and having a certificate resolver.
@@ -240,7 +233,7 @@ services:
       - "--providers.docker.network=traefik"
       - "--providers.docker.exposedbydefault=false"
       - "--entrypoints.web-secure.address=:443"
-      - "--entrypoints.synapse.address=:8448"
+      - "--entrypoints.federation.address=:8448"
       - "--certificatesresolvers.default.acme.tlschallenge=true"
       - "--certificatesresolvers.default.acme.email=YOUR EMAIL"
       - "--certificatesresolvers.default.acme.storage=/letsencrypt/acme.json"

examples/caddy/matrix-synapse

@@ -21,11 +21,11 @@ https://matrix.DOMAIN {
 	}
 
 	# Synapse Client<>Server API
-	proxy /_matrix matrix-synapse:8008 {
+	proxy /_matrix matrix-synapse-reverse-proxy-companion:8008 {
 		transparent
 		except /_matrix/identity/ /_matrix/client/r0/user_directory/search
 	}
-	proxy /_synapse/client matrix-synapse:8008 {
+	proxy /_synapse/client matrix-synapse-reverse-proxy-companion:8008 {
 		transparent
 	}
 }