New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify recommended setup #2438

Closed
maxidor opened this Issue Sep 3, 2017 · 20 comments

Comments

Projects
None yet
9 participants
@maxidor
Copy link
Contributor

maxidor commented Sep 3, 2017

EDIT as to provide guidance to new users - Synapse devs official answer #1 and #2

TL;DR

  • CLIENT and FEDERATION ports are DIFFERENT, they do not use the same port.
  • TCP 8448 (Default, can change): Federation, HTTPS, original generated self-signed certificate, directly exposing port TCP 8448 of synapse (NO reverse proxy, NO replace certificate)
  • TCP 443: Clients, HTTPS, regular certificate (e.g. Let's Encrypt), reversed proxy to port TCP 8008 of synapse

The self-signed certificate of synapse SHOULD NOT be replaced and port 8448 should only be used for federation (server traffic) and directly exposed publicly.

For clients connections, a reverse proxy should be reachable publicly with a regular certificate (e.g. Let's Encrypt) on port 443 that goes to the port 8008 of synapse.

This is the last option in the first list below.


The community is currently discussing what is the official recommended setup for synapse in terms of ports and reverse proxying. Here are the three visions:

  1. Use a regular (e.g. LTE) certificate on port 8448 and expose that for clients and servers, replacing the self-signed generated one.
  2. Use a regular certificate on 443 reversed proxy to 8008 and same cert on 8448, replacing the self-signed generated one.
  3. Use a regular certificate on 443 reversed proxy to 8008 and keep the self-signed generated one on 8448

My personal believe is to do go for 3) but it seems many users in the community for for 1) or 2), which I believe is due to misleading documentation for synapse.

Why I think 1) and 2) occurs due to misleading doc:
First mention of 8008 and 8448 is at Configuring synapse, last paragraph which gives the first misleading advise.
It says 8448 has a self-signed certificate which is fine to test with clients but should be replaced in production but also state that it's fine for Federation. So if you want to connect clients to 8448, you should put a proper certificate - yet doesn't instruct to connect clients to it outside of a testing scope.

The next bit comes with Connecting to Synapse from a client which says again that the easiest way to try a synapse install is to connect to 8448 but totally ignores the fact it's a self-signed and will be a problem. So you're made to believe you should have changed the certificate.

Then we come to Using a reverse proxy with Synapse which says it will work nicely with clients as you expose 443 and continues by saying clients and servers don't necessarily need to connect to the same port, says and shows a nginx config of 443->8008 for clients.

Finally, several generic points:

  • Changing the certificate only matters for clients. Servers don't require anything specific.
  • 8448 doesn't work out of the box for clients (due to self-signed) but does for federation
  • 443 to a unencrypted 8xxx port is the typical setup with TLS offloading
  • 8008 is enabled by default in the config. If 1) was intended, why not just keep 8448?
  • if you use 443 for clients, you don't need to give the port when configuring, just put https://example.org - easy for users, easy for admins
  • Reverse proxy 443 to 8448 with self-signed certificate might not work out of the box, so you would go for 8008 (and double TLS/SSL would not make much sense)
  • synapse README never instructs or even recommends to use 8448 for clients in production, only for testing
  • going for 2) means managing two configuration files and two copies of the same certs after each renewal
  • The Server-Server Spec talks about caching results of looking up certificates and signing keys, which could cause downtime depending on homeservers implementation/configuiration?

Could the Matrix devs please clearly state what is the recommended setup?

@realitygaps

This comment has been minimized.

Copy link

realitygaps commented Sep 3, 2017

https://matrix.org/docs/guides/lets-encrypt.html also seems to suggest putting LE on the synapse server directly is a correct approach

@ArchangeGabriel

This comment has been minimized.

Copy link
Contributor

ArchangeGabriel commented Sep 3, 2017

I do 3). Excerpt from my config:

listeners:
  -
    port: 8448
    bind_addresses:
      - '::'
    type: http
    tls: true
    x_forwarded: false
    resources:
      - names: [federation]
        compress: false

  - port: 8008
    tls: false
    bind_addresses: ['::1','127.0.0.1']
    type: http
    x_forwarded: true
    resources:
      - names: [client]
        compress: true
@turt2live

This comment has been minimized.

Copy link
Member

turt2live commented Sep 3, 2017

I also use 3, with the difference being I reverse proxy 8448 to point to the federation_reader. It's reverse proxied to make it easier to switch targets more easily, if required.

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Sep 6, 2017

My recommendation for a "production" setup is option 3, as it means you can have a proper LetsEncrypt cert on port 443 without having to worry about updating the cert used by synapse when it expires (and how that interacts with other servers in the federation).

Certainly the README isn't clear in this respect. That's partly an artefact of the fact it tries to deal with a number of different scenarios, notably somebody trying to set up a "test" instance on a local machine, where having to mess about with reverse-proxies is somewhat overkill. I'm not sure what the best way to address the confusion would be; possibly the README should be broken up into separate documents on topics such as "setting up a local synapse for development/test purposes" and "setting up a production synapse with optional federation". Either way it could do with the attention of a proper technical author rather than the occasional half-hour to from a software dev trying to patch it up.

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Sep 6, 2017

Changing the certificate only matters for clients. Servers don't require anything specific.

Yeeeees, but as you have also noted:

The Server-Server Spec talks about caching results of looking up certificates and signing keys, which could cause downtime depending on homeservers implementation/configuiration?

yes. Synapse is currently a bit lax in this respect (#1362), but you could easily find yourself with outages if you're not quite careful about how you replace your certificates for federation.

@aventrax

This comment has been minimized.

Copy link

aventrax commented Sep 9, 2017

I use the 3) option, and I agree, docs must be coherent..

@maxidor

This comment has been minimized.

Copy link
Contributor

maxidor commented Sep 12, 2017

Also, homeserver.yaml talks about replacing the self-signed certificate. That should also be replaced with a proper indication that it should not be in production, only for testing.

# You can replace the self-signed certificate that synapse
# autogenerates on launch with your own SSL certificate + key pair
# if you like.  Any required intermediary certificates can be
# appended after the primary certificate in hierarchical order.

spantaleev added a commit to spantaleev/matrix-docker-ansible-deploy that referenced this issue Sep 23, 2017

Do not use Let's Encrypt certificate for Synapse's federation port
As described here (
matrix-org/synapse#2438 (comment)
), using own SSL certificates for the federation port is more fragile,
as renewing them could cause federation outages.

The recommended setup is to use the self-signed certificates generated
by Synapse.

On the 443 port (matrix-nginx-proxy) side, we still use the Let's Encrypt
certificates, which ensures API consumers work without having to trust
"our own CA".

Having done this, we also don't need to ever restart Synapse anymore,
as no new SSL certificates need to be applied there.

It's just matrix-nginx-proxy that needs to be restarted, and it doesn't
even need a full restart as an "nginx reload" does the job of swithing
to the new SSL certificates.
@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Sep 25, 2017

hopefully improved as of #2468

@richvdh richvdh closed this Sep 25, 2017

@amanusk

This comment has been minimized.

Copy link

amanusk commented Dec 4, 2017

Could some one please clarify how to setup the reverse proxy using LetsEncrypt with apache?

Between this issue and the readme I am quite confused what goes where.

I have setup a domain and a have a letsencrypt certificate setup with --apache.

Currently I have a file /etc/apache2/sites-available/<myserver.com>-le-ssl.conf

The content of the file is

<IfModule mod_ssl.c>
<VirtualHost *:443>
  ServerName <myserver.com>
  SSLCertificateFile /etc/letsencrypt/live/<myserver.com>/fullchain.pem
  SSLCertificateKeyFile /etc/letsencrypt/live/<myserver.com>/privkey.pem
  Include /etc/letsencrypt/options-ssl-apache.conf

  SSLProxyEngine On

  <Location /_matrix>
      ProxyPass http://localhost:8008
      ProxyPassReverse http://localhost:8008
  </Location>

</VirtualHost>
</IfModule>

configuration in homeserver.yaml (no other configuration was changed)

  - port: 8008
    tls: false
    bind_addresses: ['127.0.0.1']
    type: http

    x_forwarded: true

    resources:
      - names: [client, webclient]
        compress: true
      - names: [federation]
        compress: false

When trying to connect with <myserver.com/_matrix> I get

No Such Resource

No such child resource.

@maxidor

This comment has been minimized.

Copy link
Contributor

maxidor commented Dec 4, 2017

@amanusk In your apache config, amend as such:

  <Location /_matrix>
      ProxyPass http://localhost:8008/_matrix
      ProxyPassReverse http://localhost:8008/_matrix
  </Location>
@amanusk

This comment has been minimized.

Copy link

amanusk commented Dec 4, 2017

@maxidor, thanks!
After the change I get
Your browser approached me (at /_matrix) with the method "GET". I only allow the method HEAD here.
When accessing <myserver.com>/_matrix
but <myserver.com>/_matrix/client works

@maxidor

This comment has been minimized.

Copy link
Contributor

maxidor commented Dec 4, 2017

then it's fine. If you have more issue, you should open a dedicated issue or come in #matrix:matrix.org

@maxidor

This comment has been minimized.

Copy link
Contributor

maxidor commented Jan 23, 2018

@richvdh FYI I keep seeing several people a week that still do it the wrong way, but I don't have good ideas on how to improve this further short of a full rewrite of the README. Any ideas on that front?

@ptman

This comment has been minimized.

Copy link
Contributor

ptman commented Jan 23, 2018

I think one very concrete improvement would be:

  1. Finish dockerfile/image
  2. Use said docker image in production (ideally on matrix.org), so that it's kept up to date and free of bugs
  3. write docker-compose.yaml that sets up postgres, synapse, reverse proxy (nginx? traefik? caddy?), turn? This would show ports etc. quite clearly and would make it easy to fire up something to test with
@maxidor

This comment has been minimized.

Copy link
Contributor

maxidor commented Jan 23, 2018

@ptman This is specifically for those NOT using docker images, so from source or distro packages

@ptman

This comment has been minimized.

Copy link
Contributor

ptman commented Jan 23, 2018

many elements remain: database, synapse, reverse proxy, with config that listens on different ports and proxies traffic to synapse appropriately

@ptman

This comment has been minimized.

Copy link
Contributor

ptman commented Jan 23, 2018

maybe just add the nginx/caddy configs then?

@amanusk

This comment has been minimized.

Copy link

amanusk commented Jan 23, 2018

Adding an apache2 example config when using Lets Encrypt to the README will definitely help

@g4borg

This comment has been minimized.

Copy link

g4borg commented Aug 31, 2018

Note that in the upcloud tutorial i used to install matrix as homeserver, it also had an optional paragraph about certificate replacement for matrix to directly use LetsEncrypt from the homeserver.yaml.

For this it mentioned to set sth like:
tls_certificate_path: "/etc/letsencrypt/live/<my.domain.name>/cert.pem"

of course, this won't work anyway if your default setup uses root user access only for that folder, which it does on debian and derivatives at least. so you would have to copy the certificate around, which yet again other tutorials suggest, or worse, start synapse as root.
being lazy and paranoid as i am, this is the main reason i stayed with 3)

so yes, indeed a common doc, with clear statements of what is needed and best behaviour is a good idea. i am still - up to this point in time - unsure, whether my 8448 port for federation should have self signed certs, needs renewal, etc. or not.

btw. I knew about this issue also because I found the reverse proxy apache config here.
note on debian systems - you might also want to do

a2enmod proxy
a2enmod proxy_http
@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Aug 31, 2018

please don't follow the upcloud tutorial, for this reason.

so yes, indeed a common doc

The need for this is tracked at matrix-org/matrix.org#206

@matrix-org matrix-org locked as resolved and limited conversation to collaborators Aug 31, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.