Skip to content
Permalink
Browse files

Merge tag 'v0.99.1'

Synapse 0.99.1 (2019-02-14)
===========================

Features
--------

- Include m.room.encryption on invites by default ([\#3902](#3902))
- Federation OpenID listener resource can now be activated even if federation is disabled ([\#4420](#4420))
- Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. ([\#4522](#4522))
- Add ability to update backup versions ([\#4580](#4580))
- Allow the "unavailable" presence status for /sync.
  This change makes Synapse compliant with r0.4.0 of the Client-Server specification. ([\#4592](#4592))
- There is no longer any need to specify `no_tls`: it is inferred from the absence of TLS listeners ([\#4613](#4613), [\#4615](#4615), [\#4617](#4617), [\#4636](#4636))
- The default configuration no longer requires TLS certificates. ([\#4614](#4614))

Bugfixes
--------

- Copy over room federation ability on room upgrade. ([\#4530](#4530))
- Fix noisy "twisted.internet.task.TaskStopped" errors in logs ([\#4546](#4546))
- Synapse is now tolerant of the `tls_fingerprints` option being None or not specified. ([\#4589](#4589))
- Fix 'no unique or exclusion constraint' error ([\#4591](#4591))
- Transfer Server ACLs on room upgrade. ([\#4608](#4608))
- Fix failure to start when not TLS certificate was given even if TLS was disabled. ([\#4618](#4618))
- Fix self-signed cert notice from generate-config. ([\#4625](#4625))
- Fix performance of `user_ips` table deduplication background update ([\#4626](#4626), [\#4627](#4627))

Internal Changes
----------------

- Change the user directory state query to use a filtered call to the db instead of a generic one. ([\#4462](#4462))
- Reject federation transactions if they include more than 50 PDUs or 100 EDUs. ([\#4513](#4513))
- Reduce duplication of ``synapse.app`` code. ([\#4567](#4567))
- Fix docker upload job to push -py2 images. ([\#4576](#4576))
- Add port configuration information to ACME instructions. ([\#4578](#4578))
- Update MSC1711 FAQ to calrify .well-known usage ([\#4584](#4584))
- Clean up default listener configuration ([\#4586](#4586))
- Clarifications for reverse proxy docs ([\#4607](#4607))
- Move ClientTLSOptionsFactory init out of `refresh_certificates` ([\#4611](#4611))
- Fail cleanly if listener config lacks a 'port' ([\#4616](#4616))
- Remove redundant entries from docker config ([\#4619](#4619))
- README updates ([\#4621](#4621))
  • Loading branch information...
richvdh committed Feb 14, 2019
2 parents 4588b0d + 06cd757 commit 00cf679bf22737b6b9028ba0dada47e029fc0925
Showing with 1,312 additions and 555 deletions.
  1. +46 −0 CHANGES.md
  2. +29 −12 INSTALL.md
  3. +52 −98 README.rst
  4. +0 −1 changelog.d/4576.misc
  5. +0 −1 changelog.d/4578.misc
  6. +0 −1 changelog.d/4584.misc
  7. +10 −0 debian/changelog
  8. +1 −0 debian/matrix-synapse.service
  9. +0 −17 docker/conf/dummy.tls.crt
  10. +1 −7 docker/conf/homeserver.yaml
  11. +14 −8 docs/MSC1711_certificates_FAQ.md
  12. +94 −0 docs/reverse_proxy.rst
  13. +2 −3 docs/workers.rst
  14. +2 −2 synapse/__init__.py
  15. +1 −0 synapse/api/constants.py
  16. +99 −12 synapse/app/_base.py
  17. +1 −6 synapse/app/appservice.py
  18. +1 −12 synapse/app/client_reader.py
  19. +1 −12 synapse/app/event_creator.py
  20. +13 −13 synapse/app/federation_reader.py
  21. +1 −11 synapse/app/federation_sender.py
  22. +1 −12 synapse/app/frontend_proxy.py
  23. +80 −85 synapse/app/homeserver.py
  24. +1 −12 synapse/app/media_repository.py
  25. +1 −2 synapse/app/pusher.py
  26. +1 −6 synapse/app/synchrotron.py
  27. +1 −12 synapse/app/user_dir.py
  28. +2 −7 synapse/config/_base.py
  29. +2 −0 synapse/config/api.py
  30. +1 −1 synapse/config/homeserver.py
  31. +5 −11 synapse/config/logger.py
  32. +125 −56 synapse/config/server.py
  33. +97 −44 synapse/config/tls.py
  34. +1 −3 synapse/crypto/context_factory.py
  35. +16 −0 synapse/federation/federation_server.py
  36. +95 −42 synapse/federation/transport/server.py
  37. +69 −5 synapse/handlers/e2e_room_keys.py
  38. +11 −0 synapse/handlers/room.py
  39. +17 −4 synapse/http/matrixfederationclient.py
  40. +1 −1 synapse/python_dependencies.py
  41. +34 −0 synapse/rest/client/v2_alpha/room_keys.py
  42. +1 −1 synapse/rest/client/v2_alpha/sync.py
  43. +1 −0 synapse/rest/client/versions.py
  44. +8 −1 synapse/server.py
  45. +21 −6 synapse/storage/_base.py
  46. +84 −3 synapse/storage/client_ips.py
  47. +21 −0 synapse/storage/e2e_room_keys.py
  48. +7 −3 synapse/storage/schema/delta/53/user_ips_index.sql
  49. +27 −11 synapse/storage/state.py
  50. +12 −4 synapse/storage/user_directory.py
  51. +1 −1 tests/app/test_frontend_proxy.py
  52. +119 −0 tests/app/test_openid_listener.py
  53. +7 −3 tests/config/test_tls.py
  54. +72 −0 tests/handlers/test_e2e_room_keys.py
  55. +1 −3 tests/http/test_fedclient.py
  56. +1 −0 tox.ini
@@ -1,3 +1,49 @@
Synapse 0.99.1 (2019-02-14)
===========================

Features
--------

- Include m.room.encryption on invites by default ([\#3902](https://github.com/matrix-org/synapse/issues/3902))
- Federation OpenID listener resource can now be activated even if federation is disabled ([\#4420](https://github.com/matrix-org/synapse/issues/4420))
- Synapse's ACME support will now correctly reprovision a certificate that approaches its expiry while Synapse is running. ([\#4522](https://github.com/matrix-org/synapse/issues/4522))
- Add ability to update backup versions ([\#4580](https://github.com/matrix-org/synapse/issues/4580))
- Allow the "unavailable" presence status for /sync.
This change makes Synapse compliant with r0.4.0 of the Client-Server specification. ([\#4592](https://github.com/matrix-org/synapse/issues/4592))
- There is no longer any need to specify `no_tls`: it is inferred from the absence of TLS listeners ([\#4613](https://github.com/matrix-org/synapse/issues/4613), [\#4615](https://github.com/matrix-org/synapse/issues/4615), [\#4617](https://github.com/matrix-org/synapse/issues/4617), [\#4636](https://github.com/matrix-org/synapse/issues/4636))
- The default configuration no longer requires TLS certificates. ([\#4614](https://github.com/matrix-org/synapse/issues/4614))


Bugfixes
--------

- Copy over room federation ability on room upgrade. ([\#4530](https://github.com/matrix-org/synapse/issues/4530))
- Fix noisy "twisted.internet.task.TaskStopped" errors in logs ([\#4546](https://github.com/matrix-org/synapse/issues/4546))
- Synapse is now tolerant of the `tls_fingerprints` option being None or not specified. ([\#4589](https://github.com/matrix-org/synapse/issues/4589))
- Fix 'no unique or exclusion constraint' error ([\#4591](https://github.com/matrix-org/synapse/issues/4591))
- Transfer Server ACLs on room upgrade. ([\#4608](https://github.com/matrix-org/synapse/issues/4608))
- Fix failure to start when not TLS certificate was given even if TLS was disabled. ([\#4618](https://github.com/matrix-org/synapse/issues/4618))
- Fix self-signed cert notice from generate-config. ([\#4625](https://github.com/matrix-org/synapse/issues/4625))
- Fix performance of `user_ips` table deduplication background update ([\#4626](https://github.com/matrix-org/synapse/issues/4626), [\#4627](https://github.com/matrix-org/synapse/issues/4627))


Internal Changes
----------------

- Change the user directory state query to use a filtered call to the db instead of a generic one. ([\#4462](https://github.com/matrix-org/synapse/issues/4462))
- Reject federation transactions if they include more than 50 PDUs or 100 EDUs. ([\#4513](https://github.com/matrix-org/synapse/issues/4513))
- Reduce duplication of ``synapse.app`` code. ([\#4567](https://github.com/matrix-org/synapse/issues/4567))
- Fix docker upload job to push -py2 images. ([\#4576](https://github.com/matrix-org/synapse/issues/4576))
- Add port configuration information to ACME instructions. ([\#4578](https://github.com/matrix-org/synapse/issues/4578))
- Update MSC1711 FAQ to calrify .well-known usage ([\#4584](https://github.com/matrix-org/synapse/issues/4584))
- Clean up default listener configuration ([\#4586](https://github.com/matrix-org/synapse/issues/4586))
- Clarifications for reverse proxy docs ([\#4607](https://github.com/matrix-org/synapse/issues/4607))
- Move ClientTLSOptionsFactory init out of `refresh_certificates` ([\#4611](https://github.com/matrix-org/synapse/issues/4611))
- Fail cleanly if listener config lacks a 'port' ([\#4616](https://github.com/matrix-org/synapse/issues/4616))
- Remove redundant entries from docker config ([\#4619](https://github.com/matrix-org/synapse/issues/4619))
- README updates ([\#4621](https://github.com/matrix-org/synapse/issues/4621))


Synapse 0.99.0 (2019-02-05)
===========================

@@ -350,17 +350,34 @@ Once you have installed synapse as above, you will need to configure it.

## TLS certificates

The default configuration exposes two HTTP ports: 8008 and 8448. Port 8008 is
configured without TLS; it should be behind a reverse proxy for TLS/SSL
termination on port 443 which in turn should be used for clients. Port 8448
is configured to use TLS for Federation with a self-signed or verified
certificate, but please be aware that a valid certificate will be required in
Synapse v1.0. Instructions for having Synapse automatically provision and renew federation certificates through ACME can be found at [ACME.md](docs/ACME.md).

If you would like to use your own certificates, you can do so by changing
`tls_certificate_path` and `tls_private_key_path` in `homeserver.yaml`;
alternatively, you can use a reverse-proxy. Apart from port 8448 using TLS,
both ports are the same in the default configuration.
The default configuration exposes a single HTTP port: http://localhost:8008. It
is suitable for local testing, but for any practical use, you will either need
to enable a reverse proxy, or configure Synapse to expose an HTTPS port.

For information on using a reverse proxy, see
[docs/reverse_proxy.rst](docs/reverse_proxy.rst).

To configure Synapse to expose an HTTPS port, you will need to edit
`homeserver.yaml`.

First, under the `listeners` section, uncomment the configuration for the
TLS-enabled listener. (Remove the hash sign (`#`) and space at the start of
each line). The relevant lines are like this:

```
- port: 8448
type: http
tls: true
resources:
- names: [client, federation]
```

You will also need to uncomment the `tls_certificate_path` and
`tls_private_key_path` lines under the `TLS` section. You can either point
these settings at an existing certificate and key, or you can enable Synapse's
built-in ACME (Let's Encrypt) support. Instructions for having Synapse
automatically provision and renew federation certificates through ACME can be
found at [ACME.md](docs/ACME.md).

## Registering a user

@@ -374,7 +391,7 @@ users. This can be done as follows:
```
$ source ~/synapse/env/bin/activate
$ synctl start # if not already running
$ register_new_matrix_user -c homeserver.yaml https://localhost:8448
$ register_new_matrix_user -c homeserver.yaml http://localhost:8008
New user localpart: erikj
Password:
Confirm password:
@@ -26,7 +26,6 @@ via IRC bridge at irc://irc.freenode.net/matrix.
Synapse is currently in rapid development, but as of version 0.5 we believe it
is sufficiently stable to be run as an internet-facing service for real usage!


About Matrix
============

@@ -88,18 +87,20 @@ Connecting to Synapse from a client
===================================

The easiest way to try out your new Synapse installation is by connecting to it
from a web client. The easiest option is probably the one at
https://riot.im/app. You will need to specify a "Custom server" when you log on
or register: set this to ``https://domain.tld`` if you setup a reverse proxy
following the recommended setup, or ``https://localhost:8448`` - remember to specify the
port (``:8448``) if not ``:443`` unless you changed the configuration. (Leave the identity
server as the default - see `Identity servers`_.)

If using port 8448 you will run into errors if you are using a self-signed
certificate. To overcome this, simply go to ``https://localhost:8448``
directly with your browser and accept the presented certificate. You can then
go back in your web client and proceed further. Valid federation certificates
should not have this problem.
from a web client.

Unless you are running a test instance of Synapse on your local machine, in
general, you will need to enable TLS support before you can successfully
connect from a client: see `<INSTALL.md#tls-certificates>`_.

An easy way to get started is to login or register via Riot at
https://riot.im/app/#/login or https://riot.im/app/#/register respectively.
You will need to change the server you are logging into from ``matrix.org``
and instead specify a Homeserver URL of ``https://<server_name>:8448``
(or just ``https://<server_name>`` if you are using a reverse proxy).
(Leave the identity server as the default - see `Identity servers`_.)
If you prefer to use another client, refer to our
`client breakdown <https://matrix.org/docs/projects/clients-matrix>`_.

If all goes well you should at least be able to log in, create a room, and
start sending messages.
@@ -174,9 +175,30 @@ Separately, Synapse may leak file handles if inbound HTTP requests get stuck
during processing - e.g. blocked behind a lock or talking to a remote server etc.
This is best diagnosed by matching up the 'Received request' and 'Processed request'
log lines and looking for any 'Processed request' lines which take more than
a few seconds to execute. Please let us know at #matrix-dev:matrix.org if
a few seconds to execute. Please let us know at #synapse:matrix.org if
you see this failure mode so we can help debug it, however.

Help!! Synapse eats all my RAM!
-------------------------------

Synapse's architecture is quite RAM hungry currently - we deliberately
cache a lot of recent room data and metadata in RAM in order to speed up
common requests. We'll improve this in future, but for now the easiest
way to either reduce the RAM usage (at the risk of slowing things down)
is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
variable. The default is 0.5, which can be decreased to reduce RAM usage
in memory constrained enviroments, or increased if performance starts to
degrade.

Using `libjemalloc <http://jemalloc.net/>`_ can also yield a significant
improvement in overall amount, and especially in terms of giving back RAM
to the OS. To use it, the library must simply be put in the LD_PRELOAD
environment variable when launching Synapse. On Debian, this can be done
by installing the ``libjemalloc1`` package and adding this line to
``/etc/default/matrix-synapse``::

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1


Upgrading an existing Synapse
=============================
@@ -196,12 +218,12 @@ Federation is the process by which users on different servers can participate
in the same room. For this to work, those other servers must be able to contact
yours to send messages.

The ``server_name`` in your
``homeserver.yaml`` file determines the way that other servers will reach
yours. By default, they will treat it as a hostname and try to connect to
port 8448. This is easy to set up and will work with the default configuration,
provided you set the ``server_name`` to match your machine's public DNS
hostname.
The ``server_name`` in your ``homeserver.yaml`` file determines the way that
other servers will reach yours. By default, they will treat it as a hostname
and try to connect to port 8448. This is easy to set up and will work with the
default configuration, provided you set the ``server_name`` to match your
machine's public DNS hostname, and give Synapse a TLS certificate which is
valid for your ``server_name``.

For a more flexible configuration, you can set up a DNS SRV record. This allows
you to run your server on a machine that might not have the same name as your
@@ -243,11 +265,8 @@ largest boxes pause for thought.)
Troubleshooting
---------------

You can use the federation tester to check if your homeserver is all set:
``https://matrix.org/federationtester/api/report?server_name=<your_server_name>``
If any of the attributes under "checks" is false, federation won't work.
There is also a nicer interface available from a community member at
`<https://neo.lain.haus/fed-tester>`_.
You can use the `federation tester <https://matrix.org/federationtester>`_ to
check if your homeserver is all set.

The typical failure mode with federation is that when you try to join a room,
it is rejected with "401: Unauthorized". Generally this means that other
@@ -263,6 +282,11 @@ So, things to check are:
(it should be ``_matrix._tcp.<server_name>``), and that the port and hostname
it specifies are reachable from outside your network.

Another common problem is that people on other servers can't join rooms that
you invite them to. This can be caused by an incorrectly-configured reverse
proxy: see `<docs/reverse_proxy.rst>`_ for instructions on how to correctly
configure a reverse proxy.

Running a Demo Federation of Synapses
-------------------------------------

@@ -290,7 +314,6 @@ The advantages of Postgres include:
For information on how to install and use PostgreSQL, please see
`docs/postgres.rst <docs/postgres.rst>`_.


.. _reverse-proxy:

Using a reverse proxy with Synapse
@@ -304,54 +327,7 @@ It is recommended to put a reverse proxy such as
doing so is that it means that you can expose the default https port (443) to
Matrix clients without needing to run Synapse with root privileges.

The most important thing to know here is that Matrix clients and other Matrix
servers do not necessarily need to connect to your server via the same
port. Indeed, clients will use port 443 by default, whereas servers default to
port 8448. Where these are different, we refer to the 'client port' and the
'federation port'.

All Matrix endpoints begin with ``/_matrix``, so an example nginx
configuration for forwarding client connections to Synapse might look like::

server {
listen 443 ssl;
listen [::]:443 ssl;
server_name matrix.example.com;

location /_matrix {
proxy_pass http://localhost:8008;
proxy_set_header X-Forwarded-For $remote_addr;
}
}

an example Caddy configuration might look like::

matrix.example.com {
proxy /_matrix http://localhost:8008 {
transparent
}
}

and an example Apache configuration might look like::

<VirtualHost *:443>
SSLEngine on
ServerName matrix.example.com;

<Location /_matrix>
ProxyPass http://127.0.0.1:8008/_matrix nocanon
ProxyPassReverse http://127.0.0.1:8008/_matrix
</Location>
</VirtualHost>

You will also want to set ``bind_addresses: ['127.0.0.1']`` and ``x_forwarded: true``
for port 8008 in ``homeserver.yaml`` to ensure that client IP addresses are
recorded correctly.

Having done so, you can then use ``https://matrix.example.com`` (instead of
``https://matrix.example.com:8448``) as the "Custom server" when `Connecting to
Synapse from a client`_.

For information on configuring one, see `<docs/reverse_proxy.rst>`_.

Identity Servers
================
@@ -409,7 +385,7 @@ Synapse Development

Before setting up a development environment for synapse, make sure you have the
system dependencies (such as the python header files) installed - see
`Installing from source`_.
`Installing from source <INSTALL.md#installing-from-source>`_.

To check out a synapse for development, clone the git repo into a working
directory of your choice::
@@ -420,7 +396,7 @@ directory of your choice::
Synapse has a number of external dependencies, that are easiest
to install using pip and a virtualenv::

virtualenv -p python2.7 env
virtualenv -p python3 env
source env/bin/activate
python -m pip install -e .[all]

@@ -462,25 +438,3 @@ sphinxcontrib-napoleon::
Building internal API documentation::

python setup.py build_sphinx


Help!! Synapse eats all my RAM!
===============================

Synapse's architecture is quite RAM hungry currently - we deliberately
cache a lot of recent room data and metadata in RAM in order to speed up
common requests. We'll improve this in future, but for now the easiest
way to either reduce the RAM usage (at the risk of slowing things down)
is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
variable. The default is 0.5, which can be decreased to reduce RAM usage
in memory constrained enviroments, or increased if performance starts to
degrade.

Using `libjemalloc <http://jemalloc.net/>`_ can also yield a significant
improvement in overall amount, and especially in terms of giving back RAM
to the OS. To use it, the library must simply be put in the LD_PRELOAD
environment variable when launching Synapse. On Debian, this can be done
by installing the ``libjemalloc1`` package and adding this line to
``/etc/default/matrix-synapse``::

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.

This file was deleted.

Oops, something went wrong.
@@ -1,3 +1,13 @@
matrix-synapse-py3 (0.99.1) stable; urgency=medium

[ Damjan Georgievski ]
* Added ExecReload= in service unit file to send a HUP signal

[ Synapse Packaging team ]
* New synapse release 0.99.1

-- Synapse Packaging team <packages@matrix.org> Thu, 14 Feb 2019 14:12:26 +0000

matrix-synapse-py3 (0.99.0) stable; urgency=medium

* New synapse release 0.99.0
@@ -8,6 +8,7 @@ WorkingDirectory=/var/lib/matrix-synapse
EnvironmentFile=/etc/default/matrix-synapse
ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys
ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3

This file was deleted.

Oops, something went wrong.
@@ -2,13 +2,7 @@

## TLS ##

{% if SYNAPSE_NO_TLS %}
no_tls: True

# workaround for https://github.com/matrix-org/synapse/issues/4554
tls_certificate_path: "/conf/dummy.tls.crt"

{% else %}
{% if not SYNAPSE_NO_TLS %}

tls_certificate_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.crt"
tls_private_key_path: "/data/{{ SYNAPSE_SERVER_NAME }}.tls.key"
Oops, something went wrong.

0 comments on commit 00cf679

Please sign in to comment.
You can’t perform that action at this time.