Skip to content

Commit

Permalink
Merge tag 'v1.36.0'
Browse files Browse the repository at this point in the history
Synapse 1.36.0 (2021-06-15)
===========================

No significant changes.

Synapse 1.36.0rc2 (2021-06-11)
==============================

Bugfixes
--------

- Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\matrix-org#10149](matrix-org#10149))
- Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\matrix-org#10163](matrix-org#10163))
- Fix a bug where Synapse could send the same presence update to a remote twice. ([\matrix-org#10165](matrix-org#10165))

Synapse 1.36.0rc1 (2021-06-08)
==============================

Features
--------

- Add new endpoint `/_matrix/client/r0/rooms/{roomId}/aliases` from Client-Server API r0.6.1 (previously [MSC2432](matrix-org/matrix-spec-proposals#2432)). ([\matrix-org#9224](matrix-org#9224))
- Improve performance of incoming federation transactions in large rooms. ([\matrix-org#9953](matrix-org#9953), [\matrix-org#9973](matrix-org#9973))
- Rewrite logic around verifying JSON object and fetching server keys to be more performant and use less memory. ([\matrix-org#10035](matrix-org#10035))
- Add new admin APIs for unprotecting local media from quarantine. Contributed by @dklimpel. ([\matrix-org#10040](matrix-org#10040))
- Add new admin APIs to remove media by media ID from quarantine. Contributed by @dklimpel. ([\matrix-org#10044](matrix-org#10044))
- Make reason and score parameters optional for reporting content. Implements [MSC2414](matrix-org/matrix-spec-proposals#2414). Contributed by Callum Brown. ([\matrix-org#10077](matrix-org#10077))
- Add support for routing more requests to workers. ([\matrix-org#10084](matrix-org#10084))
- Report OpenTracing spans for database activity. ([\matrix-org#10113](matrix-org#10113), [\matrix-org#10136](matrix-org#10136), [\matrix-org#10141](matrix-org#10141))
- Significantly reduce memory usage of joining large remote rooms. ([\matrix-org#10117](matrix-org#10117))

Bugfixes
--------

- Fixed a bug causing replication requests to fail when receiving a lot of events via federation. ([\matrix-org#10082](matrix-org#10082))
- Fix a bug in the `force_tracing_for_users` option introduced in Synapse v1.35 which meant that the OpenTracing spans produced were missing most tags. ([\matrix-org#10092](matrix-org#10092))
- Fixed a bug that could cause Synapse to stop notifying application services. Contributed by Willem Mulder. ([\matrix-org#10107](matrix-org#10107))
- Fix bug where the server would attempt to fetch the same history in the room from a remote server multiple times in parallel. ([\matrix-org#10116](matrix-org#10116))
- Fix a bug introduced in Synapse 1.33.0 which caused replication requests to fail when receiving a lot of very large events via federation. ([\matrix-org#10118](matrix-org#10118))
- Fix bug when using workers where pagination requests failed if a remote server returned zero events from `/backfill`. Introduced in 1.35.0. ([\matrix-org#10133](matrix-org#10133))

Improved Documentation
----------------------

- Clarify security note regarding hosting Synapse on the same domain as other web applications. ([\matrix-org#9221](matrix-org#9221))
- Update CAPTCHA documentation to mention turning off the verify origin feature. Contributed by @aaronraimist. ([\matrix-org#10046](matrix-org#10046))
- Tweak wording of database recommendation in `INSTALL.md`. Contributed by @aaronraimist. ([\matrix-org#10057](matrix-org#10057))
- Add initial infrastructure for rendering Synapse documentation with mdbook. ([\matrix-org#10086](matrix-org#10086))
- Convert the remaining Admin API documentation files to markdown. ([\matrix-org#10089](matrix-org#10089))
- Make a link in docs use HTTPS. Contributed by @RhnSharma. ([\matrix-org#10130](matrix-org#10130))
- Fix broken link in Docker docs. ([\matrix-org#10132](matrix-org#10132))

Deprecations and Removals
-------------------------

- Remove the experimental `spaces_enabled` flag. The spaces features are always available now. ([\matrix-org#10063](matrix-org#10063))

Internal Changes
----------------

- Tell CircleCI to build Docker images from `main` branch. ([\matrix-org#9906](matrix-org#9906))
- Simplify naming convention for release branches to only include the major and minor version numbers. ([\matrix-org#10013](matrix-org#10013))
- Add `parse_strings_from_args` for parsing an array from query parameters. ([\matrix-org#10048](matrix-org#10048), [\matrix-org#10137](matrix-org#10137))
- Remove some dead code regarding TLS certificate handling. ([\matrix-org#10054](matrix-org#10054))
- Remove redundant, unmaintained `convert_server_keys` script. ([\matrix-org#10055](matrix-org#10055))
- Improve the error message printed by synctl when synapse fails to start. ([\matrix-org#10059](matrix-org#10059))
- Fix GitHub Actions lint for newsfragments. ([\matrix-org#10069](matrix-org#10069))
- Update opentracing to inject the right context into the carrier. ([\matrix-org#10074](matrix-org#10074))
- Fix up `BatchingQueue` implementation. ([\matrix-org#10078](matrix-org#10078))
- Log method and path when dropping request due to size limit. ([\matrix-org#10091](matrix-org#10091))
- In Github Actions workflows, summarize the Sytest results in an easy-to-read format. ([\matrix-org#10094](matrix-org#10094))
- Make `/sync` do fewer state resolutions. ([\matrix-org#10102](matrix-org#10102))
- Add missing type hints to the admin API servlets. ([\matrix-org#10105](matrix-org#10105))
- Improve opentracing annotations for `Notifier`. ([\matrix-org#10111](matrix-org#10111))
- Enable Prometheus metrics for the jaeger client library. ([\matrix-org#10112](matrix-org#10112))
- Work to improve the responsiveness of `/sync` requests. ([\matrix-org#10124](matrix-org#10124))
- OpenTracing: use a consistent name for background processes. ([\matrix-org#10135](matrix-org#10135))
  • Loading branch information
bradtgmurray committed Jun 22, 2021
2 parents 40f8d5d + 1c8045f commit 71326c4
Show file tree
Hide file tree
Showing 120 changed files with 4,253 additions and 2,512 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ workflows:
- dockerhubuploadlatest:
filters:
branches:
only: master
only: [ master, main ]

commands:
docker_prepare:
Expand Down
31 changes: 31 additions & 0 deletions .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Deploy the documentation

on:
push:
branches:
- develop

workflow_dispatch:

jobs:
pages:
name: GitHub Pages
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Setup mdbook
uses: peaceiris/actions-mdbook@4b5ef36b314c2599664ca107bb8c02412548d79d # v1.1.14
with:
mdbook-version: '0.4.9'

- name: Build the documentation
run: mdbook build

- name: Deploy latest documentation
uses: peaceiris/actions-gh-pages@068dc23d9710f1ba62e86896f84735d869951305 # v3.8.0
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
keep_files: true
publish_dir: ./book
destination_dir: ./develop
10 changes: 8 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,13 @@ jobs:
if: ${{ github.base_ref == 'develop' || contains(github.base_ref, 'release-') }}
runs-on: ubuntu-latest
steps:
# Note: This and the script can be simplified once we drop Buildkite. See:
# https://github.com/actions/checkout/issues/266#issuecomment-638346893
# https://github.com/actions/checkout/issues/416
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0
- uses: actions/setup-python@v2
- run: pip install tox
- name: Patch Buildkite-specific test script
Expand Down Expand Up @@ -226,9 +232,9 @@ jobs:
- name: Run SyTest
run: /bootstrap.sh synapse
working-directory: /src
- name: Dump results.tap
- name: Summarise results.tap
if: ${{ always() }}
run: cat /logs/results.tap
run: /sytest/scripts/tap_to_gha.pl /logs/results.tap
- name: Upload SyTest logs
uses: actions/upload-artifact@v2
if: ${{ always() }}
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,6 @@ __pycache__/
/docs/build/
/htmlcov
/pip-wheel-metadata/

# docs
book/
85 changes: 85 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,88 @@
Synapse 1.36.0 (2021-06-15)
===========================

No significant changes.


Synapse 1.36.0rc2 (2021-06-11)
==============================

Bugfixes
--------

- Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149))
- Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163))
- Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))


Synapse 1.36.0rc1 (2021-06-08)
==============================

Features
--------

- Add new endpoint `/_matrix/client/r0/rooms/{roomId}/aliases` from Client-Server API r0.6.1 (previously [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432)). ([\#9224](https://github.com/matrix-org/synapse/issues/9224))
- Improve performance of incoming federation transactions in large rooms. ([\#9953](https://github.com/matrix-org/synapse/issues/9953), [\#9973](https://github.com/matrix-org/synapse/issues/9973))
- Rewrite logic around verifying JSON object and fetching server keys to be more performant and use less memory. ([\#10035](https://github.com/matrix-org/synapse/issues/10035))
- Add new admin APIs for unprotecting local media from quarantine. Contributed by @dklimpel. ([\#10040](https://github.com/matrix-org/synapse/issues/10040))
- Add new admin APIs to remove media by media ID from quarantine. Contributed by @dklimpel. ([\#10044](https://github.com/matrix-org/synapse/issues/10044))
- Make reason and score parameters optional for reporting content. Implements [MSC2414](https://github.com/matrix-org/matrix-doc/pull/2414). Contributed by Callum Brown. ([\#10077](https://github.com/matrix-org/synapse/issues/10077))
- Add support for routing more requests to workers. ([\#10084](https://github.com/matrix-org/synapse/issues/10084))
- Report OpenTracing spans for database activity. ([\#10113](https://github.com/matrix-org/synapse/issues/10113), [\#10136](https://github.com/matrix-org/synapse/issues/10136), [\#10141](https://github.com/matrix-org/synapse/issues/10141))
- Significantly reduce memory usage of joining large remote rooms. ([\#10117](https://github.com/matrix-org/synapse/issues/10117))


Bugfixes
--------

- Fixed a bug causing replication requests to fail when receiving a lot of events via federation. ([\#10082](https://github.com/matrix-org/synapse/issues/10082))
- Fix a bug in the `force_tracing_for_users` option introduced in Synapse v1.35 which meant that the OpenTracing spans produced were missing most tags. ([\#10092](https://github.com/matrix-org/synapse/issues/10092))
- Fixed a bug that could cause Synapse to stop notifying application services. Contributed by Willem Mulder. ([\#10107](https://github.com/matrix-org/synapse/issues/10107))
- Fix bug where the server would attempt to fetch the same history in the room from a remote server multiple times in parallel. ([\#10116](https://github.com/matrix-org/synapse/issues/10116))
- Fix a bug introduced in Synapse 1.33.0 which caused replication requests to fail when receiving a lot of very large events via federation. ([\#10118](https://github.com/matrix-org/synapse/issues/10118))
- Fix bug when using workers where pagination requests failed if a remote server returned zero events from `/backfill`. Introduced in 1.35.0. ([\#10133](https://github.com/matrix-org/synapse/issues/10133))


Improved Documentation
----------------------

- Clarify security note regarding hosting Synapse on the same domain as other web applications. ([\#9221](https://github.com/matrix-org/synapse/issues/9221))
- Update CAPTCHA documentation to mention turning off the verify origin feature. Contributed by @aaronraimist. ([\#10046](https://github.com/matrix-org/synapse/issues/10046))
- Tweak wording of database recommendation in `INSTALL.md`. Contributed by @aaronraimist. ([\#10057](https://github.com/matrix-org/synapse/issues/10057))
- Add initial infrastructure for rendering Synapse documentation with mdbook. ([\#10086](https://github.com/matrix-org/synapse/issues/10086))
- Convert the remaining Admin API documentation files to markdown. ([\#10089](https://github.com/matrix-org/synapse/issues/10089))
- Make a link in docs use HTTPS. Contributed by @RhnSharma. ([\#10130](https://github.com/matrix-org/synapse/issues/10130))
- Fix broken link in Docker docs. ([\#10132](https://github.com/matrix-org/synapse/issues/10132))


Deprecations and Removals
-------------------------

- Remove the experimental `spaces_enabled` flag. The spaces features are always available now. ([\#10063](https://github.com/matrix-org/synapse/issues/10063))


Internal Changes
----------------

- Tell CircleCI to build Docker images from `main` branch. ([\#9906](https://github.com/matrix-org/synapse/issues/9906))
- Simplify naming convention for release branches to only include the major and minor version numbers. ([\#10013](https://github.com/matrix-org/synapse/issues/10013))
- Add `parse_strings_from_args` for parsing an array from query parameters. ([\#10048](https://github.com/matrix-org/synapse/issues/10048), [\#10137](https://github.com/matrix-org/synapse/issues/10137))
- Remove some dead code regarding TLS certificate handling. ([\#10054](https://github.com/matrix-org/synapse/issues/10054))
- Remove redundant, unmaintained `convert_server_keys` script. ([\#10055](https://github.com/matrix-org/synapse/issues/10055))
- Improve the error message printed by synctl when synapse fails to start. ([\#10059](https://github.com/matrix-org/synapse/issues/10059))
- Fix GitHub Actions lint for newsfragments. ([\#10069](https://github.com/matrix-org/synapse/issues/10069))
- Update opentracing to inject the right context into the carrier. ([\#10074](https://github.com/matrix-org/synapse/issues/10074))
- Fix up `BatchingQueue` implementation. ([\#10078](https://github.com/matrix-org/synapse/issues/10078))
- Log method and path when dropping request due to size limit. ([\#10091](https://github.com/matrix-org/synapse/issues/10091))
- In Github Actions workflows, summarize the Sytest results in an easy-to-read format. ([\#10094](https://github.com/matrix-org/synapse/issues/10094))
- Make `/sync` do fewer state resolutions. ([\#10102](https://github.com/matrix-org/synapse/issues/10102))
- Add missing type hints to the admin API servlets. ([\#10105](https://github.com/matrix-org/synapse/issues/10105))
- Improve opentracing annotations for `Notifier`. ([\#10111](https://github.com/matrix-org/synapse/issues/10111))
- Enable Prometheus metrics for the jaeger client library. ([\#10112](https://github.com/matrix-org/synapse/issues/10112))
- Work to improve the responsiveness of `/sync` requests. ([\#10124](https://github.com/matrix-org/synapse/issues/10124))
- OpenTracing: use a consistent name for background processes. ([\#10135](https://github.com/matrix-org/synapse/issues/10135))


Synapse 1.35.1 (2021-06-03)
===========================

Expand Down
12 changes: 7 additions & 5 deletions INSTALL.md
Original file line number Diff line number Diff line change
Expand Up @@ -399,11 +399,9 @@ Once you have installed synapse as above, you will need to configure it.

### Using PostgreSQL

By default Synapse uses [SQLite](https://sqlite.org/) and in doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
very light workloads.

Almost all installations should opt to use [PostgreSQL](https://www.postgresql.org). Advantages include:
By default Synapse uses an [SQLite](https://sqlite.org/) database and in doing so trades
performance for convenience. Almost all installations should opt to use [PostgreSQL](https://www.postgresql.org)
instead. Advantages include:

- significant performance improvements due to the superior threading and
caching model, smarter query optimiser
Expand All @@ -412,6 +410,10 @@ Almost all installations should opt to use [PostgreSQL](https://www.postgresql.o
For information on how to install and use PostgreSQL in Synapse, please see
[docs/postgres.md](docs/postgres.md)

SQLite is only acceptable for testing purposes. SQLite should not be used in
a production server. Synapse will perform poorly when using
SQLite, especially when participating in large rooms.

### TLS certificates

The default configuration exposes a single HTTP port on the local
Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ exclude mypy.ini
exclude sytest-blacklist
exclude test_postgresql.sh

include book.toml
include pyproject.toml
recursive-include changelog.d *

Expand Down
46 changes: 35 additions & 11 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -149,21 +149,45 @@ For details on having Synapse manage your federation TLS certificates
automatically, please see `<docs/ACME.md>`_.


Security Note
Security note
=============

Matrix serves raw user generated data in some APIs - specifically the `content
repository endpoints <https://matrix.org/docs/spec/client_server/latest.html#get-matrix-media-r0-download-servername-mediaid>`_.
Matrix serves raw, user-supplied data in some APIs -- specifically the `content
repository endpoints`_.

Whilst we have tried to mitigate against possible XSS attacks (e.g.
https://github.com/matrix-org/synapse/pull/1021) we recommend running
matrix homeservers on a dedicated domain name, to limit any malicious user generated
content served to web browsers a matrix API from being able to attack webapps hosted
on the same domain. This is particularly true of sharing a matrix webclient and
server on the same domain.
.. _content repository endpoints: https://matrix.org/docs/spec/client_server/latest.html#get-matrix-media-r0-download-servername-mediaid

See https://github.com/vector-im/riot-web/issues/1977 and
https://developer.github.com/changes/2014-04-25-user-content-security for more details.
Whilst we make a reasonable effort to mitigate against XSS attacks (for
instance, by using `CSP`_), a Matrix homeserver should not be hosted on a
domain hosting other web applications. This especially applies to sharing
the domain with Matrix web clients and other sensitive applications like
webmail. See
https://developer.github.com/changes/2014-04-25-user-content-security for more
information.

.. _CSP: https://github.com/matrix-org/synapse/pull/1021

Ideally, the homeserver should not simply be on a different subdomain, but on
a completely different `registered domain`_ (also known as top-level site or
eTLD+1). This is because `some attacks`_ are still possible as long as the two
applications share the same registered domain.

.. _registered domain: https://tools.ietf.org/html/draft-ietf-httpbis-rfc6265bis-03#section-2.3

.. _some attacks: https://en.wikipedia.org/wiki/Session_fixation#Attacks_using_cross-subdomain_cookie

To illustrate this with an example, if your Element Web or other sensitive web
application is hosted on ``A.example1.com``, you should ideally host Synapse on
``example2.com``. Some amount of protection is offered by hosting on
``B.example1.com`` instead, so this is also acceptable in some scenarios.
However, you should *not* host your Synapse on ``A.example1.com``.

Note that all of the above refers exclusively to the domain used in Synapse's
``public_baseurl`` setting. In particular, it has no bearing on the domain
mentioned in MXIDs hosted on that server.

Following this advice ensures that even if an XSS is found in Synapse, the
impact to other applications will be minimal.


Upgrading an existing Synapse
Expand Down
39 changes: 39 additions & 0 deletions book.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Documentation for possible options in this file is at
# https://rust-lang.github.io/mdBook/format/config.html
[book]
title = "Synapse"
authors = ["The Matrix.org Foundation C.I.C."]
language = "en"
multilingual = false

# The directory that documentation files are stored in
src = "docs"

[build]
# Prevent markdown pages from being automatically generated when they're
# linked to in SUMMARY.md
create-missing = false

[output.html]
# The URL visitors will be directed to when they try to edit a page
edit-url-template = "https://github.com/matrix-org/synapse/edit/develop/{path}"

# Remove the numbers that appear before each item in the sidebar, as they can
# get quite messy as we nest deeper
no-section-label = true

# The source code URL of the repository
git-repository-url = "https://github.com/matrix-org/synapse"

# The path that the docs are hosted on
site-url = "/synapse/"

# Additional HTML, JS, CSS that's injected into each page of the book.
# More information available in docs/website_files/README.md
additional-css = [
"docs/website_files/table-of-contents.css",
"docs/website_files/remove-nav-buttons.css",
"docs/website_files/indent-section-headers.css",
]
additional-js = ["docs/website_files/table-of-contents.js"]
theme = "docs/website_files/theme"
6 changes: 6 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
matrix-synapse-py3 (1.36.0) stable; urgency=medium

* New synapse release 1.36.0.

-- Synapse Packaging team <packages@matrix.org> Tue, 15 Jun 2021 15:41:53 +0100

matrix-synapse-py3 (1.35.1) stable; urgency=medium

* New synapse release 1.35.1.
Expand Down
2 changes: 1 addition & 1 deletion docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,4 +226,4 @@ healthcheck:
## Using jemalloc

Jemalloc is embedded in the image and will be used instead of the default allocator.
You can read about jemalloc by reading the Synapse [README](../README.md).
You can read about jemalloc by reading the Synapse [README](../README.rst).
50 changes: 28 additions & 22 deletions docs/CAPTCHA_SETUP.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,37 @@
# Overview
Captcha can be enabled for this home server. This file explains how to do that.
The captcha mechanism used is Google's ReCaptcha. This requires API keys from Google.

## Getting keys

Requires a site/secret key pair from:

<https://developers.google.com/recaptcha/>

Must be a reCAPTCHA v2 key using the "I'm not a robot" Checkbox option

## Setting ReCaptcha Keys

The keys are a config option on the home server config. If they are not
visible, you can generate them via `--generate-config`. Set the following value:

A captcha can be enabled on your homeserver to help prevent bots from registering
accounts. Synapse currently uses Google's reCAPTCHA service which requires API keys
from Google.

## Getting API keys

1. Create a new site at <https://www.google.com/recaptcha/admin/create>
1. Set the label to anything you want
1. Set the type to reCAPTCHA v2 using the "I'm not a robot" Checkbox option.
This is the only type of captcha that works with Synapse.
1. Add the public hostname for your server, as set in `public_baseurl`
in `homeserver.yaml`, to the list of authorized domains. If you have not set
`public_baseurl`, use `server_name`.
1. Agree to the terms of service and submit.
1. Copy your site key and secret key and add them to your `homeserver.yaml`
configuration file
```
recaptcha_public_key: YOUR_SITE_KEY
recaptcha_private_key: YOUR_SECRET_KEY

In addition, you MUST enable captchas via:

```
1. Enable the CAPTCHA for new registrations
```
enable_registration_captcha: true
```
1. Go to the settings page for the CAPTCHA you just created
1. Uncheck the "Verify the origin of reCAPTCHA solutions" checkbox so that the
captcha can be displayed in any client. If you do not disable this option then you
must specify the domains of every client that is allowed to display the CAPTCHA.

## Configuring IP used for auth

The ReCaptcha API requires that the IP address of the user who solved the
captcha is sent. If the client is connecting through a proxy or load balancer,
The reCAPTCHA API requires that the IP address of the user who solved the
CAPTCHA is sent. If the client is connecting through a proxy or load balancer,
it may be required to use the `X-Forwarded-For` (XFF) header instead of the origin
IP address. This can be configured using the `x_forwarded` directive in the
listeners section of the homeserver.yaml configuration file.
listeners section of the `homeserver.yaml` configuration file.
Loading

0 comments on commit 71326c4

Please sign in to comment.