Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC2716: Incrementally importing history into existing rooms #2716

Open
wants to merge 69 commits into
base: old_master
Choose a base branch
from

Conversation

ara4n
Copy link
Member

@ara4n ara4n commented Aug 4, 2020

A proposal for letting ASes specify event parents and timestamps when submitting events, letting them much more effectively incrementally insert past conversation history. This is getting increasingly topical given the need to bridge existing conversation archives from existing chat systems into Matrix. Fixes most of https://github.com/matrix-org/matrix-doc/issues/698 hopefully.

Rendered

Homeserver implementations:

Client implementations:


Old proposal rendered

cc @tulir for feedback, as the main consumer of the ?ts= API today...

A proposal for letting ASes specify event parents and timestamps when
submitting events, letting them much more effectively insert past
conversation history.

cc @tulir for feedback, as the main consumer of the ?ts= API today...
@ara4n ara4n added the proposal A matrix spec change proposal label Aug 4, 2020
@turt2live turt2live added kind:feature MSC for not-core and not-maintenance stuff proposal-in-review labels Aug 4, 2020
@Half-Shot Half-Shot self-requested a review August 5, 2020 00:00
@turt2live turt2live self-requested a review August 5, 2020 06:02
@lieuwex
Copy link

lieuwex commented Aug 29, 2020

If I understand this correctly, this requires the application service to insert all the historical data before the user requests it.

Isn't it an idea to create a new querying API and request backlog from the AS, like homeservers currently do when they ask other federated servers for historical events?
This would block the homeserver while the AS prepares the events, provides them to the homeserver using the APIs outlined in this MSC, and then the AS could return something like an array containing all the event IDs that were created during the request.

This would lead to an even cleaner SS-like integration with ASes, without creating a buttload of work for every AS. How much would they need to import, for example. Especially if the room has events going back ten years or something like that.

@ara4n
Copy link
Member Author

ara4n commented Nov 10, 2020

If I understand this correctly, this requires the application service to insert all the historical data before the user requests it.

yup, as per the Potential Issues section:

This doesn't provide a way for a HS to tell an AS that a client has tried to call /messages beyond the beginning of a room, and that the AS should try to lazy-insert some more messages (as per https://github.com/matrix-org/matrix-doc/issues/698). For this MSC to be properly useful, we might want to flesh that out.

Copy link
Member

@tulir tulir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had commented on this in HQ earlier, but forgot to post on github.

Overall it looks good. Inserting state could get complicated, but that's just an initial feeling and I'm not sure if it's actually true. A server implementation prototype would be nice so I could just try implementing it in one of my bridges.

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved

## Unstable prefix

Feels unnecessary.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be useful to have an unstable feature flag to check if the homeserver supports this

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved
| \___________________________________
| \ \
| \ \
live timeline previous 1000 messages another block of ancient history
Copy link
Member

@kegsay kegsay Dec 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backfilling via /messages works by walking back up prev_events. If the DAG looks like this, we'll never hit different eras so /messages will return 0 events.

EDIT: Actually it uses depth which will interleave instead. /get_missing_events will however walk up prev events, so all these lovely eras will never make it to other federated servers.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this works because the /messages endpoint has no idea when to jump to a different era. That endpoint topologically walks the DAG (in Dendrite it does this by depth), meaning if you actually did this you would get interleaved events as each era's events start producing the same depth values. This at least returns the events in the forks, but not where you want them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, doing this would produce forwards extremities at the end of each era, which servers will attempt to merge.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so my expectation is that a homeserver should calculate the an appropriate depth when importing history like this, probably by tiebreaking based on origin_server_ts. Where does Dendrite get its depth param from? As it certainly shouldn't be trusting the one it receives over federation, because of https://github.com/matrix-org/matrix-doc/issues/1229.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, the fact that we create forward extremities at the end of each era which then get merged by the next message sent in the room was intended to be a feature, not a bug.

Copy link
Contributor

@MadLittleMods MadLittleMods Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From our meeting, the depth is assumed from the stream ID and can be spoofed. I may not have the details correct but we did discuss fudging it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So having rediscussed this IRL: Dendrite (and Synapse) currently get their depth parameters used for ordering from the wire. Ideally, we'd calculate the depth parameter instead - which could be easy, if we mandate that blocks of old history are always loaded contiguously in reverse chronological order. As a quick fudge to test the approach however we could set depth=1 for these events, and hopefully the default ordering will be sufficient (we think it is on synapse, but dendrite might need a tweak).

Copy link
Member

@kegsay kegsay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this proposal works at stated on its own without additional modifications for the reasons given.

Brainstorming with @neilalexander and @MadLittleMods it sounds like MSC2836: Threading would help here as you can retrospectively update pointers in the thread DAG, and arguably it separates concerns better (the structural auth DAG vs effectively a presentation DAG). This would however require clients to support threading which seems like a high barrier as it doesn't Just Work.

| \___________________________________
| \ \
| \ \
live timeline previous 1000 messages another block of ancient history
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this works because the /messages endpoint has no idea when to jump to a different era. That endpoint topologically walks the DAG (in Dendrite it does this by depth), meaning if you actually did this you would get interleaved events as each era's events start producing the same depth values. This at least returns the events in the forks, but not where you want them.

| \___________________________________
| \ \
| \ \
live timeline previous 1000 messages another block of ancient history
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition, doing this would produce forwards extremities at the end of each era, which servers will attempt to merge.

proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved
proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved
proposals/2716-importing-history-into-existing-rooms.md Outdated Show resolved Hide resolved
MadLittleMods added a commit to MadLittleMods/tardis that referenced this pull request Feb 2, 2021
Edits to make TARDIS work with Synapse while writing Complement tests for [MSC 2716](matrix-org/matrix-spec-proposals#2716).

 - matrix-org/synapse#9247
 - matrix-org/complement#68
@turt2live
Copy link
Member

@MadLittleMods I believe this MSC has received the sanity review it was after and therefore am removing it from the SCT's backlog board. If this is false, please raise it in the SCT office for reconsideration.

with `?prev_event_id` pointing at that floating state to auth the event and where we
want to insert the event.

Another way of doing this might be to store the different eras of the room as
Copy link

@flokli flokli Nov 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has there been any discussion around being able to import history from the client?

Assume someone was previously active on IRC, and creates a Matrix account, then joins an IRC channel via matrix-appservice-irc.
That person might already have had some room history or DMs (that the AS is unaware of), and it would be nice if there was a way to "import" these old logs from the IRC client into some room state, and then see it consistently in all their Matrix clients.

This would need to be stored somewhere server-side, so it's available across all devices, but it might not necessarily be shared to other room members (even though two parties might decide to migrate their conversation history that has happened on IRC to a Matrix DM room)

Copy link
Contributor

@MadLittleMods MadLittleMods Nov 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flokli The new /batch_send endpoint here is in the client API -> POST /_matrix/client/v1/rooms/<roomID>/batch_send?prev_event_id=<message3-eventID> but is currently restricted to application service tokens currently.

If there are good use cases for it, we could easily remove this restriction to allow any room admin (person with proper power levels) to do it.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I would be very much interested in this too, being able to import old chats to an appservice would be quite relevant.

bradtgmurray added a commit to beeper/synapse-legacy-fork that referenced this pull request Nov 15, 2022
Synapse 1.71.0 (2022-11-08)
===========================

Please note that, as announced in the release notes for Synapse 1.69.0, legacy Prometheus metric names are now disabled by default.
They will be removed altogether in Synapse 1.73.0.
If not already done, server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names.
See the [upgrade notes](https://matrix-org.github.io/synapse/v1.71/upgrade.html#upgrading-to-v1710) for more details.

**Note:** in line with our [deprecation policy](https://matrix-org.github.io/synapse/latest/deprecation_policy.html) for platform dependencies, this will be the last release to support PostgreSQL 10, which reaches upstream end-of-life on November 10th, 2022. Future releases of Synapse will require PostgreSQL 11+.

No significant changes since 1.71.0rc2.

Synapse 1.71.0rc2 (2022-11-04)
==============================

Improved Documentation
----------------------

- Document the changes to monthly active user metrics due to deprecation of legacy Prometheus metric names. ([\matrix-org#14358](matrix-org#14358), [\matrix-org#14360](matrix-org#14360))

Deprecations and Removals
-------------------------

- Disable legacy Prometheus metric names by default. They can still be re-enabled for now, but they will be removed altogether in Synapse 1.73.0. ([\matrix-org#14353](matrix-org#14353))

Internal Changes
----------------

- Run unit tests against Python 3.11. ([\matrix-org#13812](matrix-org#13812))

Synapse 1.71.0rc1 (2022-11-01)
==============================

Features
--------

- Support back-channel logouts from OpenID Connect providers. ([\matrix-org#11414](matrix-org#11414))
- Allow use of Postgres and SQLlite full-text search operators in search queries. ([\matrix-org#11635](matrix-org#11635), [\matrix-org#14310](matrix-org#14310), [\matrix-org#14311](matrix-org#14311))
- Implement [MSC3664](matrix-org/matrix-spec-proposals#3664), Pushrules for relations. Contributed by Nico. ([\matrix-org#11804](matrix-org#11804))
- Improve aesthetics of HTML templates. Note that these changes do not retroactively apply to templates which have been [customised](https://matrix-org.github.io/synapse/latest/templates.html#templates) by server admins. ([\matrix-org#13652](matrix-org#13652))
- Enable write-ahead logging for SQLite installations. Contributed by [@asymmetric](https://github.com/asymmetric). ([\matrix-org#13897](matrix-org#13897))
- Show erasure status when [listing users](https://matrix-org.github.io/synapse/latest/admin_api/user_admin_api.html#query-user-account) in the Admin API. ([\matrix-org#14205](matrix-org#14205))
- Provide a specific error code when a `/sync` request provides a filter which doesn't represent a JSON object. ([\matrix-org#14262](matrix-org#14262))

Bugfixes
--------

- Fix a long-standing bug where the `update_synapse_database` script could not be run with multiple databases. Contributed by @thefinn93 @ Beeper. ([\matrix-org#13422](matrix-org#13422))
- Fix a bug which prevented setting an avatar on homeservers which have an explicit port in their `server_name` and have `max_avatar_size` and/or `allowed_avatar_mimetypes` configuration. Contributed by @ashfame. ([\matrix-org#13927](matrix-org#13927))
- Check appservice user interest against the local users instead of all users in the room to align with [MSC3905](matrix-org/matrix-spec-proposals#3905). ([\matrix-org#13958](matrix-org#13958))
- Fix a long-standing bug where Synapse would accidentally include extra information in the response to [`PUT /_matrix/federation/v2/invite/{roomId}/{eventId}`](https://spec.matrix.org/v1.4/server-server-api/#put_matrixfederationv2inviteroomideventid). ([\matrix-org#14064](matrix-org#14064))
- Fix a bug introduced in Synapse 1.64.0 where presence updates could be missing from `/sync` responses. ([\matrix-org#14243](matrix-org#14243))
- Fix a bug introduced in Synapse 1.60.0 which caused an error to be logged when Synapse received a SIGHUP signal if debug logging was enabled. ([\matrix-org#14258](matrix-org#14258))
- Prevent history insertion ([MSC2716](matrix-org/matrix-spec-proposals#2716)) during an partial join ([MSC3706](matrix-org/matrix-spec-proposals#3706)). ([\matrix-org#14291](matrix-org#14291))
- Fix a bug introduced in Synapse 1.34.0 where device names would be returned via a federation user key query request when `allow_device_name_lookup_over_federation` was set to `false`. ([\matrix-org#14304](matrix-org#14304))
- Fix a bug introduced in Synapse 0.34.0 where logs could include error spam when background processes are measured as taking a negative amount of time. ([\matrix-org#14323](matrix-org#14323))
- Fix a bug introduced in Synapse 1.70.0 where clients were unable to PUT new [dehydrated devices](matrix-org/matrix-spec-proposals#2697). ([\matrix-org#14336](matrix-org#14336))

Improved Documentation
----------------------

- Explain how to disable the use of [`trusted_key_servers`](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#trusted_key_servers). ([\matrix-org#13999](matrix-org#13999))
- Add workers settings to [configuration manual](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#individual-worker-configuration). ([\matrix-org#14086](matrix-org#14086))
- Correct the name of the config option [`encryption_enabled_by_default_for_room_type`](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#encryption_enabled_by_default_for_room_type). ([\matrix-org#14110](matrix-org#14110))
- Update docstrings of `SynapseError` and `FederationError` to bettter describe what they are used for and the effects of using them are. ([\matrix-org#14191](matrix-org#14191))

Internal Changes
----------------

- Remove unused `@lru_cache` decorator. ([\matrix-org#13595](matrix-org#13595))
- Save login tokens in database and prevent login token reuse. ([\matrix-org#13844](matrix-org#13844))
- Refactor OIDC tests to better mimic an actual OIDC provider. ([\matrix-org#13910](matrix-org#13910))
- Fix type annotation causing import time error in the Complement forking launcher. ([\matrix-org#14084](matrix-org#14084))
- Refactor [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event` endpoint to loop over federation destinations with standard pattern and error handling. ([\matrix-org#14096](matrix-org#14096))
- Add initial power level event to batch of bulk persisted events when creating a new room. ([\matrix-org#14228](matrix-org#14228))
- Refactor `/key/` endpoints to use `RestServlet` classes. ([\matrix-org#14229](matrix-org#14229))
- Switch to using the `matrix-org/backend-meta` version of `triage-incoming` for new issues in CI. ([\matrix-org#14230](matrix-org#14230))
- Build wheels on macos 11, not 10.15. ([\matrix-org#14249](matrix-org#14249))
- Add debugging to help diagnose lost device list updates. ([\matrix-org#14268](matrix-org#14268))
- Add Rust cache to CI for `trial` runs. ([\matrix-org#14287](matrix-org#14287))
- Improve type hinting of `RawHeaders`. ([\matrix-org#14303](matrix-org#14303))
- Use Poetry 1.2.0 in the Twisted Trunk CI job. ([\matrix-org#14305](matrix-org#14305))

<details>
<summary>Dependency updates</summary>

Runtime:

- Bump anyhow from 1.0.65 to 1.0.66. ([\matrix-org#14278](matrix-org#14278))
- Bump jinja2 from 3.0.3 to 3.1.2. ([\matrix-org#14271](matrix-org#14271))
- Bump prometheus-client from 0.14.0 to 0.15.0. ([\matrix-org#14274](matrix-org#14274))
- Bump psycopg2 from 2.9.4 to 2.9.5. ([\matrix-org#14331](matrix-org#14331))
- Bump pysaml2 from 7.1.2 to 7.2.1. ([\matrix-org#14270](matrix-org#14270))
- Bump sentry-sdk from 1.5.11 to 1.10.1. ([\matrix-org#14330](matrix-org#14330))
- Bump serde from 1.0.145 to 1.0.147. ([\matrix-org#14277](matrix-org#14277))
- Bump serde_json from 1.0.86 to 1.0.87. ([\matrix-org#14279](matrix-org#14279))

Tooling and CI:

- Bump black from 22.3.0 to 22.10.0. ([\matrix-org#14328](matrix-org#14328))
- Bump flake8-bugbear from 21.3.2 to 22.9.23. ([\matrix-org#14042](matrix-org#14042))
- Bump peaceiris/actions-gh-pages from 3.8.0 to 3.9.0. ([\matrix-org#14276](matrix-org#14276))
- Bump peaceiris/actions-mdbook from 1.1.14 to 1.2.0. ([\matrix-org#14275](matrix-org#14275))
- Bump setuptools-rust from 1.5.1 to 1.5.2. ([\matrix-org#14273](matrix-org#14273))
- Bump twine from 3.8.0 to 4.0.1. ([\matrix-org#14332](matrix-org#14332))
- Bump types-opentracing from 2.4.7 to 2.4.10. ([\matrix-org#14133](matrix-org#14133))
- Bump types-requests from 2.28.11 to 2.28.11.2. ([\matrix-org#14272](matrix-org#14272))
</details>
Comment on lines +362 to +366
Because we also can't use the `historical` power level for controlling who can
send these events in the existing room version, we always persist but instead
only process and give meaning to the `m.room.insertion`, `m.room.batch`, and
`m.room.marker` events when the room `creator` sends them. This caveat/rule only
applies to existing room versions.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this using soft rejection on the server-side to determine valid-ness of events? Currently it reads a bit like an event auth change, which can't happen with respect to existing room versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't reject or soft-fail any events in existing room versions.

If the homeserver sees one of those event types from the room creator, the homeserver will process as necessary to make the imported history display when paginating over that area of the room.

Otherwise, it just appears as any other event in the room, nothing happens, and the imported history isn't visible. If the homeserver ever adds MSC2716 support, that history suddenly is unlocked for that server once they re-process the events in the room (probably as a background job).

Synapse, these are called `outlier`s and won't be visible in the chat history
which also allows us to insert multiple batches without having a bunch of `@mxid
joined the room` noise between each batch. **The state will not be resolved into
the current state of the room.**
Copy link

@twouters twouters Jan 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current behavior of synapse is undesirable because it fails backfilled joins for members who have already joined the room.

Does the spec need to clarify that the client implementation must validate the current state of the room before sending a batch event, or should the spec mandate server implementations to ignore batched member state events that are already fulfilled in the current state of the room?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What fails about the member already being joined to the room? What error are you seeing? I would assume this should work because a m.room.member transition from join -> join is allowed, https://spec.matrix.org/v1.4/client-server-api/#mroommember

For clarity, it would be good to see your exact request that is failing.


Back to your topic of how to resolve things; we would want the conflict resolution to be general and the same for all event types. Doing something different for m.room.member would be a recipe for complexity foot-guns.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In most cases, bridges have to include invite events, and join -> invite is not allowed.

Ignoring such state events feels like it'd be the easiest to implement on the server, but it doesn't necessarily need to be built into state_events_at_start at all, it could be a new members field that the server uses to add invite and join events as necessary, or it could even just detect senders of the events in the batch. From the bridge point of view, all of those options are easy to implement.

Beeper (hungryserv) does the last option: it gets the list of users from the event senders, filters away any users who are already in the room and then creates member events for the missing users. (we also don't have any use cases for state events, so the server forbids using state_events_at_start entirely)

netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jan 18, 2023
chat/matrix-synapse: security update

Revisions pulled up:
- chat/matrix-synapse/Makefile                                  1.34-1.36
- chat/matrix-synapse/PLIST                                     1.18-1.19
- chat/matrix-synapse/distinfo                                  1.24,1.27-1.28

-------------------------------------------------------------------
   Module Name:    pkgsrc
   Committed By:   js
   Date:           Sat Oct  2 12:23:13 UTC 2021

   Modified Files:
           pkgsrc/chat/matrix-synapse: Makefile PLIST distinfo

   Log Message:
   Update chat/matrix-synapse to 1.43.0

   Synapse 1.43.0 (2021-09-21)
   =============
   This release drops support for the deprecated, unstable API for [MSC2858 (Multiple SSO Identity
   Providers)](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), as well as the undocumented `experimental.msc2858_enabled` config
   option. Client authors should update their clients to use the stable API, available since Synapse 1.30.

   The documentation has been updated with configuration for routing `/spaces`, `/hierarchy` and `/summary` to workers. See [the upgrade
   notes](https://github.com/matrix-org/synapse/blob/release-v1.43/docs/upgrade.md#upgrading-to-v1430) for more details.

   No significant changes since 1.43.0rc2.

   Synapse 1.43.0rc2 (2021-09-17)
   ===============

   Bugfixes
   --------

   - Added opentracing logging to help debug [\#9424](https://github.com/matrix-org/synapse/issues/9424). ([\#10828](https://github.com/matrix-org/synapse/issues/10828))

   Synapse 1.43.0rc1 (2021-09-14)
   ===============

   Features
   --------

   - Allow room creators to send historical events specified by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) in existing room versions.
   ([\#10566](https://github.com/matrix-org/synapse/issues/10566))
   - Add config option to use non-default manhole password and keys. ([\#10643](https://github.com/matrix-org/synapse/issues/10643))
   - Skip final GC at shutdown to improve restart performance. ([\#10712](https://github.com/matrix-org/synapse/issues/10712))
   - Allow configuration of the oEmbed URLs used for URL previews. ([\#10714](https://github.com/matrix-org/synapse/issues/10714), [\#10759](https://github.com/matrix-org/synapse/issues/10759))
   - Prefer [room version 9](https://github.com/matrix-org/matrix-doc/pull/3375) for restricted rooms per the [room version capabilities](https://github.com/matrix-org/matrix-doc/pull/3244) API.
   ([\#10772](https://github.com/matrix-org/synapse/issues/10772))

   Bugfixes
   --------

   - Fix a long-standing bug where room avatars were not included in email notifications. ([\#10658](https://github.com/matrix-org/synapse/issues/10658))
   - Fix a bug where the ordering algorithm was skipping the `origin_server_ts` step in the spaces summary resulting in unstable room orderings.
   ([\#10730](https://github.com/matrix-org/synapse/issues/10730))
   - Fix edge case when persisting events into a room where there are multiple events we previously hadn't calculated auth chains for (and hadn't marked as needing to be calculated).
   ([\#10743](https://github.com/matrix-org/synapse/issues/10743))
   - Fix a bug which prevented calls to `/createRoom` that included the `room_alias_name` parameter from being handled by worker processes. ([\#10757](https://github.com/matrix-org/synapse/issues/10757))
   - Fix a bug which prevented user registration via SSO to require consent tracking for SSO mapping providers that don't prompt for Matrix ID selection. Contributed by @AndrewFerr.
   ([\#10733](https://github.com/matrix-org/synapse/issues/10733))
   - Only return the stripped state events for the `m.space.child` events in a room for the spaces summary from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
   ([\#10760](https://github.com/matrix-org/synapse/issues/10760))
   - Properly handle room upgrades of spaces. ([\#10774](https://github.com/matrix-org/synapse/issues/10774))
   - Fix a bug which generated invalid homeserver config when the `frontend_proxy` worker type was passed to the Synapse Worker-based Complement image.
   ([\#10783](https://github.com/matrix-org/synapse/issues/10783))

   Improved Documentation
   ----------------------

   - Minor fix to the `media_repository` developer documentation. Contributed by @cuttingedge1109. ([\#10556](https://github.com/matrix-org/synapse/issues/10556))
   - Update the documentation to note that the `/spaces` and `/hierarchy` endpoints can be routed to workers. ([\#10648](https://github.com/matrix-org/synapse/issues/10648))
   - Clarify admin API documentation on undoing room deletions. ([\#10735](https://github.com/matrix-org/synapse/issues/10735))
   - Split up the modules documentation and add examples for module developers. ([\#10758](https://github.com/matrix-org/synapse/issues/10758))
   - Correct 2 typographical errors in the [Log Contexts documentation](https://matrix-org.github.io/synapse/latest/log_contexts.html). ([\#10795](https://github.com/matrix-org/synapse/issues/10795))
   - Fix a wording mistake in the sample configuration. Contributed by @bramvdnheuvel:nltrix.net. ([\#10804](https://github.com/matrix-org/synapse/issues/10804))

   Deprecations and Removals
   -------------------------

   - Remove the [unstable MSC2858 API](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented
   `experimental.msc2858_enabled` config option. The unstable API has been deprecated since Synapse 1.35. Client authors should update their clients to use the stable API introduced in Synapse 1.30 if
   they have not already done so. ([\#10693](https://github.com/matrix-org/synapse/issues/10693))

   Internal Changes
   ----------------

   - Add OpenTracing logging to help debug stuck messages (as described by issue [#9424](https://github.com/matrix-org/synapse/issues/9424)).
   ([\#10704](https://github.com/matrix-org/synapse/issues/10704))
   - Add type annotations to the `synapse.util` package. ([\#10601](https://github.com/matrix-org/synapse/issues/10601))
   - Ensure `rooms.creator` field is always populated for easy lookup in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) usage later.
   ([\#10697](https://github.com/matrix-org/synapse/issues/10697))
   - Add missing type hints to REST servlets. ([\#10707](https://github.com/matrix-org/synapse/issues/10707), [\#10728](https://github.com/matrix-org/synapse/issues/10728),
   [\#10736](https://github.com/matrix-org/synapse/issues/10736))
   - Do not include rooms with unknown room versions in the spaces summary results. ([\#10727](https://github.com/matrix-org/synapse/issues/10727))
   - Additional error checking for the `preset` field when creating a room. ([\#10738](https://github.com/matrix-org/synapse/issues/10738))
   - Clean up some of the federation event authentication code for clarity. ([\#10744](https://github.com/matrix-org/synapse/issues/10744), [\#10745](https://github.com/matrix-org/synapse/issues/10745),
   [\#10746](https://github.com/matrix-org/synapse/issues/10746), [\#10771](https://github.com/matrix-org/synapse/issues/10771), [\#10773](https://github.com/matrix-org/synapse/issues/10773),
   [\#10781](https://github.com/matrix-org/synapse/issues/10781))
   - Add an index to `presence_stream` to hopefully speed up startups a little. ([\#10748](https://github.com/matrix-org/synapse/issues/10748))
   - Refactor event size checking code to simplify searching the codebase for the origins of certain error strings that are occasionally emitted.
   ([\#10750](https://github.com/matrix-org/synapse/issues/10750))
   - Move tests relating to rooms having encryption out of the user directory tests. ([\#10752](https://github.com/matrix-org/synapse/issues/10752))
   - Use `attrs` internally for the URL preview code & update documentation. ([\#10753](https://github.com/matrix-org/synapse/issues/10753))
   - Minor speed ups when joining large rooms over federation. ([\#10754](https://github.com/matrix-org/synapse/issues/10754), [\#10755](https://github.com/matrix-org/synapse/issues/10755),
   [\#10756](https://github.com/matrix-org/synapse/issues/10756), [\#10780](https://github.com/matrix-org/synapse/issues/10780), [\#10784](https://github.com/matrix-org/synapse/issues/10784))
   - Add a constant for `m.federate`. ([\#10775](https://github.com/matrix-org/synapse/issues/10775))
   - Add a script to update the Debian changelog in a Docker container for systems that are not Debian-based. ([\#10778](https://github.com/matrix-org/synapse/issues/10778))
   - Change the format of authenticated users in logs when a user is being puppeted by and admin user. ([\#10779](https://github.com/matrix-org/synapse/issues/10779))
   - Remove fixed and flakey tests from the Sytest blacklist. ([\#10788](https://github.com/matrix-org/synapse/issues/10788))
   - Improve internal details of the user directory code. ([\#10789](https://github.com/matrix-org/synapse/issues/10789))
   - Use direct references to config flags. ([\#10798](https://github.com/matrix-org/synapse/issues/10798))
   - Ensure the Rust reporter passes type checking with jaeger-client 4.7's type annotations. ([\#10799](https://github.com/matrix-org/synapse/issues/10799))

   Synapse 1.42.0 (2021-09-07)
   =============
   This version of Synapse removes deprecated room-management admin APIs, removes out-of-date email pushers, and improves error handling for fallback templates for user-interactive authentication. For
   more information on these points, server administrators are encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).

   No significant changes since 1.42.0rc2.

   Synapse 1.42.0rc2 (2021-09-06)
   ===============

   Features
   --------

   - Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747))

   Internal Changes
   ----------------

   - Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))

   Synapse 1.42.0rc1 (2021-09-01)
   ===============

   Features
   --------

   - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate
   themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
   - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
   - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
   - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613),
   [\#10725](https://github.com/matrix-org/synapse/issues/10725))

   Bugfixes
   --------

   - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
   - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
   - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581),
   [\#10734](https://github.com/matrix-org/synapse/issues/10734))
   - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
   - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
   - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
   - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
   - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
   - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating.
   ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
   - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang.
   ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
   - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))

   Improved Documentation
   ----------------------

   - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
   - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
   - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
   - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
   - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))

   Deprecations and Removals
   -------------------------

   - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))

   Internal Changes
   ----------------

   - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
   - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615),
   [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
   - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
   - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
   - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
   - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
   - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
   - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation.
   ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
   - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
   - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
   - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
   - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666),
   [\#10674](https://github.com/matrix-org/synapse/issues/10674))
   - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
   - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
   - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
   - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))


   To generate a diff of this commit:
   cvs rdiff -u -r1.33 -r1.34 pkgsrc/chat/matrix-synapse/Makefile
   cvs rdiff -u -r1.17 -r1.18 pkgsrc/chat/matrix-synapse/PLIST
   cvs rdiff -u -r1.23 -r1.24 pkgsrc/chat/matrix-synapse/distinfo

-------------------------------------------------------------------
   Module Name:    pkgsrc
   Committed By:   js
   Date:           Fri Nov 19 14:06:08 UTC 2021

   Modified Files:
           pkgsrc/chat/matrix-synapse: Makefile PLIST distinfo

   Log Message:
   Update chat/matrix-synapse to 1.47.0

   Synapse 1.47.0 (2021-11-17)
   =============
   No significant changes since 1.47.0rc3.

   Synapse 1.47.0rc3 (2021-11-16)
   ===============

   Bugfixes
   --------

   - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations.
   ([\#11346](https://github.com/matrix-org/synapse/issues/11346))
   - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version.
   ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))

   Synapse 1.47.0rc2 (2021-11-10)
   ===============

   This fixes an issue with publishing the Debian packages for 1.47.0rc1.
   It is otherwise identical to 1.47.0rc1.

   Synapse 1.47.0rc1 (2021-11-09)
   ===============

   Deprecations and Removals
   -------------------------

   - The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more
   information. ([\#11206](https://github.com/matrix-org/synapse/issues/11206))
   - Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). ([\#11213](https://github.com/matrix-org/synapse/issues/11213))

   Features
   --------

   - Advertise support for Client-Server API r0.6.1. ([\#11097](https://github.com/matrix-org/synapse/issues/11097))
   - Add search by room ID and room alias to the List Room admin API. ([\#11099](https://github.com/matrix-org/synapse/issues/11099))
   - Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room. ([\#11126](https://github.com/matrix-org/synapse/issues/11126))
   - Add a module API method to update a user's membership in a room. ([\#11147](https://github.com/matrix-org/synapse/issues/11147))
   - Add metrics for thread pool usage. ([\#11178](https://github.com/matrix-org/synapse/issues/11178))
   - Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288). ([\#11187](https://github.com/matrix-org/synapse/issues/11187))
   - Add a module API method to retrieve the current state of a room. ([\#11204](https://github.com/matrix-org/synapse/issues/11204))
   - Calculate a default value for `public_baseurl` based on `server_name`. ([\#11210](https://github.com/matrix-org/synapse/issues/11210))
   - Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443. ([\#11211](https://github.com/matrix-org/synapse/issues/11211))
   - Add admin APIs to pause, start and check the status of background updates. ([\#11263](https://github.com/matrix-org/synapse/issues/11263))

   Bugfixes
   --------

   - Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat. ([\#10097](https://github.com/matrix-org/synapse/issues/10097))
   - Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
   ([\#10969](https://github.com/matrix-org/synapse/issues/10969), [\#11212](https://github.com/matrix-org/synapse/issues/11212))
   - Do not accept events if a third-party rule `check_event_allowed` callback raises an exception. ([\#11033](https://github.com/matrix-org/synapse/issues/11033))
   - Fix long-standing bug where verification requests could fail in certain cases if a federation whitelist was in place but did not include your own homeserver.
   ([\#11129](https://github.com/matrix-org/synapse/issues/11129))
   - Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical
   messages is already part of the current room state at the given `?prev_event_id`. ([\#11188](https://github.com/matrix-org/synapse/issues/11188))
   - Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.
   ([\#11191](https://github.com/matrix-org/synapse/issues/11191))
   - Delete `to_device` messages for hidden devices that will never be read, reducing database size. ([\#11199](https://github.com/matrix-org/synapse/issues/11199))
   - Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error. ([\#11200](https://github.com/matrix-org/synapse/issues/11200))
   - Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at
   Beeper. ([\#11207](https://github.com/matrix-org/synapse/issues/11207))
   - Fix a bug introduced in Synapse 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats. ([\#11217](https://github.com/matrix-org/synapse/issues/11217))
   - Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried.
   ([\#11234](https://github.com/matrix-org/synapse/issues/11234))
   - Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection.
   ([\#11240](https://github.com/matrix-org/synapse/issues/11240))
   - Fix a bug preventing Synapse from being rolled back to an earlier version when using workers. ([\#11255](https://github.com/matrix-org/synapse/issues/11255),
   [\#11276](https://github.com/matrix-org/synapse/issues/11276))
   - Fix a bug introduced in Synapse 1.37.1 which caused a remote event being processed by a worker to not get processed on restart if the worker was killed.
   ([\#11262](https://github.com/matrix-org/synapse/issues/11262))
   - Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu.
   ([\#11157](https://github.com/matrix-org/synapse/issues/11157))

   Updates to the Docker image
   ---------------------------

   - Avoid changing user ID when started as a non-root user, and no explicit `UID` is set. ([\#11209](https://github.com/matrix-org/synapse/issues/11209))

   Improved Documentation
   ----------------------

   - Improve example HAProxy config in the docs to properly handle HTTP `Host` headers with port information. This is required for federation over port 443 to work correctly.
   ([\#11128](https://github.com/matrix-org/synapse/issues/11128))
   - Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5. ([\#11151](https://github.com/matrix-org/synapse/issues/11151))
   - Clarify lack of support for Windows. ([\#11198](https://github.com/matrix-org/synapse/issues/11198))
   - Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper. ([\#11221](https://github.com/matrix-org/synapse/issues/11221))
   - Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr. ([\#11257](https://github.com/matrix-org/synapse/issues/11257))

   Internal Changes
   ----------------

   - Add type annotations for the `log_function` decorator. ([\#10943](https://github.com/matrix-org/synapse/issues/10943))
   - Add type hints to `synapse.events`. ([\#11098](https://github.com/matrix-org/synapse/issues/11098))
   - Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code. ([\#11137](https://github.com/matrix-org/synapse/issues/11137))
   - Add type hints so that `synapse.http` passes `mypy` checks. ([\#11164](https://github.com/matrix-org/synapse/issues/11164))
   - Update scripts to pass Shellcheck lints. ([\#11166](https://github.com/matrix-org/synapse/issues/11166))
   - Add knock information in admin export. Contributed by Rafael Gonçalves. ([\#11171](https://github.com/matrix-org/synapse/issues/11171))
   - Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.
   ([\#11179](https://github.com/matrix-org/synapse/issues/11179))
   - Refactor `Filter` to check different fields depending on the data type. ([\#11194](https://github.com/matrix-org/synapse/issues/11194))
   - Improve type hints for the relations datastore. ([\#11205](https://github.com/matrix-org/synapse/issues/11205))
   - Replace outdated links in the pull request checklist with links to the rendered documentation. ([\#11225](https://github.com/matrix-org/synapse/issues/11225))
   - Fix a bug in unit test `test_block_room_and_not_purge`. ([\#11226](https://github.com/matrix-org/synapse/issues/11226))
   - In `ObservableDeferred`, run observers in the order they were registered. ([\#11229](https://github.com/matrix-org/synapse/issues/11229))
   - Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`. ([\#11231](https://github.com/matrix-org/synapse/issues/11231))
   - Add `twine` and `towncrier` as dev dependencies, as they're used by the release script. ([\#11233](https://github.com/matrix-org/synapse/issues/11233))
   - Allow `stream_writers.typing` config to be a list of one worker. ([\#11237](https://github.com/matrix-org/synapse/issues/11237))
   - Remove debugging statement in tests. ([\#11239](https://github.com/matrix-org/synapse/issues/11239))
   - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers.
   ([\#11244](https://github.com/matrix-org/synapse/issues/11244))
   - Add an additional test for the `cachedList` method decorator. ([\#11246](https://github.com/matrix-org/synapse/issues/11246))
   - Make minor correction to the type of `auth_checkers` callbacks. ([\#11253](https://github.com/matrix-org/synapse/issues/11253))
   - Clean up trivial aspects of the Debian package build tooling. ([\#11269](https://github.com/matrix-org/synapse/issues/11269), [\#11273](https://github.com/matrix-org/synapse/issues/11273))
   - Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse. ([\#11270](https://github.com/matrix-org/synapse/issues/11270))

   Synapse 1.46.0 (2021-11-02)
   =============
   The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed.
   ([\#11177](https://github.com/matrix-org/synapse/issues/11177))

   Bugfixes
   --------

   - Fix a bug introduced in v1.46.0rc1 where URL previews of some XML documents would fail. ([\#11196](https://github.com/matrix-org/synapse/issues/11196))

   Synapse 1.46.0rc1 (2021-10-27)
   ===============

   Features
   --------

   - Add support for Ubuntu 21.10 "Impish Indri". ([\#11024](https://github.com/matrix-org/synapse/issues/11024))
   - Port the Password Auth Providers module interface to the new generic interface. ([\#10548](https://github.com/matrix-org/synapse/issues/10548),
   [\#11180](https://github.com/matrix-org/synapse/issues/11180))
   - Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11088](https://github.com/matrix-org/synapse/issues/11088),
   [\#11181](https://github.com/matrix-org/synapse/issues/11181), [\#11192](https://github.com/matrix-org/synapse/issues/11192))
   - Users admin API can now also modify user type in addition to allowing it to be set on user creation. ([\#11174](https://github.com/matrix-org/synapse/issues/11174))

   Bugfixes
   --------

   - Newly-created public rooms are now only assigned an alias if the room's creation has not been blocked by permission settings. Contributed by @AndrewFerr.
   ([\#10930](https://github.com/matrix-org/synapse/issues/10930))
   - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#11001](https://github.com/matrix-org/synapse/issues/11001),
   [\#11009](https://github.com/matrix-org/synapse/issues/11009), [\#11012](https://github.com/matrix-org/synapse/issues/11012))
   - Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room. ([\#11027](https://github.com/matrix-org/synapse/issues/11027))
   - Fix a bug where setting a user's `external_id` via the admin API returns 500 and deletes user's existing external mappings if that external ID is already mapped.
   ([\#11051](https://github.com/matrix-org/synapse/issues/11051))
   - Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private.
   ([\#11075](https://github.com/matrix-org/synapse/issues/11075))
   - Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding. ([\#11077](https://github.com/matrix-org/synapse/issues/11077),
   [\#11089](https://github.com/matrix-org/synapse/issues/11089))
   - Fix broken export-data admin command and add test script checking the command to CI. ([\#11078](https://github.com/matrix-org/synapse/issues/11078))
   - Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API. ([\#11101](https://github.com/matrix-org/synapse/issues/11101))
   - Fix local users who left all their rooms being removed from the user directory, even if the `search_all_users` config option was enabled.
   ([\#11103](https://github.com/matrix-org/synapse/issues/11103))
   - Fix a bug which caused the module API's `get_user_ip_and_agents` function to always fail on workers. `get_user_ip_and_agents` was introduced in 1.44.0 and did not function correctly on worker
   processes at the time. ([\#11112](https://github.com/matrix-org/synapse/issues/11112))
   - Identity server connection is no longer ignoring `ip_range_whitelist`. ([\#11120](https://github.com/matrix-org/synapse/issues/11120))
   - Fix a bug introduced in Synapse 1.45.0 breaking the configuration file parsing script. ([\#11145](https://github.com/matrix-org/synapse/issues/11145))
   - Fix a performance regression introduced in 1.44.0 which could cause client requests to time out when making large numbers of outbound requests.
   ([\#11177](https://github.com/matrix-org/synapse/issues/11177), [\#11190](https://github.com/matrix-org/synapse/issues/11190))
   - Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch. ([\#10975](https://github.com/matrix-org/synapse/issues/10975))

   Improved Documentation
   ----------------------

   - Fix broken links relating to module API deprecation in the upgrade notes. ([\#11069](https://github.com/matrix-org/synapse/issues/11069))
   - Add more information about what happens when a user is deactivated. ([\#11083](https://github.com/matrix-org/synapse/issues/11083))
   - Clarify the the sample log config can be copied from the documentation without issue. ([\#11092](https://github.com/matrix-org/synapse/issues/11092))
   - Update the admin API documentation with an updated list of the characters allowed in registration tokens. ([\#11093](https://github.com/matrix-org/synapse/issues/11093))
   - Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints.
   ([\#11096](https://github.com/matrix-org/synapse/issues/11096))
   - Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion. ([\#11118](https://github.com/matrix-org/synapse/issues/11118))
   - Document the version of Synapse each module callback was introduced in. ([\#11132](https://github.com/matrix-org/synapse/issues/11132))
   - Document the version of Synapse that introduced each module API method. ([\#11183](https://github.com/matrix-org/synapse/issues/11183))

   Internal Changes
   ----------------
   - Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection. ([\#10984](https://github.com/matrix-org/synapse/issues/10984))
   - Include rejected status when we log events. ([\#11008](https://github.com/matrix-org/synapse/issues/11008))
   - Add some extra logging to the event persistence code. ([\#11014](https://github.com/matrix-org/synapse/issues/11014))
   - Rearrange the internal workings of the incremental user directory updates. ([\#11035](https://github.com/matrix-org/synapse/issues/11035))
   - Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room.
   ([\#11143](https://github.com/matrix-org/synapse/issues/11143))
   - Add and improve type hints. ([\#10972](https://github.com/matrix-org/synapse/issues/10972), [\#11055](https://github.com/matrix-org/synapse/issues/11055),
   [\#11066](https://github.com/matrix-org/synapse/issues/11066), [\#11076](https://github.com/matrix-org/synapse/issues/11076), [\#11095](https://github.com/matrix-org/synapse/issues/11095),
   [\#11109](https://github.com/matrix-org/synapse/issues/11109), [\#11121](https://github.com/matrix-org/synapse/issues/11121), [\#11146](https://github.com/matrix-org/synapse/issues/11146))
   - Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse.
   ([\#11054](https://github.com/matrix-org/synapse/issues/11054))
   - Remove dead code from `MediaFilePaths`. ([\#11056](https://github.com/matrix-org/synapse/issues/11056))
   - Be more lenient when parsing oEmbed response versions. ([\#11065](https://github.com/matrix-org/synapse/issues/11065))
   - Create a separate module for the retention configuration. ([\#11070](https://github.com/matrix-org/synapse/issues/11070))
   - Clean up some of the federation event authentication code for clarity. ([\#11115](https://github.com/matrix-org/synapse/issues/11115), [\#11116](https://github.com/matrix-org/synapse/issues/11116),
   [\#11122](https://github.com/matrix-org/synapse/issues/11122))
   - Add docstrings and comments to the application service ephemeral event sending code. ([\#11138](https://github.com/matrix-org/synapse/issues/11138))
   - Update the `sign_json` script to support inline configuration of the signing key. ([\#11139](https://github.com/matrix-org/synapse/issues/11139))
   - Fix broken link in the docker image README. ([\#11144](https://github.com/matrix-org/synapse/issues/11144))
   - Always dump logs from unit tests during CI runs. ([\#11068](https://github.com/matrix-org/synapse/issues/11068))
   - Add tests for `MediaFilePaths` class. ([\#11057](https://github.com/matrix-org/synapse/issues/11057))
   - Simplify the user admin API tests. ([\#11048](https://github.com/matrix-org/synapse/issues/11048))
   - Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s.
   ([\#11071](https://github.com/matrix-org/synapse/issues/11071))

   Synapse 1.45.1 (2021-10-20)
   =============
   Bugfixes
   --------

   - Revert change to counting of deactivated users towards the monthly active users limit, introduced in 1.45.0rc1. ([\#11127](https://github.com/matrix-org/synapse/issues/11127))

   Synapse 1.45.0 (2021-10-19)
   =============
   No functional changes since Synapse 1.45.0rc2.

   Known Issues
   ------------

   - A suspected [performance regression](https://github.com/matrix-org/synapse/issues/11049) which was first reported after the release of 1.44.0 remains unresolved.

     We have not been able to identify a probable cause. Affected users report that setting up a federation sender worker appears to alleviate symptoms of the regression.

   Improved Documentation
   ----------------------

   - Reword changelog to clarify concerns about a suspected performance regression in 1.44.0. ([\#11117](https://github.com/matrix-org/synapse/issues/11117))

   Synapse 1.45.0rc2 (2021-10-14)
   ===============

   This release candidate [fixes](https://github.com/matrix-org/synapse/issues/11053) a user directory [bug](https://github.com/matrix-org/synapse/issues/11025) present in 1.45.0rc1.

   Known Issues
   ------------

   - A suspected [performance regression](https://github.com/matrix-org/synapse/issues/11049) which was first reported after the release of 1.44.0 remains unresolved.

     We have not been able to identify a probable cause. Affected users report that setting up a federation sender worker appears to alleviate symptoms of the regression.

   Bugfixes
   --------

   - Fix a long-standing bug when using multiple event persister workers where events were not correctly sent down `/sync` due to a race. ([\#11045](https://github.com/matrix-org/synapse/issues/11045))
   - Fix a bug introduced in Synapse 1.45.0rc1 where the user directory would stop updating if it processed an event from a
     user not in the `users` table. ([\#11053](https://github.com/matrix-org/synapse/issues/11053))
   - Fix a bug introduced in Synapse 1.44.0 when logging errors during oEmbed processing. ([\#11061](https://github.com/matrix-org/synapse/issues/11061))

   Internal Changes
   ----------------

   - Add an 'approximate difference' method to `StateFilter`. ([\#10825](https://github.com/matrix-org/synapse/issues/10825))
   - Fix inconsistent behavior of `get_last_client_by_ip` when reporting data that has not been stored in the database yet. ([\#10970](https://github.com/matrix-org/synapse/issues/10970))
   - Fix a bug introduced in Synapse 1.21.0 that causes opentracing and Prometheus metrics for replication requests to be measured incorrectly.
   ([\#10996](https://github.com/matrix-org/synapse/issues/10996))
   - Ensure that cache config tests do not share state. ([\#11036](https://github.com/matrix-org/synapse/issues/11036))

   Synapse 1.45.0rc1 (2021-10-12)
   ===============

   **Note:** Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade
   notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.

   Known Issues
   ------------

   - We are investigating [a performance issue](https://github.com/matrix-org/synapse/issues/11049) which was reported after the release of 1.44.0.
   - We are aware of [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. A second release candidate is expected which will resolve this.

   Features
   --------

   - Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655))
   - Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822))
   - Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894))
   - Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910))
   - Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954))
   - Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028))

   Bugfixes
   --------

   - Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922))
   - Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924))
   - Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations.
   ([\#10927](https://github.com/matrix-org/synapse/issues/10927))
   - Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947))
   - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956))
   - Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960))
   - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator.
   ([\#10962](https://github.com/matrix-org/synapse/issues/10962))
   - Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981))
   - Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch.
   ([\#10982](https://github.com/matrix-org/synapse/issues/10982))
   - Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it.
   ([\#10995](https://github.com/matrix-org/synapse/issues/10995))
   - Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002))
   - Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public.
   ([\#11003](https://github.com/matrix-org/synapse/issues/11003))
   - Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored.
   ([\#11042](https://github.com/matrix-org/synapse/issues/11042))
   - Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches.
   ([\#10877](https://github.com/matrix-org/synapse/issues/10877))

   Improved Documentation
   ----------------------

   - Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971))
   - Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973))
   - Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990))
   - Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991))

   Internal Changes
   ----------------

   - Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888))
   - Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892))
   - Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895))
   - Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902))
   - Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903))
   - Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915))
   - Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959),
   [\#10985](https://github.com/matrix-org/synapse/issues/10985))
   - Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940),
   [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988),
   [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011))
   - Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934))
   - Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935))
   - Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936))
   - Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939))
   - Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945))
   - Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958))
   - Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961))
   - Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr.
   ([\#10963](https://github.com/matrix-org/synapse/issues/10963))
   - Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966))
   - Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974))
   - Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983))
   - Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992))
   - Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993))
   - Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994))
   - Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004))
   - Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005))
   - Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006))
   - Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017))
   - Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019))
   - Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021))
   - Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023))
   - When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034))
   - Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))

   Synapse 1.44.0 (2021-10-05)
   =============
   No significant changes since 1.44.0rc3.

   Synapse 1.44.0rc3 (2021-10-04)
   ===============

   Bugfixes
   --------

   - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error.
   ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
   - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))

   Synapse 1.44.0rc2 (2021-09-30)
   ===============

   Bugfixes
   --------

   - Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error.
   ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
   - Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))

   Improved Documentation
   ----------------------

   - Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))

   Synapse 1.44.0rc1 (2021-09-29)
   ===============

   Features
   --------

   - Only allow the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send?chunk_id=xxx` endpoint to connect to an already existing insertion event.
   ([\#10776](https://github.com/matrix-org/synapse/issues/10776))
   - Improve oEmbed URL previews by processing the author name, photo, and video information. ([\#10814](https://github.com/matrix-org/synapse/issues/10814),
   [\#10819](https://github.com/matrix-org/synapse/issues/10819))
   - Speed up responding with large JSON objects to requests. ([\#10868](https://github.com/matrix-org/synapse/issues/10868), [\#10905](https://github.com/matrix-org/synapse/issues/10905))
   - Add a `user_may_create_room_with_invites` spam checker callback to allow modules to allow or deny a room creation request based on the invites and/or 3PID invites it includes.
   ([\#10898](https://github.com/matrix-org/synapse/issues/10898))

   Bugfixes
   --------

   - Fix a long-standing bug that caused an `AssertionError` when purging history in certain rooms. Contributed by @Kokokokoka. ([\#10690](https://github.com/matrix-org/synapse/issues/10690))
   - Fix a long-standing bug which caused deactivated users that were later reactivated to be missing from the user directory. ([\#10782](https://github.com/matrix-org/synapse/issues/10782))
   - Fix a long-standing bug that caused unbanning a user by sending a membership event to fail. Contributed by @aaronraimist. ([\#10807](https://github.com/matrix-org/synapse/issues/10807))
   - Fix a long-standing bug where logging contexts would go missing when federation requests time out. ([\#10810](https://github.com/matrix-org/synapse/issues/10810))
   - Fix a long-standing bug causing an error in the deprecated `/initialSync` endpoint when using the undocumented `from` and `to` parameters.
   ([\#10827](https://github.com/matrix-org/synapse/issues/10827))
   - Fix a bug causing the `remove_stale_pushers` background job to repeatedly fail and log errors. This bug affected Synapse servers that had been upgraded from version 1.28 or older and are using
   SQLite. ([\#10843](https://github.com/matrix-org/synapse/issues/10843))
   - Fix a long-standing bug in Unicode support of the room search admin API breaking search for rooms with non-ASCII characters. ([\#10859](https://github.com/matrix-org/synapse/issues/10859))
   - Fix a bug introduced in Synapse 1.37.0 which caused `knock` membership events which we sent to remote servers to be incorrectly stored in the local database.
   ([\#10873](https://github.com/matrix-org/synapse/issues/10873))
   - Fix invalidating one-time key count cache after claiming keys. The bug was introduced in Synapse v1.41.0. Contributed by Tulir at Beeper.
   ([\#10875](https://github.com/matrix-org/synapse/issues/10875))
   - Fix a long-standing bug causing application service users to be subject to MAU blocking if the MAU limit had been reached, even if configured not to be blocked.
   ([\#10881](https://github.com/matrix-org/synapse/issues/10881))
   - Fix a long-standing bug which could cause events pulled over federation to be incorrectly rejected. ([\#10907](https://github.com/matrix-org/synapse/issues/10907))
   - Fix a long-standing bug causing URL cache files to be stored in storage providers. Server admins may safely delete the `url_cache/` and `url_cache_thumbnails/` directories from any configured
   storage providers to reclaim space. ([\#10911](https://github.com/matrix-org/synapse/issues/10911))
   - Fix a long-standing bug leading to race conditions when creating media store and config directories. ([\#10913](https://github.com/matrix-org/synapse/issues/10913))

   Improved Documentation
   ----------------------

   - Fix some crashes in the Module API example code, by adding JSON encoding/decoding. ([\#10845](https://github.com/matrix-org/synapse/issues/10845))
   - Add developer documentation about experimental configuration flags. ([\#10865](https://github.com/matrix-org/synapse/issues/10865))
   - Properly remove deleted files from GitHub pages when generating the documentation. ([\#10869](https://github.com/matrix-org/synapse/issues/10869))

   Internal Changes
   ----------------

   - Fix GitHub Actions config so we can run sytest on synapse from parallel branches. ([\#10659](https://github.com/matrix-org/synapse/issues/10659))
   - Split out [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) meta events to their own fields in the `/batch_send` response. ([\#10777](https://github.com/matrix-org/synapse/issues/10777))
   - Add missing type hints to REST servlets. ([\#10785](https://github.com/matrix-org/synapse/issues/10785), [\#10817](https://github.com/matrix-org/synapse/issues/10817))
   - Simplify the internal logic which maintains the user directory database tables. ([\#10796](https://github.com/matrix-org/synapse/issues/10796))
   - Use direct references to config flags. ([\#10812](https://github.com/matrix-org/synapse/issues/10812), [\#10885](https://github.com/matrix-org/synapse/issues/10885),
   [\#10893](https://github.com/matrix-org/synapse/issues/10893), [\#10897](https://github.com/matrix-org/synapse/issues/10897))
   - Specify the type of token in generic "Invalid token" error messages. ([\#10815](https://github.com/matrix-org/synapse/issues/10815))
   - Make `StateFilter` frozen so it is hashable. ([\#10816](https://github.com/matrix-org/synapse/issues/10816))
   - Fix a long-standing bug where an `m.room.message` event containing a null byte would cause an internal server error. ([\#10820](https://github.com/matrix-org/synapse/issues/10820))
   - Add type hints to the state database. ([\#10823](https://github.com/matrix-org/synapse/issues/10823))
   - Opt out of cache expiry for `get_users_who_share_room_with_user`, to hopefully improve `/sync` performance when you
     haven't synced recently. ([\#10826](https://github.com/matrix-org/synapse/issues/10826))
   - Track cache eviction rates more finely in Prometheus's monitoring. ([\#10829](https://github.com/matrix-org/synapse/issues/10829))
   - Add missing type hints to `synapse.handlers`. ([\#10831](https://github.com/matrix-org/synapse/issues/10831), [\#10856](https://github.com/matrix-org/synapse/issues/10856))
   - Extend the Module API to let plug-ins check whether an ID is local and to access IP + User Agent data. ([\#10833](https://github.com/matrix-org/synapse/issues/10833))
   - Factor out PNG image data to a constant to be used in several tests. ([\#10834](https://github.com/matrix-org/synapse/issues/10834))
   - Add a test to ensure state events sent by modules get persisted correctly. ([\#10835](https://github.com/matrix-org/synapse/issues/10835))
   - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) fields and event types from `chunk` to `batch` to match the `/batch_send` endpoint.
   ([\#10838](https://github.com/matrix-org/synapse/issues/10838))
   - Rename [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` query parameter from `?prev_event` to more obvious usage with `?prev_event_id`.
   ([\#10839](https://github.com/matrix-org/synapse/issues/10839))
   - Add type hints to `synapse.http.site`. ([\#10867](https://github.com/matrix-org/synapse/issues/10867))
   - Include outlier status when we log V2 or V3 events. ([\#10879](https://github.com/matrix-org/synapse/issues/10879))
   - Break down Grafana's cache expiry time series based on reason for eviction, c.f. [\#10829](https://github.com/matrix-org/synapse/issues/10829).
   ([\#10880](https://github.com/matrix-org/synapse/issues/10880))
   - Clean up some of the federation event authentication code for clarity. ([\#10883](https://github.com/matrix-org/synapse/issues/10883), [\#10884](https://github.com/matrix-org/synapse/issues/10884),
   [\#10896](https://github.com/matrix-org/synapse/issues/10896), [\#10901](https://github.com/matrix-org/synapse/issues/10901))
   - Allow the `.` and `~` characters when creating registration tokens as per the change to [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
   ([\#10887](https://github.com/matrix-org/synapse/issues/10887))
   - Clean up some unnecessary parentheses in places around the codebase. ([\#10889](https://github.com/matrix-org/synapse/issues/10889))
   - Improve type hinting in the user directory code. ([\#10891](https://github.com/matrix-org/synapse/issues/10891))
   - Update development testing script `test_postgresql.sh` to use a supported Python version and make re-runs quicker. ([\#10906](https://github.com/matrix-org/synapse/issues/10906))
   - Document and summarize changes in schema version `61` – `64`. ([\#10917](https://github.com/matrix-org/synapse/issues/10917))
   - Update release script to sign the newly created git tags. ([\#10925](https://github.com/matrix-org/synapse/issues/10925))
   - Fix Debian builds due to `dh-virtualenv` no longer being able to build their docs. ([\#10931](https://github.com/matrix-org/synapse/issues/10931))


   To generate a diff of this commit:
   cvs rdiff -u -r1.34 -r1.35 pkgsrc/chat/matrix-synapse/Makefile
   cvs rdiff -u -r1.18 -r1.19 pkgsrc/chat/matrix-synapse/PLIST
   cvs rdiff -u -r1.26 -r1.27 pkgsrc/chat/matrix-synapse/distinfo

-------------------------------------------------------------------
   Module Name:	pkgsrc
   Committed By:	gdt
   Date:		Tue Nov 23 12:47:51 UTC 2021

   Modified Files:
   	pkgsrc/chat/matrix-synapse: Makefile distinfo

   Log Message:
   chat/matrix-synapse: Update to 1.47.1 (security)

   Synapse 1.47.1 (2021-11-23)
   ======This release fixes a security issue in the media store, affecting all prior releases of Synapse. Server administrators are encouraged to update Synapse as soon as possible. We are not aware of these vulnerabilities being exploited in the wild.

   Server administrators who are unable to update Synapse may use the workarounds described in the linked GitHub Security Advisory below.

   Security advisory
   -----------------

   The following issue is fixed in 1.47.1.

   - **[GHSA-3hfw-x7gx-437c](https://github.com/matrix-org/synapse/security/advisories/GHSA-3hfw-x7gx-437c) / [CVE-2021-41281](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-41281): Path traversal when downloading remote media.**

     Synapse instances with the media repository enabled can be tricked into downloading a file from a remote server into an arbitrary directory, potentially outside the media store directory.

     The last two directories and file name of the path are chosen randomly by Synapse and cannot be controlled by an attacker, which limits the impact.

     Homeservers with the media repository disabled are unaffected. Homeservers configured with a federation whitelist are also unaffected.

     Fixed by [91f2bd090](https://github.com/matrix-org/synapse/commit/91f2bd090).


   To generate a diff of this commit:
   cvs rdiff -u -r1.35 -r1.36 pkgsrc/chat/matrix-synapse/Makefile
   cvs rdiff -u -r1.27 -r1.28 pkgsrc/chat/matrix-synapse/distinfo
**Power level:**

- `historical` (does not need prefixing because it's already under an
experimental room version)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This language is derived/inspired by MSC2285

Suggested change
experimental room version)
### While the MSC is unstable
During this period, to detect server support clients should check for the
presence of the `org.matrix.msc2716` flag in `unstable_features` on `/versions`.
Clients are also required to use the unstable prefixes (see [unstable
prefix](#unstable-prefix)) during this time.
### Once the MSC is merged but not in a spec version
Once this MSC is merged, but is not yet part of the spec, clients should rely on
the presence of the `org.matrix.msc2716.stable` flag in `unstable_features` to
determine server support. If the flag is present, clients are required to use
stable prefixes (see [unstable prefix](#unstable-prefix)).
### Once the MSC is in a spec version
Once this MSC becomes a part of a spec version, clients should rely on the
presence of the spec version, that supports the MSC, in `versions` on
`/versions`, to determine support. Servers are encouraged to keep the
`org.matrix.msc2716.stable` flag around for a reasonable amount of time
to help smooth over the transition for clients. "Reasonable" is intentionally
left as an implementation detail, however the MSC process currently recommends
*at most* 2 months from the date of spec release.

@moritzdietz
Copy link

What does this comment mean from linked PR. Could you give more information on what happened?

Abandoning PR as I don't see MSC2716 going further now that Gitter has fully migrated to Matrix

matrix-org/matrix-react-sdk#8354 (comment)

@MadLittleMods
Copy link
Contributor

@moritzdietz Based on your comment in #matrix-spec, I think you have a misunderstanding on how this relates to Gitter.

We were able to import all 141M messages from Gitter to Matrix without MSC2716. We used the single /send endpoint with the timestamp massaging ?ts=xxx query parameter split between a "historical" and live room.

The big drive to put effort into MSC2716 was the Gitter case but we were able to accomplish the Gitter migration without it in the end and there is no reliance on it now. Historical import within the DAG is still a very useful concept to have in Matrix but there are some roadblocks in the MSC before being viable:

  • Event ordering over federation
    • Currently, events in Synapse are sorted by (topological_ordering, stream_ordering) where topological_ordering is just depth and is baked into the event when it goes over federation. This means when we try to import between depth 1 and 2, we can only rely on stream_ordering to sort between 1 and 2. Since stream_ordering is just dependent on when the server receives the event, the historical messages can easily get out of order. (some more info in the MSC)
    • To totally fix this problem, it would require a different graph linearization strategy. Perhaps we would do some online topological ordering (Katriel–Bodlaender algorithm) where depth/topological_ordering is dynamically updated whenever new events are inserted into the DAG. This is something extremely sci-fi and a big task though.
  • Self-referential batches: There are some ideas in this open discussion but none stand out as great to use.

There have been lots of good learnings here but these shortcomings don't instill confidence to keep driving this forward without a underlying reason to do so. Hopefully we can come at this with some fresh ideas to solve these shortcomings when we need this sort of thing again.

Instead of leaving these experimental implementations languish around in Element and Synapse, I aim to remove them. For the Element case, the PR was never merged, so I could easily just close it.

@moritzdietz
Copy link

@MadLittleMods Thank you Eric for clarifying. As you said, I did misunderstood that. I guess the missing link was this bit of information you just shared above which I haven't seen elsewhere.

@Avamander
Copy link

I have to add that a bunch of people would really like this functionality in order to migrate to Matrix from other platforms, without losing years if not decades of message history.

Understandable that it's a difficult thing to implement, but it's be very useful to a lot of users.

@James-Mat
Copy link

It would be really unfortunate to hear that importing history won't be possible in the foreseeable future. This, like Avamander already mentioned, is for me the only thing which keeps me from adapting Matrix fully and convincing others to do so.
Right now it is inevitable to keep every other client with valuable information installed as well if one wants to look up something historical.
Most people I come across don't want to start anew but take all data with them, though this might be not the case in general.

@MadLittleMods I got a bit confused by your last comment - did I understand correctly that, since the event ordering only depends on topological_ordering and stream_ordering, the usage of the ?ts=xxx query parameter was only to achieve correct metadata for the messages, but doesn't influence ordering at all?

As a thought, is topological_ordering unsigned or signed? If it is signed, this might then make the retrospective insertion of non-interweaving history possible, i.e. if a user switches messengers without a period of using both, by simply giving the historical messages negative values to enqueue them before the messages already in the room.
In the case of unsigned, at least an import right at the beginning (before any more recent messages are sent in the room) would be possible without a change of how event ordering is handled, right?

In any case, can you point me to some PRs or discussions on how the current implementation of the event ordering came to be? When I first subscribed to this PR, the description had me thinking that event ordering would simply be done by timestamp, which would seem to solve any problems of inserting historical messages. But I'm sure that there are other issues which were circumvented by using the current implementation, and I'd like to understand this process to not make any pointless suggestions.

@MadLittleMods
Copy link
Contributor

MadLittleMods commented Apr 11, 2023

I got a bit confused by your last comment - did I understand correctly that, since the event ordering only depends on topological_ordering and stream_ordering, the usage of the ?ts=xxx query parameter was only to achieve correct metadata for the messages, but doesn't influence ordering at all?

In the Gitter case, we started with a fresh room for the historical messages and imported one by one so the topological_ordering was correct. We also used /send?ts=xxx to make the timestamps correct. Then connected the historical and "live" room together with a m.room.tombstone and MSC3946 predecessor event. This functionality is completely separate from MSC2716 and works fine today.

As a thought, is topological_ordering unsigned or signed?

See my previous comment: "topological_ordering is just depth and is baked into the event when it goes over federation"

You can see depth as part of the PDU (persistent data unit) in the spec: https://spec.matrix.org/v1.5/rooms/v10/#event-format-1

by simply giving the historical messages negative values to enqueue them before the messages already in the room.

Importing messages at the beginning of a room is only one use case. We also want to be able to import between any two events and even between already imported messages. One example is if you're importing a mail or newsgroup archive and you stumble across a lost mbox years later with a few more messages, you want to fill in that history.

If your use case is just one import blast at the beginning of a room, the way Gitter accomplished this works now and is a lot simpler (do that instead).

In any case, can you point me to some PRs or discussions on how the current implementation of the event ordering came to be? When I first subscribed to this PR, the description had me thinking that event ordering would simply be done by timestamp, which would seem to solve any problems of inserting historical messages. But I'm sure that there are other issues which were circumvented by using the current implementation, and I'd like to understand this process to not make any pointless suggestions.

Matrix is a DAG (direct acyclic graph) of events. depth being baked in is kinda a "get out of jail free" card on how to linearize the DAG.

This design decision is before my time and I don't know of any good references. Maybe someone in #matrix-spec:matrix.org has some context

MadLittleMods added a commit to matrix-org/synapse that referenced this pull request Jun 16, 2023
…story into existing rooms (#15748)

Context for why we're removing the implementation:

 - matrix-org/matrix-spec-proposals#2716 (comment)
 - matrix-org/matrix-spec-proposals#2716 (comment)

Anyone wanting to continue MSC2716, should also address these leftover tasks: #10737

Closes #10737 in the fact that it is not longer necessary to track those things.
MadLittleMods added a commit to matrix-org/complement that referenced this pull request Jun 20, 2023
yingziwu added a commit to yingziwu/synapse that referenced this pull request Aug 21, 2023
Please note that this will be the last release of Synapse that is compatible with
Python 3.7 and earlier.
This is due to Python 3.7 now having reached End of Life; see our [deprecation policy](https://matrix-org.github.io/synapse/v1.87/deprecation_policy.html)
for more details.

- Pin `pydantic` to `^1.7.4` to avoid backwards-incompatible API changes from the 2.0.0 release.
  Resolves matrix-org#15858.
  Contributed by @PaarthShah. ([\matrix-org#15862](matrix-org#15862))

- Split out 2022 changes from the changelog so the rendered version in GitHub doesn't timeout as much. ([\matrix-org#15846](matrix-org#15846))

- Improve `/messages` response time by avoiding backfill when we already have messages to return. ([\matrix-org#15737](matrix-org#15737))
- Add spam checker module API for logins. ([\matrix-org#15838](matrix-org#15838))

- Fix a long-standing bug where media files were served in an unsafe manner. Contributed by @joshqou. ([\matrix-org#15680](matrix-org#15680))
- Avoid invalidating a cache that was just prefilled. ([\matrix-org#15758](matrix-org#15758))
- Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15770](matrix-org#15770))
- Fix joining rooms through aliases where the alias server isn't a real homeserver. Contributed by @tulir @ Beeper. ([\matrix-org#15776](matrix-org#15776))
- Fix a bug in push rules handling leading to an invalid (per spec) `is_user_mention` rule sent to clients. Also fix wrong rule names for `is_user_mention` and `is_room_mention`. ([\matrix-org#15781](matrix-org#15781))
- Fix a bug introduced in 1.57.0 where the wrong table would be locked on updating database rows when using SQLite as the database backend. ([\matrix-org#15788](matrix-org#15788))
- Fix Sytest environmental variable evaluation in CI. ([\matrix-org#15804](matrix-org#15804))
- Fix forgotten rooms missing from initial sync after rejoining them. Contributed by Nico from Famedly. ([\matrix-org#15815](matrix-org#15815))
- Fix sqlite `user_filters` upgrade introduced in v1.86.0. ([\matrix-org#15817](matrix-org#15817))

- Document `looping_call()` functionality that will wait for the given function to finish before scheduling another. ([\matrix-org#15772](matrix-org#15772))
- Fix a typo in the [Admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html). ([\matrix-org#15805](matrix-org#15805))
- Fix typo in MSC number in faster remote room join architecture doc. ([\matrix-org#15812](matrix-org#15812))

- Remove experimental [MSC2716](matrix-org/matrix-spec-proposals#2716) implementation to incrementally import history into existing rooms. ([\matrix-org#15748](matrix-org#15748))

- Replace `EventContext` fields `prev_group` and `delta_ids` with field `state_group_deltas`. ([\matrix-org#15233](matrix-org#15233))
- Regularly try to send transactions to other servers after they failed instead of waiting for a new event to be available before trying. ([\matrix-org#15743](matrix-org#15743))
- Fix requesting multiple keys at once over federation, related to [MSC3983](matrix-org/matrix-spec-proposals#3983). ([\matrix-org#15755](matrix-org#15755))
- Allow for the configuration of max request retries and min/max retry delays in the matrix federation client. ([\matrix-org#15783](matrix-org#15783))
- Switch from `matrix://` to `matrix-federation://` scheme for internal Synapse routing of outbound federation traffic. ([\matrix-org#15806](matrix-org#15806))
- Fix harmless exceptions being printed when running the port DB script. ([\matrix-org#15814](matrix-org#15814))

* Bump attrs from 22.2.0 to 23.1.0. ([\matrix-org#15801](matrix-org#15801))
* Bump cryptography from 40.0.2 to 41.0.1. ([\matrix-org#15800](matrix-org#15800))
* Bump ijson from 3.2.0.post0 to 3.2.1. ([\matrix-org#15802](matrix-org#15802))
* Bump phonenumbers from 8.13.13 to 8.13.14. ([\matrix-org#15798](matrix-org#15798))
* Bump ruff from 0.0.265 to 0.0.272. ([\matrix-org#15799](matrix-org#15799))
* Bump ruff from 0.0.272 to 0.0.275. ([\matrix-org#15833](matrix-org#15833))
* Bump serde_json from 1.0.96 to 1.0.97. ([\matrix-org#15797](matrix-org#15797))
* Bump serde_json from 1.0.97 to 1.0.99. ([\matrix-org#15832](matrix-org#15832))
* Bump towncrier from 22.12.0 to 23.6.0. ([\matrix-org#15831](matrix-org#15831))
* Bump types-opentracing from 2.4.10.4 to 2.4.10.5. ([\matrix-org#15830](matrix-org#15830))
* Bump types-setuptools from 67.8.0.0 to 68.0.0.0. ([\matrix-org#15835](matrix-org#15835))
@anoadragon453 anoadragon453 removed their request for review June 24, 2024 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment