Enable cache time-based expiry by default #11849

H-Shay · 2022-01-27T20:38:52Z

Changes the default config option for time based cache expiry from None to 30m. Fixes #11787, where you will also find more discussion on this change.

anoadragon453

Note that this only enables cache expiry by default for newly generated config files, not existing installations. This is good because:

We don't want to change settings out from under people.
There isn't a way to differentiate between "someone's never set this option" and "someone explicitly disabled this option".

So this looks like the right solution to me.

changelog.d/11849.misc

reivilibre · 2022-01-28T09:41:48Z

Note that this only enables cache expiry by default for newly generated config files, not existing installations. This is good because…

I actually am not so sure (and it's not the impression I got from the meeting yesterday where we mentioned this).

I think we should enable it by default for existing configs because the original point about it being an obscure option still stands and generally homeservers should benefit from this change.

There isn't a way to differentiate between "someone's never set this option" and "someone explicitly disabled this option".

Setting it to null would explicitly disable the option; on the other hand, once we ship a default config file with options that aren't the default in the config loaders, we lose the ability to distinguish '30 minutes, explicit' and 'no preference'.
We also might cause strange behaviour later on if we decide to remove this option and start showing deprecation warnings and errors to operators who never set the option manually anyway.

I find it strange that we would enable an option by default only for 'new' homeservers — I appreciate the caution but it also conflicts with the intuition that a homeserver admin only needs to have the options they changed present in their config file.

By leaving an option unmarked in the config, I would interpret this to mean 'I, the admin, don't care and want the software to do the default reasonable behaviour' — in this case I would say the time-based expiry is a great boon for memory usage and seems like the most reasonable behaviour when the admin hasn't expressed a preference.

There is a case to be made that we should stop documenting the options inside the config file and instead have a nice config manual or something, because the comment will become out of date for old installations. Still, we could announce that in the upgrade notes in case anyone was caught out by the change in default behaviour.

H-Shay · 2022-01-28T17:40:38Z

Fascinating-it never occurred to me that we would change the behavior of existing Synapse homeservers. My understanding was something along the lines of: "we've used this setting and feel confident it's beneficial, we will make it the default going forwards." My personal feeling is that it is a little strange to change the default behavior out from under people rather than having them explicitly op-in to the new default (which I figured we would shout about in the release notes) but that's just a feeling. I am happy to implement whatever the consensus determines is the best approach.
Also plus one for having a config manual vs comments, I will open an issue about that as I think it would be beneficial.

richvdh · 2022-01-31T12:16:36Z

My personal feeling is that it is a little strange to change the default behavior out from under people rather than having them explicitly op-in to the new default (which I figured we would shout about in the release notes) but that's just a feeling.

This is certainly a reasonable argument, but I think the point is that most people don't want to have to think this hard about maintaining their deployments - they want it to "just work" and it's up to us to make it do so as best we can, which includes changing the default behaviour where we have reason to believe that will improve things for the majority of users.

There could be a minority who would prefer to retain the existing behaviour, but they can update their config files if they want to.

It's worth noting that we change Synapse's default behaviour all the time when it comes to bugfixes and the like. You could argue that not having time-based expiry enabled is a bug.

anoadragon453 · 2022-01-31T13:56:57Z

Thanks for the course-correction folks! There may be a small user-base that read the option and left it commented, assuming that would remain disabled. We would then be changing the behaviour for them.

However, shouting about it in the changelog (or perhaps upgrade notes if we deem necessary) should be enough for even this theoretical edge case.

anoadragon453

Going forward on this PR, I think we should:

Change the default (when the option is not specified) of expiry_time to 30m.
Assuming we'd like to continue allowing people to disable cache expiry, modify the sample config to make disabling the option a little simpler. We could either:
- tell people to write expiry_time: null to disable the option, but this isn't really consistent with the rest of the config file.
- introduce a new expire_caches boolean option (defaulting to true), which would be my personal preference.

callahad · 2022-01-31T14:29:03Z

tell people to write expiry_time: null to disable the option, but this isn't really consistent with the rest of the config file.

Would expiry_time: 0 or expiry_time: -1 be acceptable? Those seem to be common enough conventions in other software, but I don't see much precedent within Synapse. Adding a whole 'nother option seems pretty heavyweight, so I'd bias toward overloading the time.

richvdh · 2022-01-31T14:35:55Z

expiry_time is a pretty rubbish name for it, tbh (cf #5584 (comment)). It might be a reasonable time to sort that out.

H-Shay · 2022-01-31T18:06:26Z

Thanks everyone for chiming in, I will make these changes. Also thanks for taking time to explain your reasoning, it's very helpful for me to hear.

H-Shay · 2022-02-03T22:47:26Z

Per Rich's comment I renamed the user-facing flag to something (hopefully) a little more descriptive, but I wasn't sure if I should rename the internal var as well.

anoadragon453

I wasn't sure if I should rename the internal var as well.

Doing so is fine, as long as it doesn't affect anyone relying on Synapse (such as methods exposed by the module API. These variables are private however, and people shouldn't be reaching into and relying on our private variables.

For Synapse devs, if an in-flight PR relied on the old variable name then they'd discover a merge conflict after your PR had merged - alerting them, and allowing them to correct it.

A few things below, but overall this is looking great!

anoadragon453 · 2022-02-04T11:31:49Z

synapse/config/cache.py

-            self.expiry_time_msec: Optional[int] = self.parse_duration(expiry_time)
+        expire_caches = cache_config.get("expire_caches", True)
+        cache_entry_ttl = cache_config.get("cache_entry_ttl")
+


Since we're changing config options, we should include backwards-compatibility support for the previous config option name. Thus if someone set expiry_time previously, that same value should continue to work even after this update. At some point ™️ in the future we can remove support for the old option name.

synapse/config/cache.py

anoadragon453 · 2022-02-04T11:39:48Z

changelog.d/11849.misc

@@ -0,0 +1 @@
+Enable cache time-based expiry by default.


It may be worth mentioning the new configuration options that are available, or otherwise pointing to the relevant section of the sample config for the full details.

This could also be done via our upgrading docs, where we have more room.

I would likely still sneak a keyword or two in there: expiry_time will be superseded by expire_caches and cache_entry_ttl.
Helps perk people's ears up and helps if you ever need to do a Ctrl+F for one of the config option names.

reivilibre

just nits really :)

synapse/config/cache.py

docs/upgrade.md

reivilibre · 2022-02-10T12:23:46Z

changelog.d/11849.misc

@@ -0,0 +1 @@
+Enable cache time-based expiry by default.


I would likely still sneak a keyword or two in there: expiry_time will be superseded by expire_caches and cache_entry_ttl.
Helps perk people's ears up and helps if you ever need to do a Ctrl+F for one of the config option names.

reivilibre

Looks good; thanks!

reivilibre · 2022-02-11T11:55:53Z

synapse/config/cache.py

+        if expiry_time and expire_caches:
+            logger.warning(
+                "You have set two incompatible flags, expiry_time and expire_caches. Please only use the "
+                "expire_caches and cache_entry_ttl flags and delete the expiry_time flag as it is "
+                "deprecated."
+            )
        if expiry_time:
+            logger.warning(
+                "Expiry_time is a deprecated flag, please use the expire_caches and cache_entry_ttl flags "
+                "instead."
+            )


This is very nitty, but I would call these options rather than flags.
At least in my mind, a flag is a boolean.

This is a useful distinction, thank you!

reivilibre · 2022-02-11T11:58:04Z

docs/upgrade.md

 # Upgrading to v1.(next)

-## Time-based cache expiry is now enabled by default
-
-Formerly, entries in the cache were not evicted regardless of whether they were accessed after storing.
-This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed. 
-To change the default behavior, go to the `caches` section of the config and change the `expire_caches` and 
-`cache_entry_ttl` flags as necessary. Please note that these flags replace the `expiry_time` flag in the config.  

 ## Stablisation of MSC3231


I know it's not your mess, but as you're here, please could you clean this up by removing the 'v1.(next)' heading and bringing the announcement into the v1.53 section?
I personally think a 'v1.(next)' section would be easily forgotten as it's not in our release notes. Or at least, I'd forget! :)

H-Shay · 2022-02-11T17:15:12Z

@anoadragon453 any chance I can get you to dismiss your stale review when you get the chance?

anoadragon453 · 2022-02-11T18:04:23Z

@anoadragon453 any chance I can get you to dismiss your stale review when you get the chance?

Of course, and note that it's fine to do so yourself as well if the review is outdated (assuming you have permission?).

old

richvdh

sorry, a few belated notes. Feel free to commit fixes straight to develop.

This seems to add an empty https://github.com/matrix-org/synapse/blob/develop/synapse/config/background_updates.py, which looks wrong. Please could it be removed?

richvdh · 2022-02-14T13:20:43Z

changelog.d/11849.misc

@@ -0,0 +1 @@
+Enable cache time-based expiry by default. The `expiry_time` config flag will be superseded by `expire_caches` and `cache_entry_ttl`.


this should probably be a .feature?

and "will be" should be "has been"?

richvdh · 2022-02-14T13:21:33Z

docs/upgrade.md

+Formerly, entries in the cache were not evicted regardless of whether they were accessed after storing.
+This behavior has now changed. By default entries in the cache are now evicted after 30m of not being accessed. 
+To change the default behavior, go to the `caches` section of the config and change the `expire_caches` and 
+`cache_entry_ttl` flags as necessary. Please note that these flags replace the `expiry_time` flag in the config.  


Might be worth mentioning that expiry_time will still work (for now), but is deprecated?

H-Shay · 2022-02-14T18:23:23Z

Thanks for catching these mistakes-I have made the requested changes.

clokep · 2022-02-14T18:26:48Z

Thanks for catching these mistakes-I have made the requested changes.

For cross-referencing: 9c4563c

Synapse 1.53.0rc1 (2022-02-15) ============================== Features -------- - Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](#11215), [\#11966](#11966)) - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](#11655)) - Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](#11837)) - Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](#11849)) - Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](#11854)) - Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](#11867)) - Allow modules to retrieve the current instance's server name and worker name. ([\#11868](#11868)) - Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](#11892)) - Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](#11933), [\#11989](#11989)) - Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](#11941)) - Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](#11967)) Bugfixes -------- - Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](#11114)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](#11890)) - Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](#11930)) - Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](#11936)) - Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](#11952)) - Require that modules register their callbacks using keyword arguments. ([\#11975](#11975)) - Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](#11988)) Improved Documentation ---------------------- - Fix typo in User Admin API: unpind -> unbind. ([\#11859](#11859)) - Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](#11862)) - Remove outdated MSC1711 FAQ document. ([\#11907](#11907)) - Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](#11946)) - Add information on the Synapse release cycle. ([\#11954](#11954)) - Fix broken link in the README to the admin API for password reset. ([\#11955](#11955)) Deprecations and Removals ------------------------- - Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](#11895)) - Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](#11950)) - No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](#11961)) Internal Changes ---------------- - Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](#11615), [\#11616](#11616)) - Improve performance when fetching bundled aggregations for multiple events. ([\#11660](#11660), [\#11752](#11752)) - Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](#11832)) - Add missing type hints to replication code. ([\#11856](#11856), [\#11938](#11938)) - Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](#11869)) - Improve opentracing for incoming federation requests. ([\#11870](#11870)) - Improve internal docstrings in `synapse.util.caches`. ([\#11876](#11876)) - Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](#11878)) - Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](#11880)) - Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](#11884)) - Disable coverage calculation for olddeps build. ([\#11888](#11888)) - Preparation to support sending device list updates to application services. ([\#11905](#11905)) - Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](#11909)) - Run Complement tests sequentially. ([\#11910](#11910)) - Various refactors to the application service notifier code. ([\#11911](#11911), [\#11912](#11912)) - Tests: replace mocked `Authenticator` with the real thing. ([\#11913](#11913)) - Various refactors to the typing notifications code. ([\#11914](#11914)) - Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](#11927)) - Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](#11939)) - Add missing type hints. ([\#11953](#11953)) - Fix an import cycle in `synapse.event_auth`. ([\#11965](#11965)) - Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](#11969)) - Prepare for rename of default Complement branch. ([\#11971](#11971)) - Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](#11979))

@totallynotvaishnav

Synapse 1.53.0 (2022-02-22) =========================== No significant changes. Synapse 1.53.0rc1 (2022-02-15) ============================== Features -------- - Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](matrix-org/synapse#11215), [\#11966](matrix-org/synapse#11966)) - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](matrix-org/synapse#11655)) - Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](matrix-org/synapse#11837)) - Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](matrix-org/synapse#11849)) - Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](matrix-org/synapse#11854)) - Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](matrix-org/synapse#11867)) - Allow modules to retrieve the current instance's server name and worker name. ([\#11868](matrix-org/synapse#11868)) - Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](matrix-org/synapse#11892)) - Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](matrix-org/synapse#11933), [\#11989](matrix-org/synapse#11989)) - Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](matrix-org/synapse#11941)) - Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](matrix-org/synapse#11967)) Bugfixes -------- - Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](matrix-org/synapse#11114)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](matrix-org/synapse#11890)) - Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](matrix-org/synapse#11930)) - Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](matrix-org/synapse#11936)) - Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](matrix-org/synapse#11952)) - Require that modules register their callbacks using keyword arguments. ([\#11975](matrix-org/synapse#11975)) - Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](matrix-org/synapse#11988)) Improved Documentation ---------------------- - Fix typo in User Admin API: unpind -> unbind. ([\#11859](matrix-org/synapse#11859)) - Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](matrix-org/synapse#11862)) - Remove outdated MSC1711 FAQ document. ([\#11907](matrix-org/synapse#11907)) - Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](matrix-org/synapse#11946)) - Add information on the Synapse release cycle. ([\#11954](matrix-org/synapse#11954)) - Fix broken link in the README to the admin API for password reset. ([\#11955](matrix-org/synapse#11955)) Deprecations and Removals ------------------------- - Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](matrix-org/synapse#11895)) - Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](matrix-org/synapse#11950)) - No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](matrix-org/synapse#11961)) Internal Changes ---------------- - Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](matrix-org/synapse#11615), [\#11616](matrix-org/synapse#11616)) - Improve performance when fetching bundled aggregations for multiple events. ([\#11660](matrix-org/synapse#11660), [\#11752](matrix-org/synapse#11752)) - Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](matrix-org/synapse#11832)) - Add missing type hints to replication code. ([\#11856](matrix-org/synapse#11856), [\#11938](matrix-org/synapse#11938)) - Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](matrix-org/synapse#11869)) - Improve opentracing for incoming federation requests. ([\#11870](matrix-org/synapse#11870)) - Improve internal docstrings in `synapse.util.caches`. ([\#11876](matrix-org/synapse#11876)) - Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](matrix-org/synapse#11878)) - Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](matrix-org/synapse#11880)) - Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](matrix-org/synapse#11884)) - Disable coverage calculation for olddeps build. ([\#11888](matrix-org/synapse#11888)) - Preparation to support sending device list updates to application services. ([\#11905](matrix-org/synapse#11905)) - Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](matrix-org/synapse#11909)) - Run Complement tests sequentially. ([\#11910](matrix-org/synapse#11910)) - Various refactors to the application service notifier code. ([\#11911](matrix-org/synapse#11911), [\#11912](matrix-org/synapse#11912)) - Tests: replace mocked `Authenticator` with the real thing. ([\#11913](matrix-org/synapse#11913)) - Various refactors to the typing notifications code. ([\#11914](matrix-org/synapse#11914)) - Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](matrix-org/synapse#11927)) - Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](matrix-org/synapse#11939)) - Add missing type hints. ([\#11953](matrix-org/synapse#11953)) - Fix an import cycle in `synapse.event_auth`. ([\#11965](matrix-org/synapse#11965)) - Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](matrix-org/synapse#11969)) - Prepare for rename of default Complement branch. ([\#11971](matrix-org/synapse#11971)) - Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](matrix-org/synapse#11979)) Synapse 1.52.0 (2022-02-08) =========================== No significant changes since 1.52.0rc1. Note that [Twisted 22.1.0](https://github.com/twisted/twisted/releases/tag/twisted-22.1.0) has recently been released, which fixes a [security issue](GHSA-92x2-jw7w-xvvx) within the Twisted library. We do not believe Synapse is affected by this vulnerability, though we advise server administrators who installed Synapse via pip to upgrade Twisted with `pip install --upgrade Twisted` as a matter of good practice. The Docker image `matrixdotorg/synapse` and the Debian packages from `packages.matrix.org` are using the updated library. Synapse 1.52.0rc1 (2022-02-01) ============================== Features -------- - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11621](matrix-org/synapse#11621), [\#11788](matrix-org/synapse#11788), [\#11789](matrix-org/synapse#11789)) - Add an admin API to reset connection timeouts for remote server. ([\#11639](matrix-org/synapse#11639)) - Add an admin API to get a list of rooms that federate with a given remote homeserver. ([\#11658](matrix-org/synapse#11658)) - Add a config flag to inhibit `M_USER_IN_USE` during registration. ([\#11743](matrix-org/synapse#11743)) - Add a module callback to set username at registration. ([\#11790](matrix-org/synapse#11790)) - Allow configuring a maximum file size as well as a list of allowed content types for avatars. ([\#11846](matrix-org/synapse#11846)) Bugfixes -------- - Include the bundled aggregations in the `/sync` response, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#11612](matrix-org/synapse#11612)) - Fix a long-standing bug when previewing Reddit URLs which do not contain an image. ([\#11767](matrix-org/synapse#11767)) - Fix a long-standing bug that media streams could cause long-lived connections when generating URL previews. ([\#11784](matrix-org/synapse#11784)) - Include a `prev_content` field in state events sent to Application Services. Contributed by @totallynotvaishnav. ([\#11798](matrix-org/synapse#11798)) - Fix a bug introduced in Synapse 0.33.3 causing requests to sometimes log strings such as `HTTPStatus.OK` instead of integer status codes. ([\#11827](matrix-org/synapse#11827)) Improved Documentation ---------------------- - Update pypi installation docs to indicate that we now support Python 3.10. ([\#11820](matrix-org/synapse#11820)) - Add missing steps to the contribution submission process in the documentation. Contributed by @sequentialread. ([\#11821](matrix-org/synapse#11821)) - Remove not needed old table of contents in documentation. ([\#11860](matrix-org/synapse#11860)) - Consolidate the `access_token` information at the top of each relevant page in the Admin API documentation. ([\#11861](matrix-org/synapse#11861)) Deprecations and Removals ------------------------- - Drop support for Python 3.6, which is EOL. ([\#11683](matrix-org/synapse#11683)) - Remove the `experimental_msc1849_support_enabled` flag as the features are now stable. ([\#11843](matrix-org/synapse#11843)) Internal Changes ---------------- - Preparation for database schema simplifications: add `state_key` and `rejection_reason` columns to `events` table. ([\#11792](matrix-org/synapse#11792)) - Add `FrozenEvent.get_state_key` and use it in a couple of places. ([\#11793](matrix-org/synapse#11793)) - Preparation for database schema simplifications: stop reading from `event_reference_hashes`. ([\#11794](matrix-org/synapse#11794)) - Drop unused table `public_room_list_stream`. ([\#11795](matrix-org/synapse#11795)) - Preparation for reducing Postgres serialization errors: allow setting transaction isolation level. Contributed by Nick @ Beeper. ([\#11799](matrix-org/synapse#11799), [\#11847](matrix-org/synapse#11847)) - Docker: skip the initial amd64-only build and go straight to multiarch. ([\#11810](matrix-org/synapse#11810)) - Run Complement on the Github Actions VM and not inside a Docker container. ([\#11811](matrix-org/synapse#11811)) - Log module names at startup. ([\#11813](matrix-org/synapse#11813)) - Improve type safety of bundled aggregations code. ([\#11815](matrix-org/synapse#11815)) - Correct a type annotation in the event validation logic. ([\#11817](matrix-org/synapse#11817), [\#11830](matrix-org/synapse#11830)) - Minor updates and documentation for database schema delta files. ([\#11823](matrix-org/synapse#11823)) - Workaround a type annotation problem in `prometheus_client` 0.13.0. ([\#11834](matrix-org/synapse#11834)) - Minor performance improvement in room state lookup. ([\#11836](matrix-org/synapse#11836)) - Fix some indentation inconsistencies in the sample config. ([\#11838](matrix-org/synapse#11838)) - Add type hints to `tests/rest/admin`. ([\#11851](matrix-org/synapse#11851))

Synapse 1.53.0 (2022-02-22) =========================== No significant changes. Synapse 1.53.0rc1 (2022-02-15) ============================== Features -------- - Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\matrix-org#11215](matrix-org#11215), [\matrix-org#11966](matrix-org#11966)) - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\matrix-org#11655](matrix-org#11655)) - Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\matrix-org#11837](matrix-org#11837)) - Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\matrix-org#11849](matrix-org#11849)) - Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\matrix-org#11854](matrix-org#11854)) - Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\matrix-org#11867](matrix-org#11867)) - Allow modules to retrieve the current instance's server name and worker name. ([\matrix-org#11868](matrix-org#11868)) - Use a dedicated configurable rate limiter for 3PID invites. ([\matrix-org#11892](matrix-org#11892)) - Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\matrix-org#11933](matrix-org#11933), [\matrix-org#11989](matrix-org#11989)) - Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\matrix-org#11941](matrix-org#11941)) - Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\matrix-org#11967](matrix-org#11967)) Bugfixes -------- - Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\matrix-org#11114](matrix-org#11114)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\matrix-org#11890](matrix-org#11890)) - Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\matrix-org#11930](matrix-org#11930)) - Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\matrix-org#11936](matrix-org#11936)) - Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\matrix-org#11952](matrix-org#11952)) - Require that modules register their callbacks using keyword arguments. ([\matrix-org#11975](matrix-org#11975)) - Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\matrix-org#11988](matrix-org#11988)) Improved Documentation ---------------------- - Fix typo in User Admin API: unpind -> unbind. ([\matrix-org#11859](matrix-org#11859)) - Document images returned by the User List Media Admin API can include those generated by URL previews. ([\matrix-org#11862](matrix-org#11862)) - Remove outdated MSC1711 FAQ document. ([\matrix-org#11907](matrix-org#11907)) - Correct the structured logging configuration example. Contributed by Brad Jones. ([\matrix-org#11946](matrix-org#11946)) - Add information on the Synapse release cycle. ([\matrix-org#11954](matrix-org#11954)) - Fix broken link in the README to the admin API for password reset. ([\matrix-org#11955](matrix-org#11955)) Deprecations and Removals ------------------------- - Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\matrix-org#11895](matrix-org#11895)) - Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\matrix-org#11950](matrix-org#11950)) - No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\matrix-org#11961](matrix-org#11961)) Internal Changes ---------------- - Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\matrix-org#11615](matrix-org#11615), [\matrix-org#11616](matrix-org#11616)) - Improve performance when fetching bundled aggregations for multiple events. ([\matrix-org#11660](matrix-org#11660), [\matrix-org#11752](matrix-org#11752)) - Fix type errors introduced by new annotations in the Prometheus Client library. ([\matrix-org#11832](matrix-org#11832)) - Add missing type hints to replication code. ([\matrix-org#11856](matrix-org#11856), [\matrix-org#11938](matrix-org#11938)) - Ensure that `opentracing` scopes are activated and closed at the right time. ([\matrix-org#11869](matrix-org#11869)) - Improve opentracing for incoming federation requests. ([\matrix-org#11870](matrix-org#11870)) - Improve internal docstrings in `synapse.util.caches`. ([\matrix-org#11876](matrix-org#11876)) - Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\matrix-org#11878](matrix-org#11878)) - Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\matrix-org#11880](matrix-org#11880)) - Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\matrix-org#11884](matrix-org#11884)) - Disable coverage calculation for olddeps build. ([\matrix-org#11888](matrix-org#11888)) - Preparation to support sending device list updates to application services. ([\matrix-org#11905](matrix-org#11905)) - Add a test that checks users receive their own device list updates down `/sync`. ([\matrix-org#11909](matrix-org#11909)) - Run Complement tests sequentially. ([\matrix-org#11910](matrix-org#11910)) - Various refactors to the application service notifier code. ([\matrix-org#11911](matrix-org#11911), [\matrix-org#11912](matrix-org#11912)) - Tests: replace mocked `Authenticator` with the real thing. ([\matrix-org#11913](matrix-org#11913)) - Various refactors to the typing notifications code. ([\matrix-org#11914](matrix-org#11914)) - Use the proper type for the `Content-Length` header in the `UploadResource`. ([\matrix-org#11927](matrix-org#11927)) - Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\matrix-org#11939](matrix-org#11939)) - Add missing type hints. ([\matrix-org#11953](matrix-org#11953)) - Fix an import cycle in `synapse.event_auth`. ([\matrix-org#11965](matrix-org#11965)) - Unpin `frozendict` but exclude the known bad version 2.1.2. ([\matrix-org#11969](matrix-org#11969)) - Prepare for rename of default Complement branch. ([\matrix-org#11971](matrix-org#11971)) - Fetch Synapse's version using a helper from `matrix-common`. ([\matrix-org#11979](matrix-org#11979))

Synapse 1.53.0 (2022-02-22) =========================== No significant changes. Synapse 1.53.0rc1 (2022-02-15) ============================== Features -------- - Add experimental support for sending to-device messages to application services, as specified by [MSC2409](matrix-org/matrix-spec-proposals#2409). ([\#11215](matrix-org/synapse#11215), [\#11966](matrix-org/synapse#11966)) - Remove account data (including client config, push rules and ignored users) upon user deactivation. ([\#11655](matrix-org/synapse#11655)) - Experimental support for [MSC3666](matrix-org/matrix-spec-proposals#3666): including bundled aggregations in server side search results. ([\#11837](matrix-org/synapse#11837)) - Enable cache time-based expiry by default. The `expiry_time` config flag has been superseded by `expire_caches` and `cache_entry_ttl`. ([\#11849](matrix-org/synapse#11849)) - Add a callback to allow modules to allow or forbid a 3PID (email address, phone number) from being associated to a local account. ([\#11854](matrix-org/synapse#11854)) - Stabilize support and remove unstable endpoints for [MSC3231](matrix-org/matrix-spec-proposals#3231). Clients must switch to the stable identifier and endpoint. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#stablisation-of-msc3231) for more information. ([\#11867](matrix-org/synapse#11867)) - Allow modules to retrieve the current instance's server name and worker name. ([\#11868](matrix-org/synapse#11868)) - Use a dedicated configurable rate limiter for 3PID invites. ([\#11892](matrix-org/synapse#11892)) - Support the stable API endpoint for [MSC3283](matrix-org/matrix-spec-proposals#3283): new settings in `/capabilities` endpoint. ([\#11933](matrix-org/synapse#11933), [\#11989](matrix-org/synapse#11989)) - Support the `dir` parameter on the `/relations` endpoint, per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#11941](matrix-org/synapse#11941)) - Experimental implementation of [MSC3706](matrix-org/matrix-spec-proposals#3706): extensions to `/send_join` to support reduced response size. ([\#11967](matrix-org/synapse#11967)) Bugfixes -------- - Fix [MSC2716](matrix-org/matrix-spec-proposals#2716) historical messages backfilling in random order on remote homeservers. ([\#11114](matrix-org/synapse#11114)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#11890](matrix-org/synapse#11890)) - Fix a long-standing bug where some unknown endpoints would return HTML error pages instead of JSON `M_UNRECOGNIZED` errors. ([\#11930](matrix-org/synapse#11930)) - Implement an allow list of content types for which we will attempt to preview a URL. This prevents Synapse from making useless longer-lived connections to streaming media servers. ([\#11936](matrix-org/synapse#11936)) - Fix a long-standing bug where pagination tokens from `/sync` and `/messages` could not be provided to the `/relations` API. ([\#11952](matrix-org/synapse#11952)) - Require that modules register their callbacks using keyword arguments. ([\#11975](matrix-org/synapse#11975)) - Fix a long-standing bug where `M_WRONG_ROOM_KEYS_VERSION` errors would not include the specced `current_version` field. ([\#11988](matrix-org/synapse#11988)) Improved Documentation ---------------------- - Fix typo in User Admin API: unpind -> unbind. ([\#11859](matrix-org/synapse#11859)) - Document images returned by the User List Media Admin API can include those generated by URL previews. ([\#11862](matrix-org/synapse#11862)) - Remove outdated MSC1711 FAQ document. ([\#11907](matrix-org/synapse#11907)) - Correct the structured logging configuration example. Contributed by Brad Jones. ([\#11946](matrix-org/synapse#11946)) - Add information on the Synapse release cycle. ([\#11954](matrix-org/synapse#11954)) - Fix broken link in the README to the admin API for password reset. ([\#11955](matrix-org/synapse#11955)) Deprecations and Removals ------------------------- - Drop support for `webclient` listeners and configuring `web_client_location` to a non-HTTP(S) URL. Deprecated configurations are a configuration error. ([\#11895](matrix-org/synapse#11895)) - Remove deprecated `user_may_create_room_with_invites` spam checker callback. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#removal-of-user_may_create_room_with_invites) for more information. ([\#11950](matrix-org/synapse#11950)) - No longer build `.deb` packages for Ubuntu 21.04 Hirsute Hippo, which has now EOLed. ([\#11961](matrix-org/synapse#11961)) Internal Changes ---------------- - Enhance user registration test helpers to make them more useful for tests involving application services and devices. ([\#11615](matrix-org/synapse#11615), [\#11616](matrix-org/synapse#11616)) - Improve performance when fetching bundled aggregations for multiple events. ([\#11660](matrix-org/synapse#11660), [\#11752](matrix-org/synapse#11752)) - Fix type errors introduced by new annotations in the Prometheus Client library. ([\#11832](matrix-org/synapse#11832)) - Add missing type hints to replication code. ([\#11856](matrix-org/synapse#11856), [\#11938](matrix-org/synapse#11938)) - Ensure that `opentracing` scopes are activated and closed at the right time. ([\#11869](matrix-org/synapse#11869)) - Improve opentracing for incoming federation requests. ([\#11870](matrix-org/synapse#11870)) - Improve internal docstrings in `synapse.util.caches`. ([\#11876](matrix-org/synapse#11876)) - Do not needlessly clear the `get_users_in_room` and `get_users_in_room_with_profiles` caches when any room state changes. ([\#11878](matrix-org/synapse#11878)) - Convert `ApplicationServiceTestCase` to use `simple_async_mock`. ([\#11880](matrix-org/synapse#11880)) - Remove experimental changes to the default push rules which were introduced in Synapse 1.19.0 but never enabled. ([\#11884](matrix-org/synapse#11884)) - Disable coverage calculation for olddeps build. ([\#11888](matrix-org/synapse#11888)) - Preparation to support sending device list updates to application services. ([\#11905](matrix-org/synapse#11905)) - Add a test that checks users receive their own device list updates down `/sync`. ([\#11909](matrix-org/synapse#11909)) - Run Complement tests sequentially. ([\#11910](matrix-org/synapse#11910)) - Various refactors to the application service notifier code. ([\#11911](matrix-org/synapse#11911), [\#11912](matrix-org/synapse#11912)) - Tests: replace mocked `Authenticator` with the real thing. ([\#11913](matrix-org/synapse#11913)) - Various refactors to the typing notifications code. ([\#11914](matrix-org/synapse#11914)) - Use the proper type for the `Content-Length` header in the `UploadResource`. ([\#11927](matrix-org/synapse#11927)) - Remove an unnecessary ignoring of type hints due to fixes in upstream packages. ([\#11939](matrix-org/synapse#11939)) - Add missing type hints. ([\#11953](matrix-org/synapse#11953)) - Fix an import cycle in `synapse.event_auth`. ([\#11965](matrix-org/synapse#11965)) - Unpin `frozendict` but exclude the known bad version 2.1.2. ([\#11969](matrix-org/synapse#11969)) - Prepare for rename of default Complement branch. ([\#11971](matrix-org/synapse#11971)) - Fetch Synapse's version using a helper from `matrix-common`. ([\#11979](matrix-org/synapse#11979))

H-Shay added 3 commits January 27, 2022 12:13

make using cache expiry the default behavior

f798aba

newsfragment

adbd8ed

update comment

bdeaf52

H-Shay requested a review from a team as a code owner January 27, 2022 20:38

newsfragment and update sample config

83661c5

anoadragon453 suggested changes Jan 28, 2022

View reviewed changes

changelog.d/11849.misc Outdated Show resolved Hide resolved

anoadragon453 suggested changes Jan 31, 2022

View reviewed changes

H-Shay added 2 commits February 3, 2022 13:45

add boolean flag, rename expiry time

4eb3e6a

regenerate sample config

652c620

H-Shay requested a review from anoadragon453 February 3, 2022 22:47

anoadragon453 previously requested changes Feb 4, 2022

View reviewed changes

H-Shay added 5 commits February 4, 2022 10:26

requested changes

6784e1e

Merge branch 'develop' into shay/default_expiry

253edca

add note in upgrade notes about changes

775cbf7

lint

6dddbc7

renew sample config

8efac83

H-Shay requested review from anoadragon453 and a team February 4, 2022 19:25

H-Shay added 3 commits February 9, 2022 11:46

follow config code style

c86e536

follow config code style

327ea5a

regenerate sample config

8d06aac

reivilibre suggested changes Feb 10, 2022

View reviewed changes

H-Shay added 2 commits February 10, 2022 11:30

requested changes

f37e95f

d'oh! lints

3299583

H-Shay requested a review from reivilibre February 10, 2022 20:25

reivilibre approved these changes Feb 11, 2022

View reviewed changes

cleanup v1 header and reword flags

77ac0d7

reivilibre approved these changes Feb 11, 2022

View reviewed changes

Merge branch 'develop' into shay/default_expiry

604145a

H-Shay merged commit b2b971f into develop Feb 11, 2022

H-Shay deleted the shay/default_expiry branch February 11, 2022 19:05

richvdh reviewed Feb 14, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable cache time-based expiry by default #11849

Enable cache time-based expiry by default #11849

H-Shay commented Jan 27, 2022

anoadragon453 left a comment

reivilibre commented Jan 28, 2022

H-Shay commented Jan 28, 2022

richvdh commented Jan 31, 2022

anoadragon453 commented Jan 31, 2022

anoadragon453 left a comment

callahad commented Jan 31, 2022

richvdh commented Jan 31, 2022

H-Shay commented Jan 31, 2022

H-Shay commented Feb 3, 2022

anoadragon453 left a comment

anoadragon453 Feb 4, 2022

anoadragon453 Feb 4, 2022

reivilibre Feb 10, 2022

reivilibre left a comment

reivilibre Feb 10, 2022

reivilibre left a comment

reivilibre Feb 11, 2022

H-Shay Feb 11, 2022

reivilibre Feb 11, 2022

H-Shay Feb 11, 2022

H-Shay commented Feb 11, 2022

anoadragon453 commented Feb 11, 2022

richvdh left a comment

richvdh Feb 14, 2022

richvdh Feb 14, 2022

H-Shay commented Feb 14, 2022

clokep commented Feb 14, 2022

		@@ -0,0 +1 @@
		Enable cache time-based expiry by default. The `expiry_time` config flag will be superseded by `expire_caches` and `cache_entry_ttl`.

Enable cache time-based expiry by default #11849

Enable cache time-based expiry by default #11849

Conversation

H-Shay commented Jan 27, 2022

anoadragon453 left a comment

Choose a reason for hiding this comment

reivilibre commented Jan 28, 2022

H-Shay commented Jan 28, 2022

richvdh commented Jan 31, 2022

anoadragon453 commented Jan 31, 2022

anoadragon453 left a comment

Choose a reason for hiding this comment

callahad commented Jan 31, 2022

richvdh commented Jan 31, 2022

H-Shay commented Jan 31, 2022

H-Shay commented Feb 3, 2022

anoadragon453 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reivilibre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

H-Shay commented Feb 11, 2022

anoadragon453 commented Feb 11, 2022

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

H-Shay commented Feb 14, 2022

clokep commented Feb 14, 2022