Use importlib.metadata to read requirements #12088

DMRobertson · 2022-02-25T13:31:54Z

Part of #11537. We will have to use importlib.metadata to read our dependencies at runtime once we have moved to a declarative config. I do this now to ease the transition.

Before this change, python_dependencies has four confused purposes:

setup.py evaluates this to define matrix-synapse's dependencies and optional
dependencies.
The twisted olddeps CI job edit this file in-place so that we install the oldest
version of all dependencies. Similarly, the twisted trunk CI job patches this file
to use twisted trunk from github.
This data is made available at runtime for the application to check dependencies
are present and correctly versioned.
Finally, the script can be executed directly to print out all dependencies
(though this lacks the extras grouping information). This is mentioned in the
upgrade notes for 0.7.0 and nowhere else.

Purposes (1) and (2) have no right belonging in application code. I am relocating and reimplementing purpose (3) here. I propose nuking purpose (4) as well while we're at it---again, this is not something the application source code should be concerned with.

I've tested this with some dirty mocked test cases, and also:

$ pip install 'twisted<18.9' 'twisted>16'
Collecting twisted<18.9
  Downloading Twisted-18.7.0.tar.bz2 (3.1 MB)
     |████████████████████████████████|
 ...
    Running setup.py install for twisted ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
matrix-synapse 1.53.0 requires Twisted>=18.9.0, but you have twisted 18.7.0 which is incompatible.
Successfully installed PyHamcrest-2.0.3 twisted-18.7.0

$ python -m synapse.app
ERROR:root:Need Twisted>=18.9.0, but got Twisted==18.7.0
Missing Requirements: "Twisted"
To install run:
    pip install --upgrade --force "Twisted"

I've tried to make this clearer. We start by working out which of Synapse's requirements we need to be installed here and now. I was surprised that there wasn't an easier way to see which packages were installed by a given extra. I've pulled out the error messages into functions that deal with "is this for an extra or not". And I've rearranged the loop over two different sets of requirements into one loop with a "must be instaled" flag. I hope you agree that this is clearer.

DMRobertson · 2022-02-25T18:04:54Z

Sytest is not happy. Grr.

DMRobertson · 2022-02-25T18:12:13Z

#12089 seems to have hit the same set of sytest failures(?)

anoadragon453 · 2022-02-28T10:34:15Z

Sytest failures are affecting develop as well, yes: https://github.com/matrix-org/synapse/runs/5356507389?check_suite_focus=true

squahtx

LGTM!

Just to confirm, this PR only touches purpose (3)?

squahtx · 2022-03-01T14:59:48Z

synapse/util/check_dependencies.py

+EXTRAS = {
+    v
+    for k, v in metadata.distribution("matrix-synapse").metadata.items()
+    if k == "Provides-Extra"
+}


set(metadata.distribution("matrix-synapse").metadata.get_all("Provides-Extra")) perhaps?
Or even set(metadata.metadata("matrix-synapse").get_all("Provides-Extra"))

Disclaimer: I've only confirmed it works on my machine and not in general

That seems neater---will check this out!

Seems kosher to me: https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.PackageMetadata.get_all

squahtx · 2022-03-01T15:22:05Z

tests/util/test_check_dependencies.py

+old = DummyDistribution("0.1.2")
+new = DummyDistribution("1.2.3")
+
+# could probably use stdlib TestCase --- no need for twisted here


I've got no strong opinions on which TestCase class we use.

squahtx · 2022-03-01T15:25:37Z

synapse/util/check_dependencies.py

-    if req.marker is not None and not req.marker.evaluate():
-        # not required for this environment
-        return
+    deps_needed = []


nit: needed is a tad misleading. deps_unfulfilled maybe?

Can do. This came from the original implementation, but finding the right words for all of this stuff is tricky.

DMRobertson · 2022-03-01T15:26:46Z

Just to confirm, this PR only touches purpose (3)?

That's right. (1) and (4) will die out when I do more of this poetry stuff. And (2) will need to a replacement.

DMRobertson · 2022-03-01T15:46:13Z

Needs a linter script run, but I'll put it to auto-merge after that.

@AndrewRyanChama

Synapse 1.54.0rc1 (2022-03-02) ============================== Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier. Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later. Features -------- - Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](#11617)) - Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](#11985)) - Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](#12000)) - Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](#12001), [\#12067](#12067)) - Enable modules to set a custom display name when registering a user. ([\#12009](#12009)) - Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](#12020), ([\#12022](#12022)) - Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](#12021)) - Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](#12058)) - Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](#12062)) Bugfixes -------- - Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](#11992)) - Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](#11999)) - Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](#12024)) - Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](#12037)) - Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](#12056)) - Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](#12077)) - Fix occasional `Unhandled error in Deferred` error message. ([\#12089](#12089)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](#12098)) - Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](#12100)) - Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](#12105)) - Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](#11835)) Updates to the Docker image --------------------------- - The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](#11997)) - Use Python 3.9 in Docker images by default. ([\#12112](#12112)) Improved Documentation ---------------------- - Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](#11599)) - Explain the meaning of spam checker callbacks' return values. ([\#12003](#12003)) - Clarify information about external Identity Provider IDs. ([\#12004](#12004)) Deprecations and Removals ------------------------- - Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](#11865)) - Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](#12008)) - Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](#12018)) - Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](#12073)) Internal Changes ---------------- - Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](#11808)) - Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](#11900)) - Add tests for device list changes between local users. ([\#11972](#11972)) - Optimise calculating `device_list` changes in `/sync`. ([\#11974](#11974)) - Add missing type hints to storage classes. ([\#11984](#11984)) - Refactor the search code for improved readability. ([\#11991](#11991)) - Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](#11994)) - Limit concurrent joins from applications services. ([\#11996](#11996)) - Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](#12005), [\#12039](#12039)) - Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](#12011)) - Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](#12012)) - Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](#12013)) - Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](#12015)) - Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](#12016)) - Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](#12019)) - Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](#12025)) - Upgrade Mypy to version 0.931. ([\#12030](#12030)) - Remove legacy `HomeServer.get_datastore()`. ([\#12031](#12031), [\#12070](#12070)) - Minor typing fixes. ([\#12034](#12034), [\#12069](#12069)) - After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](#12041)) - Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](#12051)) - Move configuration out of `setup.cfg`. ([\#12052](#12052), [\#12059](#12059)) - Fix error message when a worker process fails to talk to another worker process. ([\#12060](#12060)) - Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](#12063)) - Add type hints to `tests/rest/client`. ([\#12066](#12066), [\#12072](#12072), [\#12084](#12084), [\#12094](#12094)) - Add some logging to `/sync` to try and track down #11916. ([\#12068](#12068)) - Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](#12088)) - Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](#12092)) - Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](#12099)) - Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](#12106)) - Improve exception handling for concurrent execution. ([\#12109](#12109)) - Advertise support for Python 3.10 in packaging files. ([\#12111](#12111)) - Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](#12119))

* Fix incorrect argument in test case * Add copyright header * Docstring and __all__ * Exclude dev depenencies * Use changelog from #12088 * Include version in error messages This will hopefully distinguish between the version of the source code and the version of the distribution package that is installed. * Linter script is your friend

It’s just occurred to me that #12088 pulled in the “packaging” package (~=21.3). I pulled in the newest version I had at the time, and I didn’t check this with the packagers, sorry. (Also sorry to bring up the P-word again.) I only use it for packaging.requirements.Requirements. Which was added in packaging 16.1: https://github.com/pypa/packaging/releases/tag/16.1 https://pkgs.org/download/python3-packaging suggests that the oldest version we care about is 17.1 in Ubuntu Bionic. So I think with this bound we're hunky dory.

It’s just occurred to me that #12088 pulled in the “packaging” package (~=21.3). I pulled in the newest version I had at the time. I only use it for packaging.requirements.Requirements. Which was added in packaging 16.1: https://github.com/pypa/packaging/releases/tag/16.1 https://pkgs.org/download/python3-packaging suggests that the oldest version we care about is 17.1 in Ubuntu Bionic. So I think with this bound we're hunky dory.

DMRobertson · 2022-03-29T12:46:53Z

This change didn't mark importlib_metadata as a requirement! We are fortunate that the backport is pulled in by other dependencies.

Missed in #12088.

Missed in #12088. We got away with it because we were indirectly importing it in other places. Without this, we encounter pain in `poetry export`; and besides, we should be explicit about our direct dependencies.

Missed in #12088. We got away with it because we were indirectly requiring it via other packages. Without this change, we encounter pain in `poetry export`; and besides, we should be explicit about our direct dependencies.

@AndrewRyanChama

Synapse 1.54.0 (2022-03-08) =========================== Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier. Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later. Bugfixes -------- - Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\matrix-org#12141](matrix-org#12141)) - Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\matrix-org#12129](matrix-org#12129), [\matrix-org#12177](matrix-org#12177)) Internal Changes ---------------- - Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\matrix-org#12127](matrix-org#12127)) - Relax the version guard for "packaging" added in [\matrix-org#12088](matrix-org#12088). ([\matrix-org#12166](matrix-org#12166)) Synapse 1.54.0rc1 (2022-03-02) ============================== Features -------- - Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\matrix-org#11617](matrix-org#11617)) - Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\matrix-org#11985](matrix-org#11985)) - Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\matrix-org#12000](matrix-org#12000)) - Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\matrix-org#12001](matrix-org#12001), [\matrix-org#12067](matrix-org#12067)) - Enable modules to set a custom display name when registering a user. ([\matrix-org#12009](matrix-org#12009)) - Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\matrix-org#12020](matrix-org#12020), ([\matrix-org#12022](matrix-org#12022)) - Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\matrix-org#12021](matrix-org#12021)) - Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\matrix-org#12058](matrix-org#12058)) - Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\matrix-org#12062](matrix-org#12062)) Bugfixes -------- - Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\matrix-org#11992](matrix-org#11992)) - Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\matrix-org#11999](matrix-org#11999)) - Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\matrix-org#12024](matrix-org#12024)) - Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\matrix-org#12037](matrix-org#12037)) - Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\matrix-org#12056](matrix-org#12056)) - Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\matrix-org#12077](matrix-org#12077)) - Fix occasional `Unhandled error in Deferred` error message. ([\matrix-org#12089](matrix-org#12089)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\matrix-org#12098](matrix-org#12098)) - Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\matrix-org#12100](matrix-org#12100)) - Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\matrix-org#12105](matrix-org#12105)) - Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\matrix-org#11835](matrix-org#11835)) Updates to the Docker image --------------------------- - The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\matrix-org#11997](matrix-org#11997)) - Use Python 3.9 in Docker images by default. ([\matrix-org#12112](matrix-org#12112)) Improved Documentation ---------------------- - Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\matrix-org#11599](matrix-org#11599)) - Explain the meaning of spam checker callbacks' return values. ([\matrix-org#12003](matrix-org#12003)) - Clarify information about external Identity Provider IDs. ([\matrix-org#12004](matrix-org#12004)) Deprecations and Removals ------------------------- - Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\matrix-org#11865](matrix-org#11865)) - Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\matrix-org#12008](matrix-org#12008)) - Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\matrix-org#12018](matrix-org#12018)) - Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\matrix-org#12073](matrix-org#12073)) Internal Changes ---------------- - Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\matrix-org#11808](matrix-org#11808)) - Remove unnecessary condition on knock -> leave auth rule check. ([\matrix-org#11900](matrix-org#11900)) - Add tests for device list changes between local users. ([\matrix-org#11972](matrix-org#11972)) - Optimise calculating `device_list` changes in `/sync`. ([\matrix-org#11974](matrix-org#11974)) - Add missing type hints to storage classes. ([\matrix-org#11984](matrix-org#11984)) - Refactor the search code for improved readability. ([\matrix-org#11991](matrix-org#11991)) - Move common deduplication code down into `_auth_and_persist_outliers`. ([\matrix-org#11994](matrix-org#11994)) - Limit concurrent joins from applications services. ([\matrix-org#11996](matrix-org#11996)) - Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\matrix-org#12005](matrix-org#12005), [\matrix-org#12039](matrix-org#12039)) - Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\matrix-org#12011](matrix-org#12011)) - Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\matrix-org#12012](matrix-org#12012)) - Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\matrix-org#12013](matrix-org#12013)) - Configure `tox` to use `venv` rather than `virtualenv`. ([\matrix-org#12015](matrix-org#12015)) - Fix bug in `StateFilter.return_expanded()` and add some tests. ([\matrix-org#12016](matrix-org#12016)) - Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\matrix-org#12019](matrix-org#12019)) - Update the `olddeps` CI job to use an old version of `markupsafe`. ([\matrix-org#12025](matrix-org#12025)) - Upgrade Mypy to version 0.931. ([\matrix-org#12030](matrix-org#12030)) - Remove legacy `HomeServer.get_datastore()`. ([\matrix-org#12031](matrix-org#12031), [\matrix-org#12070](matrix-org#12070)) - Minor typing fixes. ([\matrix-org#12034](matrix-org#12034), [\matrix-org#12069](matrix-org#12069)) - After joining a room, create a dedicated logcontext to process the queued events. ([\matrix-org#12041](matrix-org#12041)) - Tidy up GitHub Actions config which builds distributions for PyPI. ([\matrix-org#12051](matrix-org#12051)) - Move configuration out of `setup.cfg`. ([\matrix-org#12052](matrix-org#12052), [\matrix-org#12059](matrix-org#12059)) - Fix error message when a worker process fails to talk to another worker process. ([\matrix-org#12060](matrix-org#12060)) - Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\matrix-org#12063](matrix-org#12063)) - Add type hints to `tests/rest/client`. ([\matrix-org#12066](matrix-org#12066), [\matrix-org#12072](matrix-org#12072), [\matrix-org#12084](matrix-org#12084), [\matrix-org#12094](matrix-org#12094)) - Add some logging to `/sync` to try and track down matrix-org#11916. ([\matrix-org#12068](matrix-org#12068)) - Inspect application dependencies using `importlib.metadata` or its backport. ([\matrix-org#12088](matrix-org#12088)) - Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\matrix-org#12092](matrix-org#12092)) - Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\matrix-org#12099](matrix-org#12099)) - Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\matrix-org#12106](matrix-org#12106)) - Improve exception handling for concurrent execution. ([\matrix-org#12109](matrix-org#12109)) - Advertise support for Python 3.10 in packaging files. ([\matrix-org#12111](matrix-org#12111)) - Move CI checks out of tox, to facilitate a move to using poetry. ([\matrix-org#12119](matrix-org#12119))

@AndrewRyanChama

Synapse 1.54.0 (2022-03-08) =========================== Please note that this will be the last release of Synapse that is compatible with Mjolnir 1.3.1 and earlier. Administrators of servers which have the Mjolnir module installed are advised to upgrade Mjolnir to version 1.3.2 or later. Bugfixes -------- - Fix a bug introduced in Synapse 1.54.0rc1 preventing the new module callbacks introduced in this release from being registered by modules. ([\#12141](matrix-org/synapse#12141)) - Fix a bug introduced in Synapse 1.54.0rc1 where runtime dependency version checks would mistakenly check development dependencies if they were present and would not accept pre-release versions of dependencies. ([\#12129](matrix-org/synapse#12129), [\#12177](matrix-org/synapse#12177)) Internal Changes ---------------- - Update release script to insert the previous version when writing "No significant changes" line in the changelog. ([\#12127](matrix-org/synapse#12127)) - Relax the version guard for "packaging" added in [\#12088](matrix-org/synapse#12088). ([\#12166](matrix-org/synapse#12166)) Synapse 1.54.0rc1 (2022-03-02) ============================== Features -------- - Add support for [MSC3202](matrix-org/matrix-spec-proposals#3202): sending one-time key counts and fallback key usage states to Application Services. ([\#11617](matrix-org/synapse#11617)) - Improve the generated URL previews for some web pages. Contributed by @AndrewRyanChama. ([\#11985](matrix-org/synapse#11985)) - Track cache invalidations in Prometheus metrics, as already happens for cache eviction based on size or time. ([\#12000](matrix-org/synapse#12000)) - Implement experimental support for [MSC3720](matrix-org/matrix-spec-proposals#3720) (account status endpoints). ([\#12001](matrix-org/synapse#12001), [\#12067](matrix-org/synapse#12067)) - Enable modules to set a custom display name when registering a user. ([\#12009](matrix-org/synapse#12009)) - Advertise Matrix 1.1 and 1.2 support on `/_matrix/client/versions`. ([\#12020](matrix-org/synapse#12020), ([\#12022](matrix-org/synapse#12022)) - Support only the stable identifier for [MSC3069](matrix-org/matrix-spec-proposals#3069 `is_guest` on `/_matrix/client/v3/account/whoami`. ([\#12021](matrix-org/synapse#12021)) - Use room version 9 as the default room version (per [MSC3589](matrix-org/matrix-spec-proposals#3589)). ([\#12058](matrix-org/synapse#12058)) - Add module callbacks to react to user deactivation status changes (i.e. deactivations and reactivations) and profile updates. ([\#12062](matrix-org/synapse#12062)) Bugfixes -------- - Fix a bug introduced in Synapse 1.48.0 where an edit of the latest event in a thread would not be properly applied to the thread summary. ([\#11992](matrix-org/synapse#11992)) - Fix long-standing bug where the `get_rooms_for_user` cache was not correctly invalidated for remote users when the server left a room. ([\#11999](matrix-org/synapse#11999)) - Fix a 500 error with Postgres when looking backwards with the [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event?dir=b` endpoint. ([\#12024](matrix-org/synapse#12024)) - Properly fix a long-standing bug where wrong data could be inserted into the `event_search` table when using SQLite. This could block running `synapse_port_db` with an `argument of type 'int' is not iterable` error. This bug was partially fixed by a change in Synapse 1.44.0. ([\#12037](matrix-org/synapse#12037)) - Fix slow performance of `/logout` in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens in version 1.38.0. ([\#12056](matrix-org/synapse#12056)) - Fix a long-standing bug where Synapse would make additional failing requests over federation for missing data. ([\#12077](matrix-org/synapse#12077)) - Fix occasional `Unhandled error in Deferred` error message. ([\#12089](matrix-org/synapse#12089)) - Fix a bug introduced in Synapse 1.51.0 where incoming federation transactions containing at least one EDU would be dropped if debug logging was enabled for `synapse.8631_debug`. ([\#12098](matrix-org/synapse#12098)) - Fix a long-standing bug which could cause push notifications to malfunction if `use_frozen_dicts` was set in the configuration. ([\#12100](matrix-org/synapse#12100)) - Fix an extremely rare, long-standing bug in `ReadWriteLock` that would cause an error when a newly unblocked writer completes instantly. ([\#12105](matrix-org/synapse#12105)) - Make a `POST` to `/rooms/<room_id>/receipt/m.read/<event_id>` only trigger a push notification if the count of unread messages is different to the one in the last successfully sent push. This reduces server load and load on the receiving device. ([\#11835](matrix-org/synapse#11835)) Updates to the Docker image --------------------------- - The Docker image no longer automatically creates a temporary volume at `/data`. This is not expected to affect normal usage. ([\#11997](matrix-org/synapse#11997)) - Use Python 3.9 in Docker images by default. ([\#12112](matrix-org/synapse#12112)) Improved Documentation ---------------------- - Document support for the `to_device`, `account_data`, `receipts`, and `presence` stream writers for workers. ([\#11599](matrix-org/synapse#11599)) - Explain the meaning of spam checker callbacks' return values. ([\#12003](matrix-org/synapse#12003)) - Clarify information about external Identity Provider IDs. ([\#12004](matrix-org/synapse#12004)) Deprecations and Removals ------------------------- - Deprecate using `synctl` with the config option `synctl_cache_factor` and print a warning if a user still uses this option. ([\#11865](matrix-org/synapse#11865)) - Remove support for the legacy structured logging configuration (please see the the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#legacy-structured-logging-configuration-removal) if you are using `structured: true` in the Synapse configuration). ([\#12008](matrix-org/synapse#12008)) - Drop support for [MSC3283](matrix-org/matrix-spec-proposals#3283) unstable flags now that the stable flags are supported. ([\#12018](matrix-org/synapse#12018)) - Remove the unstable `/spaces` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#12073](matrix-org/synapse#12073)) Internal Changes ---------------- - Make the `get_room_version` method use `get_room_version_id` to benefit from caching. ([\#11808](matrix-org/synapse#11808)) - Remove unnecessary condition on knock -> leave auth rule check. ([\#11900](matrix-org/synapse#11900)) - Add tests for device list changes between local users. ([\#11972](matrix-org/synapse#11972)) - Optimise calculating `device_list` changes in `/sync`. ([\#11974](matrix-org/synapse#11974)) - Add missing type hints to storage classes. ([\#11984](matrix-org/synapse#11984)) - Refactor the search code for improved readability. ([\#11991](matrix-org/synapse#11991)) - Move common deduplication code down into `_auth_and_persist_outliers`. ([\#11994](matrix-org/synapse#11994)) - Limit concurrent joins from applications services. ([\#11996](matrix-org/synapse#11996)) - Preparation for faster-room-join work: when parsing the `send_join` response, get the `m.room.create` event from `state`, not `auth_chain`. ([\#12005](matrix-org/synapse#12005), [\#12039](matrix-org/synapse#12039)) - Preparation for faster-room-join work: parse MSC3706 fields in send_join response. ([\#12011](matrix-org/synapse#12011)) - Preparation for faster-room-join work: persist information on which events and rooms have partial state to the database. ([\#12012](matrix-org/synapse#12012)) - Preparation for faster-room-join work: Support for calling `/federation/v1/state` on a remote server. ([\#12013](matrix-org/synapse#12013)) - Configure `tox` to use `venv` rather than `virtualenv`. ([\#12015](matrix-org/synapse#12015)) - Fix bug in `StateFilter.return_expanded()` and add some tests. ([\#12016](matrix-org/synapse#12016)) - Use Matrix v1.1 endpoints (`/_matrix/client/v3/auth/...`) in fallback auth HTML forms. ([\#12019](matrix-org/synapse#12019)) - Update the `olddeps` CI job to use an old version of `markupsafe`. ([\#12025](matrix-org/synapse#12025)) - Upgrade Mypy to version 0.931. ([\#12030](matrix-org/synapse#12030)) - Remove legacy `HomeServer.get_datastore()`. ([\#12031](matrix-org/synapse#12031), [\#12070](matrix-org/synapse#12070)) - Minor typing fixes. ([\#12034](matrix-org/synapse#12034), [\#12069](matrix-org/synapse#12069)) - After joining a room, create a dedicated logcontext to process the queued events. ([\#12041](matrix-org/synapse#12041)) - Tidy up GitHub Actions config which builds distributions for PyPI. ([\#12051](matrix-org/synapse#12051)) - Move configuration out of `setup.cfg`. ([\#12052](matrix-org/synapse#12052), [\#12059](matrix-org/synapse#12059)) - Fix error message when a worker process fails to talk to another worker process. ([\#12060](matrix-org/synapse#12060)) - Fix using the `complement.sh` script without specifying a directory or a branch. Contributed by Nico on behalf of Famedly. ([\#12063](matrix-org/synapse#12063)) - Add type hints to `tests/rest/client`. ([\#12066](matrix-org/synapse#12066), [\#12072](matrix-org/synapse#12072), [\#12084](matrix-org/synapse#12084), [\#12094](matrix-org/synapse#12094)) - Add some logging to `/sync` to try and track down #11916. ([\#12068](matrix-org/synapse#12068)) - Inspect application dependencies using `importlib.metadata` or its backport. ([\#12088](matrix-org/synapse#12088)) - Use `assertEqual` instead of the deprecated `assertEquals` in test code. ([\#12092](matrix-org/synapse#12092)) - Move experimental support for [MSC3440](matrix-org/matrix-spec-proposals#3440) to `/versions`. ([\#12099](matrix-org/synapse#12099)) - Add `stop_cancellation` utility function to stop `Deferred`s from being cancelled. ([\#12106](matrix-org/synapse#12106)) - Improve exception handling for concurrent execution. ([\#12109](matrix-org/synapse#12109)) - Advertise support for Python 3.10 in packaging files. ([\#12111](matrix-org/synapse#12111)) - Move CI checks out of tox, to facilitate a move to using poetry. ([\#12119](matrix-org/synapse#12119))

DMRobertson force-pushed the dmr/python_dependencies_metadata branch from d08d7e4 to dad6028 Compare February 25, 2022 13:34

David Robertson added 2 commits February 25, 2022 13:37

Pull runtime dep checks into their own module

e49fd50

Changelog

94ddc49

DMRobertson force-pushed the dmr/python_dependencies_metadata branch 3 times, most recently from 3fd5e4d to 3ec1214 Compare February 25, 2022 17:36

David Robertson added 2 commits February 25, 2022 17:45

Type hints for existing code

51e76ec

DMRobertson force-pushed the dmr/python_dependencies_metadata branch from 3ec1214 to 8f84471 Compare February 25, 2022 17:45

Test cases

c7eee4f

DMRobertson force-pushed the dmr/python_dependencies_metadata branch from 8f84471 to c7eee4f Compare February 25, 2022 17:48

DMRobertson marked this pull request as ready for review February 25, 2022 17:55

DMRobertson requested a review from a team as a code owner February 25, 2022 17:55

DMRobertson changed the title ~~WIP: use importlib.metadata to read requirements~~ Use importlib.metadata to read requirements Feb 25, 2022

DMRobertson removed the request for review from a team February 25, 2022 18:04

DMRobertson requested a review from a team February 28, 2022 11:24

squahtx approved these changes Mar 1, 2022

View reviewed changes

squahtx reviewed Mar 1, 2022

View reviewed changes

David Robertson added 2 commits March 1, 2022 15:37

Use .get_all; add distribution pkg name constant

9bed98f

needed -> unfulfilled

903e95c

DMRobertson requested a review from squahtx March 1, 2022 15:39

squahtx approved these changes Mar 1, 2022

View reviewed changes

isort && flake8

40dbcb2

DMRobertson enabled auto-merge (squash) March 1, 2022 15:50

And fix an extra newline

24a4b54

DMRobertson merged commit 313581e into develop Mar 1, 2022

DMRobertson deleted the dmr/python_dependencies_metadata branch March 1, 2022 17:44

DMRobertson pushed a commit that referenced this pull request Mar 2, 2022

Use changelog from #12088

52c274d

DMRobertson mentioned this pull request Mar 2, 2022

Don't impose version checks on dev extras at runtime #12129

Merged

DMRobertson pushed a commit that referenced this pull request Mar 3, 2022

Use changelog from #12088

572a1f3

DMRobertson mentioned this pull request Mar 4, 2022

Relax version guard for packaging #12166

Merged

reivilibre mentioned this pull request Mar 8, 2022

RC versions of dependencies don't satisfy the run-time dependency checker; Need Twisted>=18.9.0, but got Twisted==21.7.0rc3 (1.54.0rc1 suspected regression) #12176

Closed

DMRobertson pushed a commit that referenced this pull request Mar 29, 2022

Add missing dependency on importlib_metadata

cf6e424

Missed in #12088.

DMRobertson pushed a commit that referenced this pull request Mar 30, 2022

Add missing dependency on importlib_metadata

c15ffca

Missed in #12088.

DMRobertson pushed a commit that referenced this pull request Mar 30, 2022

Add missing dependency on importlib_metadata

0775dfc

Missed in #12088.

DMRobertson mentioned this pull request Apr 5, 2022

Add missing dependency on importlib_metadata #12384

Merged

DMRobertson mentioned this pull request Apr 12, 2022

Specify tls extra for Twisted dependency. #12444

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use importlib.metadata to read requirements #12088

Use importlib.metadata to read requirements #12088

DMRobertson commented Feb 25, 2022 •

edited

Loading

DMRobertson commented Feb 25, 2022

DMRobertson commented Feb 25, 2022

anoadragon453 commented Feb 28, 2022

squahtx left a comment

squahtx Mar 1, 2022

DMRobertson Mar 1, 2022

DMRobertson Mar 1, 2022

squahtx Mar 1, 2022

squahtx Mar 1, 2022

DMRobertson Mar 1, 2022

DMRobertson commented Mar 1, 2022

DMRobertson commented Mar 1, 2022

DMRobertson commented Mar 29, 2022

Use importlib.metadata to read requirements #12088

Use importlib.metadata to read requirements #12088

Conversation

DMRobertson commented Feb 25, 2022 • edited Loading

DMRobertson commented Feb 25, 2022

DMRobertson commented Feb 25, 2022

anoadragon453 commented Feb 28, 2022

squahtx left a comment

Choose a reason for hiding this comment

squahtx Mar 1, 2022

Choose a reason for hiding this comment

DMRobertson Mar 1, 2022

Choose a reason for hiding this comment

DMRobertson Mar 1, 2022

Choose a reason for hiding this comment

squahtx Mar 1, 2022

Choose a reason for hiding this comment

squahtx Mar 1, 2022

Choose a reason for hiding this comment

DMRobertson Mar 1, 2022

Choose a reason for hiding this comment

DMRobertson commented Mar 1, 2022

DMRobertson commented Mar 1, 2022

DMRobertson commented Mar 29, 2022

DMRobertson commented Feb 25, 2022 •

edited

Loading