Track device list updates per room. #12321

erikjohnston · 2022-03-29T10:01:14Z

This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

we can defer calculating the set of remote servers that need to be poked about the change; and
during /sync and /keys/changes we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both device_lists_changes_in_room and the device_lists_outbound_pokes table synchronously. In a future release we can then bump the database schema compat version to 69 and then we can assume that the new device_lists_changes_in_room exists and is handled.

There is a temporary option to disable writing to device_lists_outbound_pokes synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the device_list_outbound_pokes table.

Reviewable commit-by-commit.

Currently we update the `device_lists_outbound_pokes` at the same time as `device_lists_changes_in_room`, but in future we'll do them at different times.

…perf

synapse/storage/databases/main/devices.py

richvdh · 2022-04-01T14:18:32Z

synapse/storage/databases/main/devices.py

-            await self.db_pool.runInteraction(
-                "add_device_change_to_stream",
-                self._add_device_change_to_stream_txn,
+        num_stream_ids = max(


it might be good to add a comment clarifying why this is a max rather than a total (vis, that we use the same set of stream ids for each of the three tables)

The max got replaced, but I've added a comment the equivalent code

synapse/storage/databases/main/devices.py

synapse/storage/schema/main/delta/69/01device_list_oubound_by_room.sql

synapse/storage/databases/main/devices.py

synapse/storage/schema/main/delta/69/01device_list_oubound_by_room.sql

richvdh · 2022-04-01T14:41:53Z

synapse/storage/databases/main/devices.py

+                    "stream_id": stream_id,
+                    "room_id": room_id,
+                },
+                updatevalues={"converted_to_destinations": True},


this might be a silly question, but why do we do it this way, instead of keeping track of the point we've processed up to?

Err, we could. I don't think I had much reasoning beyond not having to worry about extra tables or handling the duplicate stream IDs.

I've also been thinking about how you'd distribute this across multiple workers, but I'm not sure this helps that or not 🤷

mmmhmm. Generally I prefer to keep our tables append-only where possible (it saves a world of caching problems later on), but fair enough.

synapse/storage/schema/main/delta/69/01device_list_oubound_by_room.sql

richvdh · 2022-04-01T14:47:13Z

synapse/handlers/device.py

-                joined_users = await self.store.get_users_in_room(room_id)
-                hosts.update(get_domain_from_id(u) for u in joined_users)
+        hosts: Optional[Set[str]] = None
+        if self.use_new_device_lists_changes_in_room:


is this the wrong way round? we should skip calculating hosts if use_new_device_lists_changes_in_room?

Bah, yes. And this is the only place that uses it directly ─ everything else infers it from hosts being None ─ so nothing broke.

I'm not sure if there is an easy way to test that this is the right way round?

you might be able to block the _handle_new_device_update_async job in the tests (eg by setting _handle_new_device_update_is_processing), and check what actually gets written to the db?

Not sure it's worth it though.

I guess this code is going to be ripped out in the next release anyway

…room.sql Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

…perf

erikjohnston · 2022-04-04T12:52:52Z

(The last three commits are bug fixes, rather than responding to review comments)

synapse/storage/databases/main/devices.py

synapse/storage/schema/main/delta/69/01device_list_oubound_by_room.sql

richvdh · 2022-04-04T13:09:11Z

synapse/storage/databases/main/devices.py

+                    "stream_id": stream_id,
+                    "room_id": room_id,
+                },
+                updatevalues={"converted_to_destinations": True},


mmmhmm. Generally I prefer to keep our tables append-only where possible (it saves a world of caching problems later on), but fair enough.

richvdh · 2022-04-04T13:15:44Z

synapse/handlers/device.py

@@ -469,19 +488,26 @@ async def notify_device_update(
            # No changes to notify about, so this is a no-op.
            return

-        users_who_share_room = await self.store.get_users_who_share_room_with_user(


FederationSenderDevicesTestCases still stubs this out. Is that still necessary?

richvdh · 2022-04-04T13:17:01Z

synapse/handlers/device.py

-                joined_users = await self.store.get_users_in_room(room_id)
-                hosts.update(get_domain_from_id(u) for u in joined_users)
+        hosts: Optional[Set[str]] = None
+        if self.use_new_device_lists_changes_in_room:


you might be able to block the _handle_new_device_update_async job in the tests (eg by setting _handle_new_device_update_is_processing), and check what actually gets written to the db?

Not sure it's worth it though.

synapse/storage/databases/main/devices.py

richvdh · 2022-04-04T13:21:09Z

synapse/storage/schema/main/delta/69/01device_list_oubound_by_room.sql

+    -- This initialy matches `device_lists_stream.stream_id`. Note that we
+    -- delete older values from `device_lists_stream`, so we can't use a foreign
+    -- constraint here.


this is no longer true as of cf04f1a, right?

That commit is about device_list_outbound_pokes? Rather than device_lists_stream or device_lists_changes_in_room?

oh. confused. sorry.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

richvdh

lgtm!

Carries on from #12321.

DMRobertson · 2022-04-21T10:41:20Z

synapse/storage/databases/main/devices.py

+
+    async def get_uncoverted_outbound_room_pokes(
+        self, limit: int = 10
+    ) -> List[Tuple[str, str, str, int, Optional[Dict[str, str]]]]:


I think the return type here is incorrect: the optional dict should be Optional[str]. (Presumably we json-decode the string elsewhere when needed.)

Xref #12485 (comment)

Synapse 1.57.0 (2022-04-19) =========================== This version includes a [change](matrix-org/synapse#12209) to the way transaction IDs are managed for application services. If your deployment uses a dedicated worker for application service traffic, **it must be stopped** when the database is upgraded (which normally happens when the main process is upgraded), to ensure the change is made safely without any risk of reusing transaction IDs. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/v1.57.0rc1/docs/upgrade.md#upgrading-to-v1570) for more details. No significant changes since 1.57.0rc1. Synapse 1.57.0rc1 (2022-04-12) ============================== Features -------- - Send device list changes to application services as specified by [MSC3202](matrix-org/matrix-spec-proposals#3202), using unstable prefixes. The `msc3202_transaction_extensions` experimental homeserver config option must be enabled and `org.matrix.msc3202: true` must be present in the application service registration file for device list changes to be sent. The "left" field is currently always empty. ([\#11881](matrix-org/synapse#11881)) - Optimise fetching large quantities of missing room state over federation. ([\#12040](matrix-org/synapse#12040)) - Offload the `update_client_ip` background job from the main process to the background worker, when using Redis-based replication. ([\#12251](matrix-org/synapse#12251)) - Move `update_client_ip` background job from the main process to the background worker. ([\#12252](matrix-org/synapse#12252)) - Add a module callback to react to new 3PID (email address, phone number) associations. ([\#12302](matrix-org/synapse#12302)) - Add a configuration option to remove a specific set of rooms from sync responses. ([\#12310](matrix-org/synapse#12310)) - Add a module callback to react to account data changes. ([\#12327](matrix-org/synapse#12327)) - Allow setting user admin status using the module API. Contributed by Famedly. ([\#12341](matrix-org/synapse#12341)) - Reduce overhead of restarting synchrotrons. ([\#12367](matrix-org/synapse#12367), [\#12372](matrix-org/synapse#12372)) - Update `/messages` to use historic pagination tokens if no `from` query parameter is given. ([\#12370](matrix-org/synapse#12370)) - Add a module API for reading and writing global account data. ([\#12391](matrix-org/synapse#12391)) - Support the stable `v1` endpoint for `/relations`, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#12403](matrix-org/synapse#12403)) - Include bundled aggregations in search results ([MSC3666](matrix-org/matrix-spec-proposals#3666)). ([\#12436](matrix-org/synapse#12436)) Bugfixes -------- - Fix a long-standing bug where updates to the server notices user profile (display name/avatar URL) in the configuration would not be applied to pre-existing rooms. Contributed by Jorge Florian. ([\#12115](matrix-org/synapse#12115)) - Fix a long-standing bug where events from ignored users were still considered for bundled aggregations. ([\#12235](matrix-org/synapse#12235), [\#12338](matrix-org/synapse#12338)) - Fix non-member state events not resolving for historical events when used in [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` `state_events_at_start`. ([\#12329](matrix-org/synapse#12329)) - Fix a long-standing bug affecting URL previews that would generate a 500 response instead of a 403 if the previewed URL includes a port that isn't allowed by the relevant blacklist. ([\#12333](matrix-org/synapse#12333)) - Default to `private` room visibility rather than `public` when a client does not specify one, according to spec. ([\#12350](matrix-org/synapse#12350)) - Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string. ([\#12364](matrix-org/synapse#12364), [\#12410](matrix-org/synapse#12410)) - Fix a bug introduced in Synapse 1.49.0 which caused the `synapse_event_persisted_position` metric to have invalid values. ([\#12390](matrix-org/synapse#12390)) Updates to the Docker image --------------------------- - Bundle locked versions of dependencies into the Docker image. ([\#12385](matrix-org/synapse#12385), [\#12439](matrix-org/synapse#12439)) - Fix up healthcheck generation for workers docker image. ([\#12405](matrix-org/synapse#12405)) Improved Documentation ---------------------- - Clarify documentation for running SyTest against Synapse, including use of Postgres and worker mode. ([\#12271](matrix-org/synapse#12271)) - Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` methods when transactions are retried. ([\#12315](matrix-org/synapse#12315)) - Update dead links in `check-newsfragment.sh` to point to the correct documentation URL. ([\#12331](matrix-org/synapse#12331)) - Upgrade the version of `mdbook` in CI to 0.4.17. ([\#12339](matrix-org/synapse#12339)) - Updates to the Room DAG concepts development document to clarify that we mark events as outliers because we don't have any state for them. ([\#12345](matrix-org/synapse#12345)) - Update the link to Redis pub/sub documentation in the workers documentation. ([\#12369](matrix-org/synapse#12369)) - Remove documentation for converting a legacy structured logging configuration to the new format. ([\#12392](matrix-org/synapse#12392)) Deprecations and Removals ------------------------- - Remove the unused and unstable `/aggregations` endpoint which was removed from [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\#12293](matrix-org/synapse#12293)) Internal Changes ---------------- - Remove lingering unstable references to MSC2403 (knocking). ([\#12165](matrix-org/synapse#12165)) - Avoid trying to calculate the state at outlier events. ([\#12191](matrix-org/synapse#12191), [\#12316](matrix-org/synapse#12316), [\#12330](matrix-org/synapse#12330), [\#12332](matrix-org/synapse#12332), [\#12409](matrix-org/synapse#12409)) - Omit sending "offline" presence updates to application services after they are initially configured. ([\#12193](matrix-org/synapse#12193)) - Switch to using a sequence to generate AS transaction IDs. Contributed by Nick @ Beeper. If running synapse with a dedicated appservice worker, this MUST be stopped before upgrading the main process and database. ([\#12209](matrix-org/synapse#12209)) - Add missing type hints for storage. ([\#12267](matrix-org/synapse#12267)) - Add missing type definitions for scripts in docker folder. Contributed by Jorge Florian. ([\#12280](matrix-org/synapse#12280)) - Move [MSC2654](matrix-org/matrix-spec-proposals#2654) support behind an experimental configuration flag. ([\#12295](matrix-org/synapse#12295)) - Update docstrings to explain how to decipher live and historic pagination tokens. ([\#12317](matrix-org/synapse#12317)) - Add ground work for speeding up device list updates for users in large numbers of rooms. ([\#12321](matrix-org/synapse#12321)) - Fix typechecker problems exposed by signedjson 1.1.2. ([\#12326](matrix-org/synapse#12326)) - Remove the `tox` packaging job: it will be redundant once #11537 lands. ([\#12334](matrix-org/synapse#12334)) - Ignore `.envrc` for `direnv` users. ([\#12335](matrix-org/synapse#12335)) - Remove the (broadly unused, dev-only) dockerfile for pg tests. ([\#12336](matrix-org/synapse#12336)) - Remove redundant `get_success` calls in test code. ([\#12346](matrix-org/synapse#12346)) - Add type annotations for `tests/unittest.py`. ([\#12347](matrix-org/synapse#12347)) - Move single-use methods out of `TestCase`. ([\#12348](matrix-org/synapse#12348)) - Remove broken and unused development scripts. ([\#12349](matrix-org/synapse#12349), [\#12351](matrix-org/synapse#12351), [\#12355](matrix-org/synapse#12355)) - Convert `Linearizer` tests from `inlineCallbacks` to async. ([\#12353](matrix-org/synapse#12353)) - Update docstrings for `ReadWriteLock` tests. ([\#12354](matrix-org/synapse#12354)) - Refactor `Linearizer`, convert methods to async and use an async context manager. ([\#12357](matrix-org/synapse#12357)) - Fix a long-standing bug where `Linearizer`s could get stuck if a cancellation were to happen at the wrong time. ([\#12358](matrix-org/synapse#12358)) - Make `StreamToken.from_string` and `RoomStreamToken.parse` propagate cancellations instead of replacing them with `SynapseError`s. ([\#12366](matrix-org/synapse#12366)) - Add type hints to tests files. ([\#12371](matrix-org/synapse#12371)) - Allow specifying the Postgres database's port when running unit tests with Postgres. ([\#12376](matrix-org/synapse#12376)) - Remove temporary pin of signedjson<=1.1.1 that was added in Synapse 1.56.0. ([\#12379](matrix-org/synapse#12379)) - Add opentracing spans to calls to external cache. ([\#12380](matrix-org/synapse#12380)) - Lay groundwork for using `poetry` to manage Synapse's dependencies. ([\#12381](matrix-org/synapse#12381), [\#12407](matrix-org/synapse#12407), [\#12412](matrix-org/synapse#12412), [\#12418](matrix-org/synapse#12418)) - Make missing `importlib_metadata` dependency explicit. ([\#12384](matrix-org/synapse#12384), [\#12400](matrix-org/synapse#12400)) - Update type annotations for compatiblity with prometheus_client 0.14. ([\#12389](matrix-org/synapse#12389)) - Remove support for the unstable identifiers specified in [MSC3288](matrix-org/matrix-spec-proposals#3288). ([\#12398](matrix-org/synapse#12398)) - Add missing type hints to configuration classes. ([\#12402](matrix-org/synapse#12402)) - Add files used to build the Docker image used for complement testing into the Synapse repository. ([\#12404](matrix-org/synapse#12404)) - Do not include groups in the sync response when disabled. ([\#12408](matrix-org/synapse#12408)) - Improve type hints related to HTTP query parameters. ([\#12415](matrix-org/synapse#12415)) - Stop maintaining a list of lint targets. ([\#12420](matrix-org/synapse#12420)) - Make `synapse._scripts` pass type checks. ([\#12421](matrix-org/synapse#12421), [\#12422](matrix-org/synapse#12422)) - Add some type hints to datastore. ([\#12423](matrix-org/synapse#12423)) - Enable certificate checking during complement tests. ([\#12435](matrix-org/synapse#12435)) - Explicitly specify the `tls` extra for Twisted dependency. ([\#12444](matrix-org/synapse#12444))

Synapse 1.57.0 (2022-04-19) =========================== This version includes a [change](matrix-org#12209) to the way transaction IDs are managed for application services. If your deployment uses a dedicated worker for application service traffic, **it must be stopped** when the database is upgraded (which normally happens when the main process is upgraded), to ensure the change is made safely without any risk of reusing transaction IDs. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/v1.57.0rc1/docs/upgrade.md#upgrading-to-v1570) for more details. No significant changes since 1.57.0rc1. Synapse 1.57.0rc1 (2022-04-12) ============================== Features -------- - Send device list changes to application services as specified by [MSC3202](matrix-org/matrix-spec-proposals#3202), using unstable prefixes. The `msc3202_transaction_extensions` experimental homeserver config option must be enabled and `org.matrix.msc3202: true` must be present in the application service registration file for device list changes to be sent. The "left" field is currently always empty. ([\matrix-org#11881](matrix-org#11881)) - Optimise fetching large quantities of missing room state over federation. ([\matrix-org#12040](matrix-org#12040)) - Offload the `update_client_ip` background job from the main process to the background worker, when using Redis-based replication. ([\matrix-org#12251](matrix-org#12251)) - Move `update_client_ip` background job from the main process to the background worker. ([\matrix-org#12252](matrix-org#12252)) - Add a module callback to react to new 3PID (email address, phone number) associations. ([\matrix-org#12302](matrix-org#12302)) - Add a configuration option to remove a specific set of rooms from sync responses. ([\matrix-org#12310](matrix-org#12310)) - Add a module callback to react to account data changes. ([\matrix-org#12327](matrix-org#12327)) - Allow setting user admin status using the module API. Contributed by Famedly. ([\matrix-org#12341](matrix-org#12341)) - Reduce overhead of restarting synchrotrons. ([\matrix-org#12367](matrix-org#12367), [\matrix-org#12372](matrix-org#12372)) - Update `/messages` to use historic pagination tokens if no `from` query parameter is given. ([\matrix-org#12370](matrix-org#12370)) - Add a module API for reading and writing global account data. ([\matrix-org#12391](matrix-org#12391)) - Support the stable `v1` endpoint for `/relations`, per [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\matrix-org#12403](matrix-org#12403)) - Include bundled aggregations in search results ([MSC3666](matrix-org/matrix-spec-proposals#3666)). ([\matrix-org#12436](matrix-org#12436)) Bugfixes -------- - Fix a long-standing bug where updates to the server notices user profile (display name/avatar URL) in the configuration would not be applied to pre-existing rooms. Contributed by Jorge Florian. ([\matrix-org#12115](matrix-org#12115)) - Fix a long-standing bug where events from ignored users were still considered for bundled aggregations. ([\matrix-org#12235](matrix-org#12235), [\matrix-org#12338](matrix-org#12338)) - Fix non-member state events not resolving for historical events when used in [MSC2716](matrix-org/matrix-spec-proposals#2716) `/batch_send` `state_events_at_start`. ([\matrix-org#12329](matrix-org#12329)) - Fix a long-standing bug affecting URL previews that would generate a 500 response instead of a 403 if the previewed URL includes a port that isn't allowed by the relevant blacklist. ([\matrix-org#12333](matrix-org#12333)) - Default to `private` room visibility rather than `public` when a client does not specify one, according to spec. ([\matrix-org#12350](matrix-org#12350)) - Fix a spec compliance issue where requests to the `/publicRooms` federation API would specify `limit` as a string. ([\matrix-org#12364](matrix-org#12364), [\matrix-org#12410](matrix-org#12410)) - Fix a bug introduced in Synapse 1.49.0 which caused the `synapse_event_persisted_position` metric to have invalid values. ([\matrix-org#12390](matrix-org#12390)) Updates to the Docker image --------------------------- - Bundle locked versions of dependencies into the Docker image. ([\matrix-org#12385](matrix-org#12385), [\matrix-org#12439](matrix-org#12439)) - Fix up healthcheck generation for workers docker image. ([\matrix-org#12405](matrix-org#12405)) Improved Documentation ---------------------- - Clarify documentation for running SyTest against Synapse, including use of Postgres and worker mode. ([\matrix-org#12271](matrix-org#12271)) - Document the behaviour of `LoggingTransaction.call_after` and `LoggingTransaction.call_on_exception` methods when transactions are retried. ([\matrix-org#12315](matrix-org#12315)) - Update dead links in `check-newsfragment.sh` to point to the correct documentation URL. ([\matrix-org#12331](matrix-org#12331)) - Upgrade the version of `mdbook` in CI to 0.4.17. ([\matrix-org#12339](matrix-org#12339)) - Updates to the Room DAG concepts development document to clarify that we mark events as outliers because we don't have any state for them. ([\matrix-org#12345](matrix-org#12345)) - Update the link to Redis pub/sub documentation in the workers documentation. ([\matrix-org#12369](matrix-org#12369)) - Remove documentation for converting a legacy structured logging configuration to the new format. ([\matrix-org#12392](matrix-org#12392)) Deprecations and Removals ------------------------- - Remove the unused and unstable `/aggregations` endpoint which was removed from [MSC2675](matrix-org/matrix-spec-proposals#2675). ([\matrix-org#12293](matrix-org#12293)) Internal Changes ---------------- - Remove lingering unstable references to MSC2403 (knocking). ([\matrix-org#12165](matrix-org#12165)) - Avoid trying to calculate the state at outlier events. ([\matrix-org#12191](matrix-org#12191), [\matrix-org#12316](matrix-org#12316), [\matrix-org#12330](matrix-org#12330), [\matrix-org#12332](matrix-org#12332), [\matrix-org#12409](matrix-org#12409)) - Omit sending "offline" presence updates to application services after they are initially configured. ([\matrix-org#12193](matrix-org#12193)) - Switch to using a sequence to generate AS transaction IDs. Contributed by Nick @ Beeper. If running synapse with a dedicated appservice worker, this MUST be stopped before upgrading the main process and database. ([\matrix-org#12209](matrix-org#12209)) - Add missing type hints for storage. ([\matrix-org#12267](matrix-org#12267)) - Add missing type definitions for scripts in docker folder. Contributed by Jorge Florian. ([\matrix-org#12280](matrix-org#12280)) - Move [MSC2654](matrix-org/matrix-spec-proposals#2654) support behind an experimental configuration flag. ([\matrix-org#12295](matrix-org#12295)) - Update docstrings to explain how to decipher live and historic pagination tokens. ([\matrix-org#12317](matrix-org#12317)) - Add ground work for speeding up device list updates for users in large numbers of rooms. ([\matrix-org#12321](matrix-org#12321)) - Fix typechecker problems exposed by signedjson 1.1.2. ([\matrix-org#12326](matrix-org#12326)) - Remove the `tox` packaging job: it will be redundant once matrix-org#11537 lands. ([\matrix-org#12334](matrix-org#12334)) - Ignore `.envrc` for `direnv` users. ([\matrix-org#12335](matrix-org#12335)) - Remove the (broadly unused, dev-only) dockerfile for pg tests. ([\matrix-org#12336](matrix-org#12336)) - Remove redundant `get_success` calls in test code. ([\matrix-org#12346](matrix-org#12346)) - Add type annotations for `tests/unittest.py`. ([\matrix-org#12347](matrix-org#12347)) - Move single-use methods out of `TestCase`. ([\matrix-org#12348](matrix-org#12348)) - Remove broken and unused development scripts. ([\matrix-org#12349](matrix-org#12349), [\matrix-org#12351](matrix-org#12351), [\matrix-org#12355](matrix-org#12355)) - Convert `Linearizer` tests from `inlineCallbacks` to async. ([\matrix-org#12353](matrix-org#12353)) - Update docstrings for `ReadWriteLock` tests. ([\matrix-org#12354](matrix-org#12354)) - Refactor `Linearizer`, convert methods to async and use an async context manager. ([\matrix-org#12357](matrix-org#12357)) - Fix a long-standing bug where `Linearizer`s could get stuck if a cancellation were to happen at the wrong time. ([\matrix-org#12358](matrix-org#12358)) - Make `StreamToken.from_string` and `RoomStreamToken.parse` propagate cancellations instead of replacing them with `SynapseError`s. ([\matrix-org#12366](matrix-org#12366)) - Add type hints to tests files. ([\matrix-org#12371](matrix-org#12371)) - Allow specifying the Postgres database's port when running unit tests with Postgres. ([\matrix-org#12376](matrix-org#12376)) - Remove temporary pin of signedjson<=1.1.1 that was added in Synapse 1.56.0. ([\matrix-org#12379](matrix-org#12379)) - Add opentracing spans to calls to external cache. ([\matrix-org#12380](matrix-org#12380)) - Lay groundwork for using `poetry` to manage Synapse's dependencies. ([\matrix-org#12381](matrix-org#12381), [\matrix-org#12407](matrix-org#12407), [\matrix-org#12412](matrix-org#12412), [\matrix-org#12418](matrix-org#12418)) - Make missing `importlib_metadata` dependency explicit. ([\matrix-org#12384](matrix-org#12384), [\matrix-org#12400](matrix-org#12400)) - Update type annotations for compatiblity with prometheus_client 0.14. ([\matrix-org#12389](matrix-org#12389)) - Remove support for the unstable identifiers specified in [MSC3288](matrix-org/matrix-spec-proposals#3288). ([\matrix-org#12398](matrix-org#12398)) - Add missing type hints to configuration classes. ([\matrix-org#12402](matrix-org#12402)) - Add files used to build the Docker image used for complement testing into the Synapse repository. ([\matrix-org#12404](matrix-org#12404)) - Do not include groups in the sync response when disabled. ([\matrix-org#12408](matrix-org#12408)) - Improve type hints related to HTTP query parameters. ([\matrix-org#12415](matrix-org#12415)) - Stop maintaining a list of lint targets. ([\matrix-org#12420](matrix-org#12420)) - Make `synapse._scripts` pass type checks. ([\matrix-org#12421](matrix-org#12421), [\matrix-org#12422](matrix-org#12422)) - Add some type hints to datastore. ([\matrix-org#12423](matrix-org#12423)) - Enable certificate checking during complement tests. ([\matrix-org#12435](matrix-org#12435)) - Explicitly specify the `tls` extra for Twisted dependency. ([\matrix-org#12444](matrix-org#12444))

erikjohnston marked this pull request as ready for review March 29, 2022 10:41

erikjohnston requested a review from a team as a code owner March 29, 2022 10:41

erikjohnston added 8 commits March 29, 2022 13:12

Operate on room IDs

b5a9c6b

Don't copy the hosts list

6fa639e

Write to a new device_lists_changes_in_room table.

ecf98b9

Convert to doing everything in a transaction

c5dd83f

Track if we've calculated remote hosts

bb44214

Currently we update the `device_lists_outbound_pokes` at the same time as `device_lists_changes_in_room`, but in future we'll do them at different times.

We actually don't want stream_id to be unique

98fceb3

Handle room pokes that haven't been converted to outbound pokes

eda0e64

Deduplicate outbound pokes

c7790ab

erikjohnston force-pushed the erikj/device_list_perf branch from 78e2c0a to 7707cab Compare March 29, 2022 12:12

erikjohnston added 3 commits March 29, 2022 13:13

Add a config option that allows using new code path

6e9b31a

Add tests for new code path

3b2ab93

Newsfile

f8af30f

erikjohnston force-pushed the erikj/device_list_perf branch from 7707cab to f8af30f Compare March 29, 2022 12:13

Fix tests

7266580

erikjohnston force-pushed the erikj/device_list_perf branch from 0af131a to 7266580 Compare March 29, 2022 13:16

Merge remote-tracking branch 'origin/develop' into erikj/device_list_…

f24b70b

…perf

richvdh self-assigned this Apr 1, 2022

richvdh suggested changes Apr 1, 2022

View reviewed changes

richvdh removed their assignment Apr 1, 2022

erikjohnston and others added 9 commits April 4, 2022 09:50

Update synapse/storage/schema/main/delta/69/01device_list_oubound_by_…

8bd8ee2

…room.sql Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Update synapse/storage/schema/main/delta/69/01device_list_oubound_by_…

56f0913

…room.sql Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Encode opentracing context just once.

90d41a0

Rename var

c470a12

Remove if not room_ids check.

d030062

Add unique index

ad5d46b

Note lack of foreign key constraint

d5031b0

Add comment about stream_id duplicates

28dacc8

Update synapse_port_db

f48527f

erikjohnston added 3 commits April 4, 2022 10:24

Inequality the wrong way round

bd45f19

Add note about 'num_stream_ids'

dee8f55

Merge remote-tracking branch 'origin/develop' into erikj/device_list_…

3574541

…perf

erikjohnston requested a review from richvdh April 4, 2022 10:03

erikjohnston added 3 commits April 4, 2022 12:27

Use different stream IDs for device_list_outbound_pokes

cf04f1a

Correctly order device list stream updates

89e10d7

Wake up replication after adding otubound pokes

e54d2d4

richvdh reviewed Apr 4, 2022

View reviewed changes

richvdh suggested changes Apr 4, 2022

View reviewed changes

erikjohnston and others added 2 commits April 4, 2022 14:42

Apply suggestions from code review

7d79dee

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Remove get_users_who_share_room_with_user stub in test

b61c5c7

richvdh approved these changes Apr 4, 2022

View reviewed changes

erikjohnston merged commit 5c9e39e into develop Apr 4, 2022

erikjohnston deleted the erikj/device_list_perf branch April 4, 2022 14:25

erikjohnston added a commit that referenced this pull request Apr 4, 2022

Process device list updates asynchronously

658b648

Carries on from #12321.

erikjohnston mentioned this pull request Apr 4, 2022

Process device list updates asynchronously #12365

Merged

erikjohnston added a commit that referenced this pull request Apr 5, 2022

Process device list updates asynchronously

f2b3d45

Carries on from #12321.

erikjohnston mentioned this pull request Apr 6, 2022

Use new device_lists_changes_in_room table for fetching device lists in /sync #12388

Closed

DMRobertson reviewed Apr 21, 2022

View reviewed changes

This was referenced Apr 21, 2022

Add some type hints to datastore #12485

Merged

log_kv within _handle_new_device_update_async is not associated with an opentracing span, causing sentry logspam and large memory usage #12552

Closed

herb mentioned this pull request Jun 7, 2022

db on-disk space usage increased v1.58+ #12975

Closed

erikjohnston mentioned this pull request Jun 14, 2022

Clear out device_lists_changes_in_room table periodically. #13043

Open

clokep mentioned this pull request Jun 14, 2022

Type annotations in synapse.databases.main.devices #13025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track device list updates per room. #12321

Track device list updates per room. #12321

erikjohnston commented Mar 29, 2022 •

edited

richvdh Apr 1, 2022

erikjohnston Apr 4, 2022

richvdh Apr 1, 2022

erikjohnston Apr 4, 2022

richvdh Apr 4, 2022

richvdh Apr 1, 2022

erikjohnston Apr 4, 2022

richvdh Apr 4, 2022

erikjohnston Apr 4, 2022

erikjohnston commented Apr 4, 2022

richvdh Apr 4, 2022

richvdh Apr 4, 2022

richvdh Apr 4, 2022

richvdh Apr 4, 2022

erikjohnston Apr 4, 2022

richvdh Apr 4, 2022

richvdh left a comment

DMRobertson Apr 21, 2022

Track device list updates per room. #12321

Track device list updates per room. #12321

Conversation

erikjohnston commented Mar 29, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Apr 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Mar 29, 2022 •

edited