Use a chain cover index to efficiently calculate auth chain difference #8868

erikjohnston · 2020-12-03T11:56:30Z

See the docs.

There's two main aspects to this PR:

The ewn auth chain difference algorithm; and
The creation of the index when we persist events

Hopefully fixes: #8622

synapse/storage/database.py

synapse/storage/databases/main/event_federation.py

erikjohnston · 2020-12-10T13:44:48Z

Another issue:

Really old rooms that the server left a long time ago won't have a row in the rooms table, so we add one with the has_auth_chain_index set to true, even though we already have a bunch of events

@erikjohnston

Dismissing the review as @erikjohnston found a few bugs while testing.

erikjohnston · 2021-01-04T17:25:14Z

#9016 fixes that rejected events aren't added to state_events (or previously to the event_auth) tables

We may not have an entry in the rooms table for old rooms that the server is no longer in. If so then we should set the `has_auth_chain_index` to false.

…alog

erikjohnston · 2021-01-06T13:35:00Z

I think I've fixed the issues I discovered last time. Will try putting on jki.re again.

clokep · 2021-01-06T14:06:22Z

synapse/storage/databases/main/event_federation.py

+            logger.info(
+                "Unexpectedly found that events don't have chain IDs in room %s: %s",
+                room_id,
+                events_missing_chain_info,
+            )


Is there any way to recover from this? Will it happen automatically?

It should mostly happen automatically, unless there are events we don't have the full auth chain for (I have a couple of those on jki.re somehow). There's not really a way to recover from that failure mode, so I'm tempted to just leave it as is and revisit if it turns out that it happens a lot.

erikjohnston · 2021-01-11T14:17:10Z

I think I've fixed the issues I discovered last time. Will try putting on jki.re again.

It's been running on jki.re since Thursday without any issues \o/

clokep · 2021-01-11T14:25:54Z

I think I've fixed the issues I discovered last time. Will try putting on jki.re again.

It's been running on jki.re since Thursday without any issues \o/

Any thoughts on if perf is improved?

erikjohnston · 2021-01-11T14:29:53Z

I think I've fixed the issues I discovered last time. Will try putting on jki.re again.

It's been running on jki.re since Thursday without any issues \o/

Any thoughts on if perf is improved?

Not really, but then jki.re normall just sits there at ~10% CPU so 🤷

synapse/storage/databases/main/room.py

clokep · 2021-01-11T14:53:19Z

synapse/storage/databases/main/event_federation.py

@@ -207,6 +219,18 @@ def _get_auth_chain_difference_using_cover_index_txn(
                seen_chains.add(chain_id)
                chain_to_event.setdefault(chain_id, {})[sequence_number] = event_id

+        # Check that we actually have a chain ID for all the events.
+        events_missing_chain_info = initial_events.difference(chain_info)


I was going to suggest using initial_events >= chain_info as we don't care about the difference, just if there is one. But chain_info is a dict and it would probably be more expensive to convert it to a set first. 🤷

tests/storage/test_event_federation.py

clokep · 2021-01-11T14:56:36Z

tests/storage/test_event_federation.py

@@ -259,6 +259,169 @@ def insert_event(txn):
        )
        self.assertSetEqual(difference, set())

+    def test_auth_difference_partial_cover(self):
+        """Test that we correctly handle rooms where not all events have a chain
+        cover calculated. This can happen due to a downgrade/upgrade.


If we move these to schema 59 is the downgrade really a potential cause of this? It is likely worth keeping the code to handle this case regardless.

Yeah, I've updated it

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

…alog

clokep · 2021-01-11T15:22:09Z

synapse/storage/databases/main/room.py

        # It's possible that we already have events for the room in our DB
        # without a corresponding room entry. If we do then we don't want to
        # mark the room as having an auth chain cover index.


We probably don't need this comment anymore (and the one near line 1267) as it is in the has_auth_chain_index method now.

I think its a bit unclear why we're bothering to call the function otherwise?

I don't feel strongly. I'm OK with leaving it.

@chris-ruecker

Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous verious is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](#9015), [\#9017](#9017), [\#9036](#9036), [\#9067](#9067), [\#9081](#9081), [\#9082](#9082), [\#9105](#9105), [\#9107](#9107), [\#9109](#9109), [\#9110](#9110), [\#9127](#9127), [\#9153](#9153), [\#9154](#9154), [\#9177](#9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](#9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](#9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](#9024)) - Implement [MSC2176](matrix-org/matrix-spec-proposals#2176) in an experimental room version. ([\#8984](#8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](#9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](#8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](#8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](#9042), [\#9043](#9043), [\#9044](#9044), [\#9130](#9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](#9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](#9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](#9104), [\#9166](#9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](#9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](#9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](#9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](#9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](#9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](#9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](#9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](#9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](#9114), [\#9116](#9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](#9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](#9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](#9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](#9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](#9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](#8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](#9035)) - Correct a typo in `INSTALL.md`. ([\#9040](#9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](#9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](#9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](#9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](#8868), [\#9029](#9029), [\#9115](#9115), [\#9118](#9118), [\#9124](#9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](#8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](#9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](#9018)) - Remove an unused column from `access_tokens` table. ([\#9025](#9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](#9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](#9031)) - Allow bumping schema version when using split out state database. ([\#9033](#9033)) - Configure the linters to run on a consistent set of files. ([\#9038](#9038)) - Various cleanups to device inbox store. ([\#9041](#9041)) - Drop unused database tables. ([\#9055](#9055)) - Remove unused `SynapseService` class. ([\#9058](#9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](#9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](#9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](#9080)) - Add type hints to media repository. ([\#9093](#9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](#9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](#9106)) - Improve `UsernamePickerTestCase`. ([\#9112](#9112)) - Remove dependency on `distutils`. ([\#9125](#9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](#9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](#9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](#9157))

@chris-ruecker

Synapse 1.26.0 (2021-01-27) =========================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. No significant changes since 1.26.0rc2. Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](#9193), [\#9195](#9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](#9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](#9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](#9204)) Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](#9015), [\#9017](#9017), [\#9036](#9036), [\#9067](#9067), [\#9081](#9081), [\#9082](#9082), [\#9105](#9105), [\#9107](#9107), [\#9109](#9109), [\#9110](#9110), [\#9127](#9127), [\#9153](#9153), [\#9154](#9154), [\#9177](#9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](#9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](#9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](#9024)) - Implement [MSC2176](matrix-org/matrix-spec-proposals#2176) in an experimental room version. ([\#8984](#8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](#9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](#8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](#8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](#9042), [\#9043](#9043), [\#9044](#9044), [\#9130](#9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](#9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](#9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](#9104), [\#9166](#9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](#9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](#9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](#9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](#9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](#9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](#9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](#9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](#9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](#9114), [\#9116](#9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](#9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](#9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](#9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](#9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](#9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](#8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](#9035)) - Correct a typo in `INSTALL.md`. ([\#9040](#9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](#9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](#9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](#9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](#8868), [\#9029](#9029), [\#9115](#9115), [\#9118](#9118), [\#9124](#9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](#8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](#9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](#9018)) - Remove an unused column from `access_tokens` table. ([\#9025](#9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](#9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](#9031)) - Allow bumping schema version when using split out state database. ([\#9033](#9033)) - Configure the linters to run on a consistent set of files. ([\#9038](#9038)) - Various cleanups to device inbox store. ([\#9041](#9041)) - Drop unused database tables. ([\#9055](#9055)) - Remove unused `SynapseService` class. ([\#9058](#9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](#9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](#9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](#9080)) - Add type hints to media repository. ([\#9093](#9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](#9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](#9106)) - Improve `UsernamePickerTestCase`. ([\#9112](#9112)) - Remove dependency on `distutils`. ([\#9125](#9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](#9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](#9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](#9157))

@chris-ruecker

Synapse 1.26.0 (2021-01-27) =========================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. No significant changes since 1.26.0rc2. Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](matrix-org/synapse#9193), [\#9195](matrix-org/synapse#9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](matrix-org/synapse#9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](matrix-org/synapse#9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](matrix-org/synapse#9204)) Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](matrix-org/synapse#9015), [\#9017](matrix-org/synapse#9017), [\#9036](matrix-org/synapse#9036), [\#9067](matrix-org/synapse#9067), [\#9081](matrix-org/synapse#9081), [\#9082](matrix-org/synapse#9082), [\#9105](matrix-org/synapse#9105), [\#9107](matrix-org/synapse#9107), [\#9109](matrix-org/synapse#9109), [\#9110](matrix-org/synapse#9110), [\#9127](matrix-org/synapse#9127), [\#9153](matrix-org/synapse#9153), [\#9154](matrix-org/synapse#9154), [\#9177](matrix-org/synapse#9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](matrix-org/synapse#9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](matrix-org/synapse#9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](matrix-org/synapse#9024)) - Implement [MSC2176](matrix-org/matrix-spec-proposals#2176) in an experimental room version. ([\#8984](matrix-org/synapse#8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](matrix-org/synapse#9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](matrix-org/synapse#8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](matrix-org/synapse#8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](matrix-org/synapse#9042), [\#9043](matrix-org/synapse#9043), [\#9044](matrix-org/synapse#9044), [\#9130](matrix-org/synapse#9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](matrix-org/synapse#9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](matrix-org/synapse#9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](matrix-org/synapse#9104), [\#9166](matrix-org/synapse#9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](matrix-org/synapse#9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](matrix-org/synapse#9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](matrix-org/synapse#9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](matrix-org/synapse#9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](matrix-org/synapse#9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](matrix-org/synapse#9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](matrix-org/synapse#9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](matrix-org/synapse#9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](matrix-org/synapse#9114), [\#9116](matrix-org/synapse#9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](matrix-org/synapse#9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](matrix-org/synapse#9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](matrix-org/synapse#9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](matrix-org/synapse#9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](matrix-org/synapse#9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](matrix-org/synapse#8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](matrix-org/synapse#9035)) - Correct a typo in `INSTALL.md`. ([\#9040](matrix-org/synapse#9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](matrix-org/synapse#9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](matrix-org/synapse#9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](matrix-org/synapse#9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](matrix-org/synapse#8868), [\#9029](matrix-org/synapse#9029), [\#9115](matrix-org/synapse#9115), [\#9118](matrix-org/synapse#9118), [\#9124](matrix-org/synapse#9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](matrix-org/synapse#8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](matrix-org/synapse#9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](matrix-org/synapse#9018)) - Remove an unused column from `access_tokens` table. ([\#9025](matrix-org/synapse#9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](matrix-org/synapse#9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](matrix-org/synapse#9031)) - Allow bumping schema version when using split out state database. ([\#9033](matrix-org/synapse#9033)) - Configure the linters to run on a consistent set of files. ([\#9038](matrix-org/synapse#9038)) - Various cleanups to device inbox store. ([\#9041](matrix-org/synapse#9041)) - Drop unused database tables. ([\#9055](matrix-org/synapse#9055)) - Remove unused `SynapseService` class. ([\#9058](matrix-org/synapse#9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](matrix-org/synapse#9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](matrix-org/synapse#9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](matrix-org/synapse#9080)) - Add type hints to media repository. ([\#9093](matrix-org/synapse#9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](matrix-org/synapse#9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](matrix-org/synapse#9106)) - Improve `UsernamePickerTestCase`. ([\#9112](matrix-org/synapse#9112)) - Remove dependency on `distutils`. ([\#9125](matrix-org/synapse#9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](matrix-org/synapse#9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](matrix-org/synapse#9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](matrix-org/synapse#9157))

erikjohnston force-pushed the erikj/auth_chains_alog branch from ed86ee9 to c1080b7 Compare December 4, 2020 14:48

clokep mentioned this pull request Dec 4, 2020

Pass room_id to get_auth_chain_difference #8879

Merged

erikjohnston added 12 commits December 4, 2020 15:53

Change alg

49e888d

Calculate chain ID/seq no on event insertion

8c760ff

Add some docs about the chain cover

85348e1

Handle old rooms

02d1198

Fix schema for sqlite

61ab47e

Fix up _get_auth_chain_difference_using_chains_txn

6141825

Newsfile

c7e2ce5

Add type

66e779d

Fixup

cf2243f

Fix take1

bd30c9e

Fixup

55f03b9

More fixups

3e98fb7

erikjohnston force-pushed the erikj/auth_chains_alog branch from 3166431 to 3e98fb7 Compare December 4, 2020 15:54

erikjohnston added 4 commits December 4, 2020 15:54

Newsfile

9087033

Test both new and old methods

21b3ef0

Note

fdaf4da

isort

7f5ac13

erikjohnston requested a review from a team December 4, 2020 16:38

erikjohnston added 5 commits December 7, 2020 14:53

Don't add links where start and end chain are the same

afb7f80

Have exists_path_from handle same chain case correctly

dec1f74

Add some tests

9279940

Fix unit tests on postgres

6a74e21

Add missing 'auth'

654eff1

clokep reviewed Dec 7, 2020

View reviewed changes

synapse/storage/database.py Show resolved Hide resolved

clokep reviewed Dec 7, 2020

View reviewed changes

synapse/storage/databases/main/event_federation.py Outdated Show resolved Hide resolved

clokep reviewed Dec 7, 2020

View reviewed changes

synapse/storage/databases/main/event_federation.py Show resolved Hide resolved

erikjohnston added 2 commits December 8, 2020 10:43

Fixup typing for execute_values

dbecefd

Rename _get_auth_chain_difference_using_chains_txn and add comment

123b431

erikjohnston mentioned this pull request Jan 4, 2021

Ensure rejected events get added to some metadata tables #9016

Merged

erikjohnston added 4 commits January 5, 2021 15:26

Handle rooms the server used to be in correctly.

c9422b6

We may not have an entry in the rooms table for old rooms that the server is no longer in. If so then we should set the `has_auth_chain_index` to false.

Handle case where we don't have chain info for an event

c8758af

Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…

b2ac553

…alog

Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…

d96264d

…alog

clokep reviewed Jan 6, 2021

View reviewed changes

erikjohnston mentioned this pull request Jan 6, 2021

Add background update for add chain cover index #9029

Merged

clokep suggested changes Jan 11, 2021

View reviewed changes

erikjohnston and others added 5 commits January 11, 2021 15:00

Typo

d64f5f8

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Split out a has_auth_chain_index

6071eff

Update docstring

368d3b8

Move to schema 59

bea2c47

Merge remote-tracking branch 'origin/develop' into erikj/auth_chains_…

03dd636

…alog

erikjohnston force-pushed the erikj/auth_chains_alog branch from 768ecf0 to 03dd636 Compare January 11, 2021 15:17

erikjohnston requested a review from clokep January 11, 2021 15:18

clokep approved these changes Jan 11, 2021

View reviewed changes

Fix tests after merge from develop

8c1e32c

erikjohnston merged commit 1315a2e into develop Jan 11, 2021

erikjohnston deleted the erikj/auth_chains_alog branch January 11, 2021 16:09

krithin mentioned this pull request Feb 11, 2021

Postgres port script breaks the database after synapse 1.26, doesn't allow creating new DMs. #9382

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use a chain cover index to efficiently calculate auth chain difference #8868

Use a chain cover index to efficiently calculate auth chain difference #8868

erikjohnston commented Dec 3, 2020 •

edited

erikjohnston commented Dec 10, 2020

erikjohnston commented Jan 4, 2021

erikjohnston commented Jan 6, 2021

clokep Jan 6, 2021

erikjohnston Jan 11, 2021

erikjohnston commented Jan 11, 2021

clokep commented Jan 11, 2021

erikjohnston commented Jan 11, 2021

clokep Jan 11, 2021

erikjohnston Jan 11, 2021

clokep Jan 11, 2021

erikjohnston Jan 11, 2021

clokep Jan 11, 2021

erikjohnston Jan 11, 2021

clokep Jan 11, 2021

Use a chain cover index to efficiently calculate auth chain difference #8868

Use a chain cover index to efficiently calculate auth chain difference #8868

Conversation

erikjohnston commented Dec 3, 2020 • edited

erikjohnston commented Dec 10, 2020

erikjohnston commented Jan 4, 2021

erikjohnston commented Jan 6, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Jan 11, 2021

clokep commented Jan 11, 2021

erikjohnston commented Jan 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Dec 3, 2020 •

edited