Limit and filter the number of backfill points to get from the database #13879

MadLittleMods · 2022-09-22T20:09:16Z

Limit and filter the number of backfill points to get from the database. There is no need to grab thousands of backfill points when we only need 5 to make the /backfill request with. We need to grab a few extra in case the first few aren't visible in the history.

Previously, we grabbed thousands of backfill points from the database, then sorted and filtered them in the app. Fetching the 4.6k backfill points for #matrix:matrix.org from the database takes ~50ms - ~570ms so it's not like this saves a lot of time 🤷. But it might save us more time now that get_backfill_points_in_room/get_insertion_event_backward_extremities_in_room are more complicated after #13635

This PR moves the filtering and limiting to the SQL query so we just have less data to work with in the first place.

Part of #13356

This change was originally included in #13635 (comment) but was split off.

Dev notes

Can we do anything smarter with two already sorted lists?

See #13635 (comment) and #13879 (comment)

Todo

Add limit to get_backfill_points_in_room and get_insertion_event_backward_extremities_in_room

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

…-backfill-an-event

See #13589 (comment)

See #13635 (comment)

…nd-limits Conflicts: synapse/handlers/federation.py synapse/storage/databases/main/event_federation.py tests/storage/test_event_federation.py

synapse/handlers/federation.py

synapse/storage/databases/main/event_federation.py

synapse/handlers/federation.py

richvdh · 2022-09-27T17:55:13Z

synapse/handlers/federation.py

-        if not backwards_extremities and not insertion_events_to_be_backfilled:
-            logger.debug("Not backfilling as no extremeties found.")
-            return False
-
        # we now have a list of potential places to backpaginate from. We prefer to
        # start with the most recent (ie, max depth), so let's sort the list.
        sorted_backfill_points: List[_BackfillPoint] = sorted(


we could maybe use heapq.merge if both these lists are sorted?

@squahtx mentioned this in #13635 (comment) when I asked about any potential two already sorted lists optimizations. Do we want to do this though?

In this PR, we're also adding the limit so we're only sorting 100 events max, I am leaning towards keeping it simple or at least more familiar to the rest of the Synapse code 🤷 - No benchmarks done though and I don't have a reference for how common/known heapq.merge is besides the two usages in Synapse already.

synapse/handlers/federation.py

synapse/storage/databases/main/event_federation.py

See #13879 (comment)

…nd-limits

See #13879 (comment)

synapse/storage/databases/main/event_federation.py

richvdh · 2022-09-28T14:53:28Z

synapse/handlers/federation.py

-        if not backwards_extremities and not insertion_events_to_be_backfilled:
-            logger.debug("Not backfilling as no extremeties found.")
-            return False
-
        # we now have a list of potential places to backpaginate from. We prefer to
        # start with the most recent (ie, max depth), so let's sort the list.
        sorted_backfill_points: List[_BackfillPoint] = sorted(


richvdh

lgtm!

See #13879 (comment)

MadLittleMods · 2022-09-28T20:26:43Z

Thanks for the review @richvdh 🦔

@lukasdenk

Synapse 1.69.0rc1 (2022-10-04) ============================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\#11866](#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\#13556](#13556)) - Exponentially backoff from backfilling the same event over and over. ([\#13635](#13635), [\#13936](#13936)) - Add cache invalidation across workers to module API. ([\#13667](#13667), [\#13947](#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\#13722](#13722), [\#13868](#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\#13782](#13782), [\#13893](#13893), [\#13932](#13932), [\#13937](#13937), [\#13939](#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\#13799](#13799), [\#13831](#13831), [\#13860](#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\#13815](#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\#13832](#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\#13892](#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#13920](#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\#13957](#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\#13719](#13719), [\#14014](#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\#13830](#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\#13840](#13840)) - Fix access token leak to logs from proxy agent. ([\#13855](#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\#13863](#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\#13872](#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\#13904](#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\#13909](#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\#13922](#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\#13952](#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\#13956](#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\#13972](#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\#14009](#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\#14025](#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\#13885](#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\#13934](#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\#13772](#13772)) - Update URL for the NixOS module for Synapse. ([\#13818](#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\#13836](#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\#13870](#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\#13911](#13911), [\#13930](#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\#13915](#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\#13928](#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\#13931](#13931)) - Add some cross references to worker documentation. ([\#13974](#13974)) - Linkify urls in config documentation. ([\#14003](#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\#13843](#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\#14024](#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\#13487](#13487), [\#13800](#13800)) - Port push rules to using Rust. ([\#13768](#13768), [\#13838](#13838), [\#13889](#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\#13787](#13787)) - Update the script which makes full schema dumps. ([\#13792](#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\#13796](#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\#13809](#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\#13823](#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\#13839](#13839)) - Fix the release script not publishing binary wheels. ([\#13850](#13850)) - Raise issue if complement fails with latest deps. ([\#13859](#13859)) - Correct the comments in the complement dockerfile. ([\#13867](#13867)) - Create a new snapshot of the database schema. ([\#13873](#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\#13874](#13874), [\#14013](#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\#13876](#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\#13879](#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\#13888](#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\#13890](#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\#13905](#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\#13913](#13913)) - Complement image: propagate SIGTERM to all workers. ([\#13914](#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\#13924](#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\#13925](#13925), [\#13993](#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\#13960](#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\#13966](#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\#13969](#13969)) - Speed up calculating push actions in large rooms. ([\#13973](#13973), [\#13992](#13992)) - Enable update notifications from Github's dependabot. ([\#13976](#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\#13998](#13998), [\#14011](#14011), [\#14017](#14017), [\#14021](#14021), [\#14027](#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\#14012](#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\#14020](#14020)) - Bump versions of GitHub actions. ([\#13978](#13978), [\#13979](#13979), [\#13980](#13980), [\#13982](#13982), [\#14015](#14015), [\#14019](#14019), [\#14022](#14022), [\#14023](#14023))

@lukasdenk

NOTE: this is absolutely *not* safe for Beeper usage as-is. I have merged all of the Python code in but all our customizations to the base rules and push rule evaluator are not yet present in the new Rust module. This will fail tests as-is and future commits will re-apply our changes in Rust. Synapse 1.69.0 (2022-10-17) =========================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. No significant changes since 1.69.0rc4. Synapse 1.69.0rc4 (2022-10-14) ============================== Bugfixes -------- - Fix poor performance of the `event_push_backfill_thread_id` background update, which was introduced in Synapse 1.68.0rc1. ([\matrix-org#14172](matrix-org#14172), [\matrix-org#14181](matrix-org#14181)) Updates to the Docker image --------------------------- - Fix docker build OOMing in CI for arm64 builds. ([\matrix-org#14173](matrix-org#14173)) Synapse 1.69.0rc3 (2022-10-12) ============================== Bugfixes -------- - Fix an issue with Docker images causing the Rust dependencies to not be pinned correctly. Introduced in v1.68.0 ([\matrix-org#14129](matrix-org#14129)) - Fix a bug introduced in Synapse 1.69.0rc1 which would cause registration replication requests to fail if the worker sending the request is not running Synapse 1.69. ([\matrix-org#14135](matrix-org#14135)) - Fix error in background update when rotating existing notifications. Introduced in v1.69.0rc2. ([\matrix-org#14138](matrix-org#14138)) Internal Changes ---------------- - Rename the `url_preview` extra to `url-preview`, for compatability with poetry-core 1.3.0 and [PEP 685](https://peps.python.org/pep-0685/). From-source installations using this extra will need to install using the new name. ([\matrix-org#14085](matrix-org#14085)) Synapse 1.69.0rc2 (2022-10-06) ============================== Deprecations and Removals ------------------------- - Deprecate the `generate_short_term_login_token` method in favor of an async `create_login_token` method in the Module API. ([\matrix-org#13842](matrix-org#13842)) Internal Changes ---------------- - Ensure Synapse v1.69 works with upcoming database changes in v1.70. ([\matrix-org#14045](matrix-org#14045)) - Fix a bug introduced in Synapse v1.68.0 where messages could not be sent in rooms with non-integer `notifications` power level. ([\matrix-org#14073](matrix-org#14073)) - Temporarily pin build-system requirements to workaround an incompatibility with poetry-core 1.3.0. This will be reverted before the v1.69.0 release proper, see [\matrix-org#14079](matrix-org#14079). ([\matrix-org#14080](matrix-org#14080)) Synapse 1.69.0rc1 (2022-10-04) ============================== Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\matrix-org#11866](matrix-org#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\matrix-org#13556](matrix-org#13556)) - Exponentially backoff from backfilling the same event over and over. ([\matrix-org#13635](matrix-org#13635), [\matrix-org#13936](matrix-org#13936)) - Add cache invalidation across workers to module API. ([\matrix-org#13667](matrix-org#13667), [\matrix-org#13947](matrix-org#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\matrix-org#13722](matrix-org#13722), [\matrix-org#13868](matrix-org#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\matrix-org#13782](matrix-org#13782), [\matrix-org#13893](matrix-org#13893), [\matrix-org#13932](matrix-org#13932), [\matrix-org#13937](matrix-org#13937), [\matrix-org#13939](matrix-org#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\matrix-org#13799](matrix-org#13799), [\matrix-org#13831](matrix-org#13831), [\matrix-org#13860](matrix-org#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\matrix-org#13815](matrix-org#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\matrix-org#13832](matrix-org#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\matrix-org#13892](matrix-org#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\matrix-org#13920](matrix-org#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\matrix-org#13957](matrix-org#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\matrix-org#13719](matrix-org#13719), [\matrix-org#14014](matrix-org#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\matrix-org#13830](matrix-org#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\matrix-org#13840](matrix-org#13840)) - Fix access token leak to logs from proxy agent. ([\matrix-org#13855](matrix-org#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\matrix-org#13863](matrix-org#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\matrix-org#13872](matrix-org#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\matrix-org#13904](matrix-org#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\matrix-org#13909](matrix-org#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\matrix-org#13922](matrix-org#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\matrix-org#13952](matrix-org#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\matrix-org#13956](matrix-org#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\matrix-org#13972](matrix-org#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\matrix-org#14009](matrix-org#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\matrix-org#14025](matrix-org#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\matrix-org#13885](matrix-org#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\matrix-org#13934](matrix-org#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\matrix-org#13772](matrix-org#13772)) - Update URL for the NixOS module for Synapse. ([\matrix-org#13818](matrix-org#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\matrix-org#13836](matrix-org#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\matrix-org#13870](matrix-org#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\matrix-org#13911](matrix-org#13911), [\matrix-org#13930](matrix-org#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\matrix-org#13915](matrix-org#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\matrix-org#13928](matrix-org#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\matrix-org#13931](matrix-org#13931)) - Add some cross references to worker documentation. ([\matrix-org#13974](matrix-org#13974)) - Linkify urls in config documentation. ([\matrix-org#14003](matrix-org#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\matrix-org#13843](matrix-org#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\matrix-org#14024](matrix-org#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\matrix-org#13487](matrix-org#13487), [\matrix-org#13800](matrix-org#13800)) - Port push rules to using Rust. ([\matrix-org#13768](matrix-org#13768), [\matrix-org#13838](matrix-org#13838), [\matrix-org#13889](matrix-org#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13787](matrix-org#13787)) - Update the script which makes full schema dumps. ([\matrix-org#13792](matrix-org#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13796](matrix-org#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\matrix-org#13809](matrix-org#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\matrix-org#13823](matrix-org#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\matrix-org#13839](matrix-org#13839)) - Fix the release script not publishing binary wheels. ([\matrix-org#13850](matrix-org#13850)) - Raise issue if complement fails with latest deps. ([\matrix-org#13859](matrix-org#13859)) - Correct the comments in the complement dockerfile. ([\matrix-org#13867](matrix-org#13867)) - Create a new snapshot of the database schema. ([\matrix-org#13873](matrix-org#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\matrix-org#13874](matrix-org#13874), [\matrix-org#14013](matrix-org#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\matrix-org#13876](matrix-org#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\matrix-org#13879](matrix-org#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\matrix-org#13888](matrix-org#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\matrix-org#13890](matrix-org#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\matrix-org#13905](matrix-org#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\matrix-org#13913](matrix-org#13913)) - Complement image: propagate SIGTERM to all workers. ([\matrix-org#13914](matrix-org#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\matrix-org#13924](matrix-org#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\matrix-org#13925](matrix-org#13925), [\matrix-org#13993](matrix-org#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\matrix-org#13960](matrix-org#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\matrix-org#13966](matrix-org#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\matrix-org#13969](matrix-org#13969)) - Speed up calculating push actions in large rooms. ([\matrix-org#13973](matrix-org#13973), [\matrix-org#13992](matrix-org#13992)) - Enable update notifications from Github's dependabot. ([\matrix-org#13976](matrix-org#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\matrix-org#13998](matrix-org#13998), [\matrix-org#14011](matrix-org#14011), [\matrix-org#14017](matrix-org#14017), [\matrix-org#14021](matrix-org#14021), [\matrix-org#14027](matrix-org#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\matrix-org#14012](matrix-org#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\matrix-org#14020](matrix-org#14020)) - Bump versions of GitHub actions. ([\matrix-org#13978](matrix-org#13978), [\matrix-org#13979](matrix-org#13979), [\matrix-org#13980](matrix-org#13980), [\matrix-org#13982](matrix-org#13982), [\matrix-org#14015](matrix-org#14015), [\matrix-org#14019](matrix-org#14019), [\matrix-org#14022](matrix-org#14022), [\matrix-org#14023](matrix-org#14023)) # -----BEGIN PGP SIGNATURE----- # # iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmNNMOMQHGVyaWtAbWF0 # cml4Lm9yZwAKCRClQuTtGw+sCVnjB/9jpJRVnicteEpDfVX9iLo2qfIfcO/GhUJK # pJhv4yuY9whAldvJpmNw2f9tfUbAMcvrjlFvNrjihWmXcAGFazC6i3fNBjPgZW2e # Sxsuuy8xc9X/OqH2EUpHtNZQX3FfSbdBS93Z62ZO3R8tEbCQvjw6FXBdjjjf5uLO # y5Lsx94+41FJYOhs1Kt4fN92B9WMACR6e/O1YcsDjIXsoZI3uqO1h8filbQIZee7 # DTATE7eIPtShs2Ezaaeuc7tZGVDyPvgWIbuxuT6OGx20zmuChYJgIcVaD1me4UzJ # i9bVigtpYN0eUxuWnjLf7YC6Ys/Y9wZ7/lhdgaBwdbQKEJdpi+S4 # =JWaO # -----END PGP SIGNATURE----- # gpg: Signature made Mon Oct 17 11:39:31 2022 BST # gpg: using RSA key 053191DFF4670330465227F7A542E4ED1B0FAC09 # gpg: issuer "erik@matrix.org" # gpg: Can't check signature: No public key # Conflicts: # docker/Dockerfile # pyproject.toml # synapse/_scripts/update_synapse_database.py # synapse/handlers/message.py # synapse/handlers/receipts.py # synapse/logging/context.py # synapse/push/baserules.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/replication/http/send_event.py # synapse/rest/admin/users.py # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/rest/client/room.py # synapse/storage/_base.py # synapse/storage/databases/main/__init__.py # synapse/storage/databases/main/cache.py # synapse/storage/databases/main/events.py # synapse/storage/databases/main/receipts.py # tests/push/test_push_rule_evaluator.py

MadLittleMods added 30 commits August 22, 2022 20:13

Keep track when we tried to backfill an event

e0d7fab

Record in some fail spots

b8d55d3

Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…

f63d823

…-backfill-an-event

Record and clear attempts

bec26e2

Add changelog

fee37c3

Remove from when spam checker fails

d1290be

See #13589 (comment)

Custom upsert to increment

f9119d0

See #13589 (comment)

Fix lints

f5c6fe7

Remove extra whitespace

16ebec0

Move to correct folder

ce07aa1

Set the version back

5811ba1

Fix TypeError: not all arguments converted during string formatting

cf2b093

Add test to make sure failed backfill attempts are recorded

cbb4194

Clean up test

621c5d3

Fix comments

75c07bb

Add test for clearing backfill attempts

783cce5

Maybe a better comment

54ef84b

WIP: Just working on the query

7bf3e7f

Move comment to where it matters

37ff009

Silly graph pt 1

a58d191

Silly graph pt 2

f127ad1

Tests running (not working)

18abbf4

Passing test

23310f5

Add test for A and B

64e01d8

Add tests for backfill attempts

47bac25

Remove GROUP BY backward_extrem.event_id (seems unnecessary)

2ebed9d

See #13635 (comment)

Clarify why that much time

60b3b92

Label ? slot

e9f603d

Better explanation

a8f1464

Add changelog

bbd1c94

Base automatically changed from madlittlemods/13622-do-not-retry-backfilling-the-same-event-over-and-over to develop September 23, 2022 19:01

MadLittleMods added 3 commits September 23, 2022 14:11

Merge branch 'develop' into madlittlemods/backfill-points-filtering-a…

aae06fd

…nd-limits Conflicts: synapse/handlers/federation.py synapse/storage/databases/main/event_federation.py tests/storage/test_event_federation.py

I didn't try to change the lock (shrug)

bae72e4

Make sure our timing doesn't get thrown off

4f2b3ae

MadLittleMods added A-Messages-Endpoint /messages client API endpoint (`RoomMessageListRestServlet`) (which also triggers /backfill) T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. labels Sep 23, 2022

Add limit, fix lints and add changelog

44e2a5d

MadLittleMods marked this pull request as ready for review September 23, 2022 22:00

MadLittleMods requested a review from a team as a code owner September 23, 2022 22:00

richvdh self-requested a review September 27, 2022 17:35

richvdh suggested changes Sep 27, 2022

View reviewed changes

MadLittleMods added 9 commits September 27, 2022 14:16

Fixup language

685bbee

Remove redundant comment

9921f24

See #13879 (comment)

Simpler, behavior equivalent condition when nothing to backfill

7c96481

See #13879 (comment)

Explain what passing None does

b7368ba

See #13879 (comment)

Only explain what's necessary to understand current_depth

4823bb2

Explain what the depth correlates to

829cef8

See #13879 (comment)

More accurate descroption of get_backfill_points_in_room

db90283

See #13879 (comment)

Remove extra context that describes our usage

aba8b0f

See #13879 (comment)

Merge branch 'develop' into madlittlemods/backfill-points-filtering-a…

e727a31

…nd-limits

MadLittleMods requested a review from richvdh September 27, 2022 20:57

Point at specific event

379ccb4

See #13879 (comment)

richvdh reviewed Sep 28, 2022

View reviewed changes

richvdh approved these changes Sep 28, 2022

View reviewed changes

Start with a one line description

21f5933

See #13879 (comment)

MadLittleMods merged commit df8b91e into develop Sep 28, 2022

MadLittleMods deleted the madlittlemods/backfill-points-filtering-and-limits branch September 28, 2022 20:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit and filter the number of backfill points to get from the database #13879

Limit and filter the number of backfill points to get from the database #13879

MadLittleMods commented Sep 22, 2022 •

edited

richvdh Sep 27, 2022

MadLittleMods Sep 27, 2022 •

edited

richvdh Sep 28, 2022

richvdh Sep 28, 2022

richvdh left a comment

MadLittleMods commented Sep 28, 2022

Limit and filter the number of backfill points to get from the database #13879

Limit and filter the number of backfill points to get from the database #13879

Conversation

MadLittleMods commented Sep 22, 2022 • edited

Dev notes

Can we do anything smarter with two already sorted lists?

Todo

Pull Request Checklist

richvdh Sep 27, 2022

Choose a reason for hiding this comment

MadLittleMods Sep 27, 2022 • edited

Choose a reason for hiding this comment

richvdh Sep 28, 2022

Choose a reason for hiding this comment

richvdh Sep 28, 2022

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

MadLittleMods commented Sep 28, 2022

MadLittleMods commented Sep 22, 2022 •

edited

MadLittleMods Sep 27, 2022 •

edited