Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Commit

Permalink
devices: use combined ANY clause for faster cleanup
Browse files Browse the repository at this point in the history
Old device entries for the same user were being removed in individual
SQL commands, making the batch take way longer than necessary.

This combines the commands into a single one with a IN/ANY clause.

Example of log entry before the change, regularly observed with
"log_min_duration_statement = 10000" in PostgreSQL's config:

    LOG:  duration: 42538.282 ms  statement:
    DELETE FROM device_lists_stream
    WHERE user_id = '@Someone' AND device_id = 'someid1'
    AND stream_id < 123456789
    ;
    DELETE FROM device_lists_stream
    WHERE user_id = '@Someone' AND device_id = 'someid2'
    AND stream_id < 123456789
    ;
    [repeated for each device ID of that user, potentially a lot...]

With the patch applied on my instance for the past couple of days, I
no longer notice overly long statements of that particular kind.

Signed-off-by: pacien <pacien.trangirard@pacien.net>
  • Loading branch information
pacien committed Jul 2, 2023
1 parent 13fc891 commit b0c614e
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 5 deletions.
1 change: 1 addition & 0 deletions changelog.d/15861.misc
@@ -0,0 +1 @@
Optimised cleanup of old entries in device_lists_stream.
14 changes: 9 additions & 5 deletions synapse/storage/databases/main/devices.py
Expand Up @@ -1950,12 +1950,16 @@ def _add_device_change_to_stream_txn(

# Delete older entries in the table, as we really only care about
# when the latest change happened.
txn.execute_batch(
"""
cleanup_obsolete_stmt = """
DELETE FROM device_lists_stream
WHERE user_id = ? AND device_id = ? AND stream_id < ?
""",
[(user_id, device_id, min_stream_id) for device_id in device_ids],
WHERE user_id = ? AND stream_id < ? AND %s
"""
device_ids_clause, device_ids_args = make_in_list_sql_clause(
txn.database_engine, "device_id", device_ids
)
txn.execute(
cleanup_obsolete_stmt % (device_ids_clause,),
[user_id, min_stream_id] + device_ids_args,
)

self.db_pool.simple_insert_many_txn(
Expand Down

0 comments on commit b0c614e

Please sign in to comment.