Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Speed up persisting redacted events #10756

Merged
merged 3 commits into from
Sep 6, 2021
Merged

Speed up persisting redacted events #10756

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0ecb870
Speed up persisting redacted events
erikjohnston Sep 3, 2021
14908b0
Newsfile
erikjohnston Sep 3, 2021
1e9f130
Fixup
erikjohnston Sep 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/10756.misc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Minor speed ups when joining large rooms over federation.
22 changes: 11 additions & 11 deletions synapse/storage/databases/main/events.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1379,18 +1379,18 @@ def get_internal_metadata(event):
# If we're persisting an unredacted event we go and ensure
# that we mark any redactions that reference this event as
# requiring censoring.
sql = "UPDATE redactions SET have_censored = ? WHERE redacts = ?"
txn.execute_batch(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not execute_values here? I guess it's even faster to just do a single query?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, so the rough order of speed I believe is make_in_list_sql_clause > execute_values > execute_batch. execute_values is just a helper function that internally does a bunch of sql building to turn a list of rows into something that postgres will understand, so just passing in the array bypasses all of that nonsense.

sql,
(
(
False,
event.event_id,
)
for event, _ in events_and_contexts
if not event.internal_metadata.is_redacted()
),
unredacted_events = [
event.event_id
for event, _ in events_and_contexts
if not event.internal_metadata.is_redacted()
]
sql = "UPDATE redactions SET have_censored = ? WHERE "
clause, args = make_in_list_sql_clause(
self.database_engine,
"redacts",
unredacted_events,
)
txn.execute(sql + clause, [False] + args)

state_events_and_contexts = [
ec for ec in events_and_contexts if ec[0].is_state()
Expand Down