Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Don't keep old stream_ordering_to_exterm around #15382

Merged
merged 7 commits into from Apr 6, 2023
Merged

Don't keep old stream_ordering_to_exterm around #15382

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fe0f38e
Don't keep old stream_ordering_to_exterm around
erikjohnston Apr 3, 2023
81a27da
Newsfile
erikjohnston Apr 3, 2023
6665a7f
Handle edge case
erikjohnston Apr 3, 2023
d5ccd2d
Check if extremities have changed
erikjohnston Apr 5, 2023
c9ee761
Rename func
erikjohnston Apr 6, 2023
4b3c536
Merge branch 'develop' into erikj/better_old_stream_orderings
erikjohnston Apr 6, 2023
527812c
Merge branch 'develop' into erikj/better_old_stream_orderings
erikjohnston Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/15382.misc
View file
Open in desktop
@@ -0,0 +1 @@
Improve DB performance of clearing out old data from `stream_ordering_to_exterm`.
20 changes: 10 additions & 10 deletions synapse/storage/databases/main/event_federation.py
View file
Open in desktop
Expand Up @@ -1232,10 +1232,17 @@ def get_forward_extremeties_for_room_txn(txn: LoggingTransaction) -> List[str]:
txn.execute(sql, (stream_ordering, room_id))
return [event_id for event_id, in txn]

return await self.db_pool.runInteraction(
event_ids = await self.db_pool.runInteraction(
"get_forward_extremeties_for_room", get_forward_extremeties_for_room_txn
)

# If we didn't find any IDs, then we must have cleared out the
# associated `stream_ordering_to_exterm`.
if not event_ids:
raise StoreError(400, "stream_ordering too old %s" % (stream_ordering,))

return event_ids

Comment on lines +1271 to +1277
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a new error condition, i.e. was it impossible for the interaction to return a falsey value before?

I think before this could fail, but we'd try to keep it working for the last month's worth of stream orderings?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not, and note that the one caller treats empty list and StoreError the same.

This is and was hit when we asked for the extremities at a recent ish stream ordering where we had purged all entries in the room before that stream ordering (we'd keep around one entry per room but that might be after that).

def _get_connected_batch_event_backfill_results_txn(
self, txn: LoggingTransaction, insertion_event_id: str, limit: int
) -> List[BackfillQueueNavigationItem]:
Expand Down Expand Up @@ -1664,19 +1671,12 @@ async def get_successor_events(self, event_id: str) -> List[str]:
@wrap_as_background_process("delete_old_forward_extrem_cache")
async def _delete_old_forward_extrem_cache(self) -> None:
def _delete_old_forward_extrem_cache_txn(txn: LoggingTransaction) -> None:
# Delete entries older than a month, while making sure we don't delete
# the only entries for a room.
Comment on lines -1667 to -1668
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chesterton's Fence: do we know why this was here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've been trying to figure this out, but couldn't see any reason for it until last night.

I think it may have been to protect against the case of a room that hasn't been updated in the last month, where you want to return the most recent extremities, so that the caller then becomes a no-op.

Let me think how best to keep that optimisation.

sql = """
DELETE FROM stream_ordering_to_exterm
WHERE
room_id IN (
SELECT room_id
FROM stream_ordering_to_exterm
WHERE stream_ordering > ?
) AND stream_ordering < ?
WHERE stream_ordering < ?
"""
txn.execute(
sql, (self.stream_ordering_month_ago, self.stream_ordering_month_ago) # type: ignore[attr-defined]
sql, (self.stream_ordering_month_ago) # type: ignore[attr-defined]
)

await self.db_pool.runInteraction(
Expand Down