Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Batch up storing state groups when creating new room #14918

Merged
merged 15 commits into from Feb 24, 2023
Merged

Batch up storing state groups when creating new room #14918

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a47ee14
add a function to store state groups for batched events/contexts
H-Shay Nov 3, 2022
8f4282c
add a function to batch persist unpersisted event contexts
H-Shay Jan 26, 2023
b9bef42
refactor function that batch persists state groups to use unpersisted…
H-Shay Jan 26, 2023
4f991a0
batch persist state groups when creating room
H-Shay Jan 26, 2023
eba04f5
persist contexts after `creat_event` instead of during
H-Shay Jan 26, 2023
bc0ccd1
lints
H-Shay Jan 26, 2023
7ec7b9a
newsfragment
H-Shay Jan 26, 2023
c4646cb
requested changes
H-Shay Feb 8, 2023
4f6f50f
requested changes
H-Shay Feb 14, 2023
1f752ff
re-add mistakenly removed spaces
H-Shay Feb 14, 2023
8634ba6
deal more effectively with state/non-state
H-Shay Feb 15, 2023
467a731
Merge branch 'develop' into shay/finish_batching
H-Shay Feb 15, 2023
3d564ef
fix tests typing
H-Shay Feb 15, 2023
5811fd9
requested changes
H-Shay Feb 23, 2023
d92b3a3
add a test for store_state_deltas_for_batched
H-Shay Feb 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
41 changes: 19 additions & 22 deletions synapse/storage/databases/state/store.py
View file
Open in desktop
Expand Up @@ -457,13 +457,14 @@ def insert_deltas_group_txn(
)
H-Shay marked this conversation as resolved.
Show resolved Hide resolved

sg_before = prev_group
for index, (event, context) in enumerate(events_and_context):
state_group_iter = iter(state_groups)
for event, context in events_and_context:
if not event.is_state():
context.state_group_after_event = sg_before
context.state_group_before_event = sg_before
pass
continue
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hahaha, I did not notice that


sg_after = state_groups[index]
sg_after = next(state_group_iter)
context.state_group_after_event = sg_after
context.state_group_before_event = sg_before
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These context don't have a prev_group, which is technically fine, but there is other code that looks at prev_group for optimisation purposes. We probably do want to add them, hopefully easy enough.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do set the prev group immediately after this function returns and the UnpersistedEventContext is turned into an EventContext (in the batch_persist_unpersisted_contexts function), the UnpersistedEventContext isn't used anywhere else so I think it is fine?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, OK.

context.state_delta_due_to_event = {
Expand All @@ -478,6 +479,7 @@ def insert_deltas_group_txn(
values=[
(context.state_group_after_event, room_id, event.event_id)
for event, context in events_and_context
H-Shay marked this conversation as resolved.
Show resolved Hide resolved
if event.is_state()
],
)

Expand All @@ -490,32 +492,27 @@ def insert_deltas_group_txn(
context.state_group_after_event,
context.state_group_before_event,
)
for _, context in events_and_context
for event, context in events_and_context
if event.is_state()
],
)

values = []
for _, context in events_and_context:
assert context.state_delta_due_to_event is not None
for (
key,
state_id,
) in context.state_delta_due_to_event.items():
values.append(
(
context.state_group_after_event,
room_id,
key[0],
key[1],
state_id,
)
)

self.db_pool.simple_insert_many_txn(
txn,
table="state_groups_state",
keys=("state_group", "room_id", "type", "state_key", "event_id"),
values=values,
values=[
(
context.state_group_after_event,
room_id,
key[0],
key[1],
state_id,
)
for event, context in events_and_context
if context.state_delta_due_to_event is not None
for key, state_id in context.state_delta_due_to_event.items()
],
)
return events_and_context

Expand Down