Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix backfill storing incorrect state for events #4718

Merged
merged 7 commits into from Feb 25, 2019
Merged
Diff settings

Always

Just for now

Copy path View file
@@ -0,0 +1 @@
Fix paginating over federation persisting incorrect state

This comment has been minimized.

Copy link
@richvdh

richvdh Feb 25, 2019

Member

needs a full-stop

Copy path View file
@@ -770,10 +770,25 @@ def backfill(self, dest, room_id, limit, extremities):
set(auth_events.keys()) | set(state_events.keys())
)

# We now have a chunk of events plus associated state and auth chain to
# persist. We do the persistence in two steps:
# 1. Auth events and state get persisted as outliers, plus the
# backward extremities get persisted (as non-outliers).
# 2. The rest of the events in the chunk get persisted one by one, as
# each one depends on the previous event for its state.
#
# The important thing is that events in the chunk get persisted as
# non-outliers, including when those events are also in the state or
# auth chain. Caution must therefore be taken to ensure that they are
# not accidentally marked as outliers.

ev_infos = []
for a in auth_events.values():
if a.event_id in seen_events:
# We only want to persist auth events as outliers that we haven't
# seen and aren't about to persist as part of the backfilled chunk.
if a.event_id in seen_events or a.event_id in event_map:
continue

a.internal_metadata.outlier = True
ev_infos.append({
"event": a,
@@ -786,13 +801,18 @@ def backfill(self, dest, room_id, limit, extremities):
})

for e_id in events_to_state:
# For paranoia we ensure that these events are marked as

This comment has been minimized.

Copy link
@richvdh

richvdh Feb 25, 2019

Member

this seems to be in conflict with the comment at line 775 that says "... state get persisted as outliers". What am I missing?

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Feb 25, 2019

Author Member

This needs more comments, obviously. The keys of events_to_state are the new backwards extremities, as those are the events that we fetched state for

This comment has been minimized.

Copy link
@richvdh

richvdh Feb 25, 2019

Member

it would seem a whole lot less opaque to iterate over edges rather than the keys of events_to_state, but ok.

# non-outliers
ev = event_map[e_id]
ev.internal_metadata.outlier = False

This comment has been minimized.

Copy link
@richvdh

richvdh Feb 25, 2019

Member

this sort of thing always alarms me slightly; if outlier was set doesn't it mean that something else has gone wrong? shouldn't we just raise an AssertionError or something?

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Feb 25, 2019

Author Member

Assert is actually probably what we want here, yes


ev_infos.append({
"event": event_map[e_id],
"event": ev,
"state": events_to_state[e_id],
"auth_events": {
(auth_events[a_id].type, auth_events[a_id].state_key):
auth_events[a_id]
for a_id in event_map[e_id].auth_event_ids()
for a_id in ev.auth_event_ids()
if a_id in auth_events
}
})
@@ -808,6 +828,10 @@ def backfill(self, dest, room_id, limit, extremities):
if event in events_to_state:
continue

# For paranoia we ensure that these events are marked as
# non-outliers
event.internal_metadata.outlier = False

# We store these one at a time since each event depends on the
# previous to work out the state.
# TODO: We can probably do something more clever here.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.