Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Don't add rejections to the state_group, persist all rejections #948

Merged
merged 5 commits into from Jul 26, 2016

Conversation

Projects
None yet
3 participants
Contributor

NegativeMjark commented Jul 25, 2016

No description provided.

Owner

erikjohnston commented Jul 25, 2016 edited

LGTM

Err, i take that back

NegativeMjark added some commits Jul 25, 2016

Member

richvdh commented Jul 26, 2016

I can't speak to the correctness of the changes themselves, but in my experience, when stuff starts getting complicated, lots of comments explaining what is going on can really help. I'm surprised to see quite a lot of code here, with little attention to comments explaining why things are doing what they are doing.

@richvdh richvdh commented on an outdated diff Jul 26, 2016

synapse/storage/events.py
@@ -442,6 +433,9 @@ def _persist_events_txn(self, txn, events_and_contexts, backfilled):
# Handle the case of the list including the same event multiple
# times. The tricky thing here is when they differ by whether
# they are an outlier.
+ if context.rejected:
@richvdh

richvdh Jul 26, 2016

Member

this code appears to have landed between a comment and the code it refers to, thus making the comment confusing. Because this method wasn't confusing enough before, I suppose.

@richvdh richvdh commented on an outdated diff Jul 26, 2016

synapse/storage/events.py
@@ -591,10 +557,28 @@ def event_dict(event):
],
)
- if context.rejected:
- self._store_rejections_txn(
- txn, event.event_id, context.rejected
- )
+ to_remove = set()
@richvdh

richvdh Jul 26, 2016

Member

to_remove is already declared at line 440. Is reinitialising it deliberate? needs explanation if so

@richvdh

richvdh Jul 26, 2016

Member

ok I see it's deliberate. reusing the same variable is bad karma imho. But basically it stems from this method being too big. Factor some of this shit out.

@richvdh richvdh commented on an outdated diff Jul 26, 2016

synapse/storage/events.py
@@ -610,6 +594,42 @@ def event_dict(event):
],
)
+ if event.type == EventTypes.Redaction and event.redacts is not None:
@richvdh

richvdh Jul 26, 2016

Member

is this acting on the last event in the loop at line 577 deliberately? needs explanation if so

@richvdh

richvdh Jul 26, 2016

Member

or maybe it's the event from line 608? NOBODY KNOWS

@richvdh richvdh commented on the diff Jul 26, 2016

synapse/storage/events.py
@@ -610,6 +594,42 @@ def event_dict(event):
],
)
+ if event.type == EventTypes.Redaction and event.redacts is not None:
+ self._remove_push_actions_for_event_id_txn(
+ txn, event.room_id, event.redacts
+ )
+
+ self._store_mult_state_groups_txn(txn, events_and_contexts)
+
+ self._handle_mult_prev_events(
+ txn,
+ events=[event for event, _ in events_and_contexts],
+ )
+
+ for event, _ in events_and_contexts:
+ if event.type == EventTypes.Name:
@richvdh

richvdh Jul 26, 2016

Member

factor this fucker out: you've already gone to the effort of moving it. Move it somewhere sensible instead.

@NegativeMjark

NegativeMjark Jul 26, 2016

Contributor

I'm not sure I can reasonably factor it out since it is such a mess in it's current state.
Note the wonderfully inconsistent mixture of tables being interfered with here.

@richvdh

richvdh Jul 26, 2016

Member

surely you can make a _store_event_in_tables(self, txn, event) which is literally lines 646-663?

@erikjohnston erikjohnston commented on the diff Jul 26, 2016

synapse/storage/events.py
- elif event.type == EventTypes.Redaction:
- self._store_redaction(txn, event)
- elif event.type == EventTypes.RoomHistoryVisibility:
- self._store_history_visibility_txn(txn, event)
- elif event.type == EventTypes.GuestAccess:
- self._store_guest_access_txn(txn, event)
-
- self._store_room_members_txn(
- txn,
- [
- event
- for event, _ in events_and_contexts
- if event.type == EventTypes.Member
- ],
- backfilled=backfilled,
- )
@erikjohnston

erikjohnston Jul 26, 2016

Owner

I'd be sorely tempted to keep store this information for rejected events, at least for now, to keep this changeset minimal.

@NegativeMjark

NegativeMjark Jul 26, 2016

Contributor

Do we have any mechanism to unreject an event yet?
Do we have any mechanism to asynchronously reject an event that we already accepted?
It looks like the answer to both question is no in the current code base. The only place we write to the rejections table is in _persist_event, after we've filtered out events that we already have in the db

Therefore I'm sorely tempted to exclude everything except the events and the event_json table until it is demonstrated that it is necessary and safe to do so.

Owner

erikjohnston commented Jul 26, 2016

I guess the main changes here are: don't add rejected events to event forward extremities table (and don#t and them to the push actions).

Owner

erikjohnston commented Jul 26, 2016

Although I'm slightly concerned that we have stopped storing various bits of information about the event, and how that affects other storage functions. I'd prefer if, for now, the changes were as minimal as possible so that we can get the fix out and have refactoring (and commenting) as a separate PR.

NegativeMjark added some commits Jul 26, 2016

Don't add rejected events if we've seen them befrore. Add some commen…
…ts to explain what the code is doing mechanically

@NegativeMjark NegativeMjark merged commit 9c4cf83 into develop Jul 26, 2016

9 of 10 checks passed

Flake8 + Packaging (Merged PR) Build finished.
Details
Flake8 + Packaging (Commit) Build #1234 origin/markjh/auth_fixes succeeded in 56 sec
Details
Sytest Dendron (Commit) Build #345 origin/markjh/auth_fixes succeeded in 6 min 51 sec
Details
Sytest Dendron (Merged PR) Build finished.
Details
Sytest Postgres (Commit) Build #1180 origin/markjh/auth_fixes succeeded in 6 min 31 sec
Details
Sytest Postgres (Merged PR) Build finished.
Details
Sytest SQLite (Commit) Build #1205 origin/markjh/auth_fixes succeeded in 5 min 24 sec
Details
Sytest SQLite (Merged PR) Build finished.
Details
Unit Tests (Commit) Build #1271 origin/markjh/auth_fixes succeeded in 2 min 25 sec
Details
Unit Tests (Merged PR) Build finished.
Details

@richvdh richvdh deleted the markjh/auth_fixes branch Dec 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment