Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split purge API into events vs state and add PurgeEventsStorage #6295

Merged
merged 14 commits into from Nov 8, 2019

Conversation

@erikjohnston
Copy link
Member

erikjohnston commented Oct 30, 2019

This is a bit big as we needed to split out the purge functions to work on events and state in separate transactions.

(Split out from #6245.)

@erikjohnston erikjohnston force-pushed the erikj/split_purge_history branch from 0251a95 to ecfba89 Oct 30, 2019
@erikjohnston erikjohnston requested a review from matrix-org/synapse-core Oct 30, 2019
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
@erikjohnston erikjohnston requested a review from richvdh Oct 31, 2019
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
synapse/storage/data_stores/main/state.py Outdated Show resolved Hide resolved
erikjohnston added 3 commits Nov 1, 2019
This does mean that we won't clean up orphaned state groups (i.e. state
groups that were persisted but the associated event wasn't).
@erikjohnston erikjohnston requested a review from richvdh Nov 4, 2019
# Now we fetch all the state groups that should be deleted.
txn.execute(
"""
SELECT DISTINCT state_group FROM events

This comment has been minimized.

Copy link
@richvdh

richvdh Nov 6, 2019

Member

erm, haven't we just deleted all the relevant rows from events?

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Nov 6, 2019

Author Member

Gah. And the tests were broken.

synapse/storage/data_stores/main/events.py Outdated Show resolved Hide resolved
Copy link
Member

richvdh left a comment

^

erikjohnston added 2 commits Nov 6, 2019
And fix the tests to actually test that things got deleted.
@erikjohnston erikjohnston requested a review from richvdh Nov 6, 2019
@richvdh
richvdh approved these changes Nov 6, 2019
Copy link
Member

richvdh left a comment

well, it looks plausible now, but your CI is borked

@erikjohnston

This comment has been minimized.

Copy link
Member Author

erikjohnston commented Nov 6, 2019

Oh, Collection isn't available on 3.5?

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Nov 6, 2019

it's in typing_extensions

@erikjohnston

This comment has been minimized.

Copy link
Member Author

erikjohnston commented Nov 6, 2019

Hmm, I don't see it in typing_extensions anywhere?

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Nov 6, 2019

gah. seems you're right. Maybe we could define our own on 3.5. This is starting to get out of scope though.

@erikjohnston erikjohnston merged commit f713c01 into develop Nov 8, 2019
20 checks passed
20 checks passed
buildkite/synapse Build #5482 passed (22 minutes, 51 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 25 seconds)
Details
buildkite/synapse/check-style Passed (2 minutes, 9 seconds)
Details
buildkite/synapse/isort Passed (17 seconds)
Details
buildkite/synapse/mypy Passed (24 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (14 seconds)
Details
buildkite/synapse/packaging Passed (41 seconds)
Details
buildkite/synapse/pipeline Passed (10 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (18 minutes, 31 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (6 minutes, 38 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (9 minutes, 12 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (7 minutes, 27 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (18 minutes, 55 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (18 minutes, 42 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (7 minutes, 10 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-5-slash-postgres-9-dot-5 Passed (1 minute, 45 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-7-slash-postgres-11 Passed (1 minute, 48 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (16 minutes, 4 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (14 minutes, 42 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (14 minutes, 19 seconds)
Details
@erikjohnston

This comment has been minimized.

Copy link
Member Author

erikjohnston commented Nov 8, 2019

I've just moved it into the docstring for now until we figure out what we're doing :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.