Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make synapse_port_db correctly create indexes #6102

Merged
merged 37 commits into from Oct 23, 2019

Conversation

@babolivier
Copy link
Member

babolivier commented Sep 24, 2019

Make synapse_port_db correctly create indexes in the PostgreSQL database, by having it run the background updates on the database before migrating the data.

To ensure we're migrating the right data, also block the port if the SQLite3 database still has pending or ongoing background updates.

Fixes #4877

babolivier added 19 commits Jun 4, 2019
…ng the db
This is so it doesn't need a homeserver object to run
@babolivier babolivier requested a review from matrix-org/synapse-core Sep 24, 2019
babolivier added 2 commits Sep 24, 2019
@erikjohnston erikjohnston removed the request for review from matrix-org/synapse-core Sep 26, 2019
This way we don't have to rely on devs thinking about updating the script each time a new store starts using background updates.
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 1, 2019
@babolivier

This comment has been minimized.

Copy link
Member Author

babolivier commented Oct 1, 2019

ftr I'm looking at adding CI to the synapse_port_db script in another PR: #6140

@anoadragon453

This comment has been minimized.

Copy link
Member

anoadragon453 commented Oct 1, 2019

Manually tested on a homeserver with a few users and everything seemed to work ok \o/

Copy link
Member

richvdh left a comment

So this looks like a good plan, but I have some architectural concerns...

scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
babolivier added 3 commits Oct 3, 2019
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 4, 2019
babolivier added 5 commits Oct 3, 2019
The StatsStore class hasn't been split because it's tied to a lot of code, including code outside of its own file (such as get_event()), and doesn't register any looping call, so it should be fine to leave it as is.
…db_background_updates
@babolivier babolivier force-pushed the babolivier/port_db_background_updates branch from 87d1de3 to 1eb0600 Oct 7, 2019
@babolivier

This comment has been minimized.

Copy link
Member Author

babolivier commented Oct 7, 2019

As requested by @richvdh, the commits about factoring out the background updates from the database code have been moved to a dedicated PR: #6178 - this means that the reviewer can ignore any change to the synapse/storage directory when reviewing this PR.

The commits shown here are the ones pulled by merging that PR's branch into the current branch, since the current PR (#6102) now depends on it.

@hawkowl hawkowl added this to In progress in Homeserver Task Board via automation Oct 10, 2019
@hawkowl hawkowl moved this from In progress to Review in Homeserver Task Board Oct 10, 2019
Copy link
Member

richvdh left a comment

a few more suggestions for cleanups

scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts/synapse_port_db Show resolved Hide resolved
babolivier added 4 commits Oct 22, 2019
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 22, 2019
Copy link
Member

richvdh left a comment

lgtm otherwise. feel free to fix up and merge.

scripts/synapse_port_db Show resolved Hide resolved
scripts/synapse_port_db Outdated Show resolved Hide resolved
@babolivier babolivier merged commit c97ed64 into develop Oct 23, 2019
18 checks passed
18 checks passed
buildkite/synapse Build #5068 passed (19 minutes, 49 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 33 seconds)
Details
buildkite/synapse/check-style Passed (1 minute, 43 seconds)
Details
buildkite/synapse/isort Passed (17 seconds)
Details
buildkite/synapse/mypy Passed (49 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (16 seconds)
Details
buildkite/synapse/packaging Passed (17 seconds)
Details
buildkite/synapse/pipeline Passed (3 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (17 minutes, 28 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (6 minutes, 52 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (9 minutes, 13 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (6 minutes, 49 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (18 minutes)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (16 minutes, 43 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (6 minutes, 46 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (15 minutes, 12 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (13 minutes, 1 second)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (17 minutes, 51 seconds)
Details
Homeserver Task Board automation moved this from Review to Done Oct 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
3 participants
You can’t perform that action at this time.