Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor storage layer to support multiple databases #6231

Merged
merged 11 commits into from Oct 23, 2019

Conversation

@erikjohnston
Copy link
Member

erikjohnston commented Oct 21, 2019

This is an initial refactor that starts the split out of DataStore into the concept of "data stores". Broadly, this moves all the stores in synapse.storage into synapse.storage.data_stores.main and fixes up prepare_database to correctly apply schemas from multiple data stores.

The commits should be independently reviewable. The first one is the big one, and is the moving of files. The only changes are that helper classes/exceptions/etc that are imported from outside synapse.storage get left in synapse.storage rather than getting moved into a data store (the aim being that things outside storage rarely if ever need to know about the concept of data stores).

Currently this still exports the same DataStore in synapse.storage, but splitting that out some more is a job for a separate PR

This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
@erikjohnston erikjohnston marked this pull request as ready for review Oct 22, 2019
@erikjohnston erikjohnston force-pushed the erikj/refactor_stores branch from 402067b to acf47c7 Oct 22, 2019
@erikjohnston erikjohnston requested a review from matrix-org/synapse-core Oct 22, 2019
synapse/storage/prepare_database.py Show resolved Hide resolved
synapse/storage/prepare_database.py Show resolved Hide resolved
synapse/storage/prepare_database.py Show resolved Hide resolved
synapse/storage/prepare_database.py Outdated Show resolved Hide resolved
synapse/storage/prepare_database.py Outdated Show resolved Hide resolved
synapse/storage/prepare_database.py Outdated Show resolved Hide resolved
synapse/storage/README.md Outdated Show resolved Hide resolved
@erikjohnston erikjohnston requested a review from richvdh Oct 22, 2019
Copy link
Member

richvdh left a comment

lgtm

@erikjohnston erikjohnston merged commit 7b6d99f into develop Oct 23, 2019
18 checks passed
18 checks passed
buildkite/synapse Build #5046 passed (20 minutes, 34 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 28 seconds)
Details
buildkite/synapse/check-style Passed (1 minute, 39 seconds)
Details
buildkite/synapse/isort Passed (25 seconds)
Details
buildkite/synapse/mypy Passed (21 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (14 seconds)
Details
buildkite/synapse/packaging Passed (18 seconds)
Details
buildkite/synapse/pipeline Passed (3 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (18 minutes, 30 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (6 minutes, 39 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (8 minutes, 54 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (6 minutes, 23 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (17 minutes, 54 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (18 minutes, 42 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (6 minutes, 27 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (15 minutes, 18 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (11 minutes, 55 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (15 minutes, 1 second)
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.