Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add database config class #6513

Merged
merged 7 commits into from Dec 18, 2019
Merged

Add database config class #6513

merged 7 commits into from Dec 18, 2019

Conversation

@erikjohnston
Copy link
Member

erikjohnston commented Dec 10, 2019

Replaces #6481.

Based on #6510, #6511 and #6512.

@erikjohnston erikjohnston force-pushed the erikj/split_database_config branch 3 times, most recently from 3137d27 to 210b520 Dec 10, 2019
This encapsulates config for a given database and is the way to get new
connections.
@erikjohnston erikjohnston force-pushed the erikj/split_database_config branch from 210b520 to 427c9e3 Dec 12, 2019
@erikjohnston erikjohnston force-pushed the erikj/split_database_config branch from 427c9e3 to 24bed14 Dec 12, 2019
@erikjohnston erikjohnston requested a review from matrix-org/synapse-core Dec 12, 2019
@erikjohnston

This comment has been minimized.

Copy link
Member Author

erikjohnston commented Dec 12, 2019

This is still quite large I'm afraid, though a fair chunk of it is just fixing up tests.

synapse/config/database.py Outdated Show resolved Hide resolved
from twisted.enterprise import adbapi

from synapse.config._base import Config, ConfigError
from synapse.storage.engines import create_engine

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 13, 2019

Member

having config import from synapse.storage sounds like a great way to end up with horrible import loops (I found a fun one yesterday where if you import synapse.storage first you're ok, but if you import synapse.state it all explodes).

It seems to me that DatabaseConnectionConfig is doing more than a config object should: why can't the call to create_engine, and the get_pool and make_conn and co move into the Database class?

An alternative looks like it might be to move create_engine into config, but that feels wrong to me.

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 13, 2019

Member

(that said: a big +1 to moving these things out of the HomeServer god object)

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Dec 13, 2019

Author Member

I don't disagree, it is a pain. At the moment the Database object represents a fully prepared and configured database, we could move the preparation into the __init__, but then we need want to reuse the connection to give to the data stores, which is fiddly if we create the connection in the Database :(

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 13, 2019

Member

then we need want to reuse the connection to give to the data stores, which is fiddly if we create the connection in the Database :(

I'm not quite following this?

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Dec 16, 2019

Author Member

Basically its a bit hard to see how to rewrite https://github.com/matrix-org/synapse/pull/6513/files#diff-23406f76811c2c98a6ed1758c0bda2abR42-R54 if we move make_conn to Database

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 16, 2019

Member

I see. Make it a top-level function, in synapse.storage.database (or similar), which takes a DatabaseConfig as a param?

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Dec 16, 2019

Author Member

Hmm, ok, I can give that a go

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Dec 16, 2019

Author Member

So 439e043 is my attempt at that, the tests are a bit hacky but that's true of them already.

synapse/config/database.py Show resolved Hide resolved
synapse/config/database.py Show resolved Hide resolved
@erikjohnston erikjohnston force-pushed the erikj/split_database_config branch 2 times, most recently from 7318971 to 7b297a4 Dec 16, 2019
@erikjohnston erikjohnston force-pushed the erikj/split_database_config branch from 7b297a4 to 439e043 Dec 16, 2019
@erikjohnston erikjohnston requested a review from richvdh Dec 16, 2019
Copy link
Member

richvdh left a comment

looks much better!

logger = logging.getLogger(__name__)


class DatabaseConnectionConfig:

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 18, 2019

Member

this might make more sense as an attr.s ? Just a thought

This comment has been minimized.

Copy link
@erikjohnston

erikjohnston Dec 18, 2019

Author Member

Yeah, I thought about that, but then I find the validation stuff non intuitive.

@erikjohnston erikjohnston merged commit 2284eb3 into develop Dec 18, 2019
22 checks passed
22 checks passed
buildkite/synapse Build #6180 passed (22 minutes, 29 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 34 seconds)
Details
buildkite/synapse/check-style Passed (2 minutes, 13 seconds)
Details
buildkite/synapse/isort Passed (17 seconds)
Details
buildkite/synapse/mypy Passed (31 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (15 seconds)
Details
buildkite/synapse/packaging Passed (19 seconds)
Details
buildkite/synapse/pipeline Passed (3 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (17 minutes, 19 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (7 minutes, 19 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (8 minutes, 43 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (7 minutes)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (19 minutes, 17 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (20 minutes, 1 second)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (6 minutes, 58 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-5-slash-postgres-9-dot-5 Passed (1 minute, 19 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-7-slash-postgres-11 Passed (1 minute, 28 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (13 minutes, 15 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (17 minutes, 7 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (10 minutes, 39 seconds)
Details
buildkite/synapse/sytest-python-3-dot-7-slash-postgres-11-slash-monolith Passed (11 minutes, 29 seconds)
Details
buildkite/synapse/sytest-python-3-dot-7-slash-postgres-11-slash-workers Passed (13 minutes, 55 seconds)
Details
@erikjohnston erikjohnston deleted the erikj/split_database_config branch Jan 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.