Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dev script to generate full SQL schema files #6394

Merged
merged 13 commits into from Dec 11, 2019

Conversation

@anoadragon453
Copy link
Member

anoadragon453 commented Nov 20, 2019

Fixes #6476

So we ran into a bug that was a result of INSERT statements getting lost from converting SQL delta files to full schema. This was due to that process just copying over the schema of fully updated SQLite and Postgres tables, instead of the actual data.

Additionally, there wasn't any automated way (or much documentation) on how to generate full schemas, and I think if we had a canonical, automated way we might do it more often.

The script does this by generating a sqlite database with all background updates applied to it, then making use of the now-working synapse_port_db script to convert it to a postgres database. These two databases are then dumped to full.{sqlite,postgres}.sql files.

Also converted the full schema README from .txt to .md.

@anoadragon453 anoadragon453 requested a review from matrix-org/synapse-core Nov 20, 2019
@anoadragon453 anoadragon453 self-assigned this Nov 20, 2019
@anoadragon453 anoadragon453 added this to In progress in Homeserver Task Board via automation Nov 20, 2019
@anoadragon453 anoadragon453 moved this from In progress to Review in Homeserver Task Board Nov 20, 2019
@anoadragon453 anoadragon453 force-pushed the anoa/update_full_schema_instructions branch from f019aff to f6eff4b Nov 20, 2019
Copy link
Member

richvdh left a comment

looks generally good, though it's bash so I have Opinions.

scripts/synapse_port_db Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/generate_full_schema.sh Outdated Show resolved Hide resolved
rm "$KEY_FILE"
dropdb $POSTGRES_DB_NAME

# Remove last pesky instance of this table from the output

This comment has been minimized.

Copy link
@richvdh

richvdh Nov 27, 2019

Member

what's going on here, ooi?

anoadragon453 and others added 4 commits Dec 2, 2019
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
scripts-dev/make_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/make_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/make_full_schema.sh Outdated Show resolved Hide resolved
cat > "$SQLITE_CONFIG" <<EOF
server_name: "test"
signing_key_path: "$KEY_FILE"

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 4, 2019

Member

so what happens if you don't set this?

This comment has been minimized.

Copy link
@anoadragon453

anoadragon453 Dec 4, 2019

Author Member

It looks for test.signing.key in the current directory, not the temp dir.

This comment has been minimized.

Copy link
@anoadragon453

anoadragon453 Dec 10, 2019

Author Member

Might be a bug. It's due to config_dir_path being "" here:

def read_config(self, config, config_dir_path, **kwargs):

This comment has been minimized.

Copy link
@richvdh

richvdh Dec 11, 2019

Member

well, that doesn't sound right. Ok, let's not get distracted by going on a bughunt now.

scripts-dev/make_full_schema.sh Outdated Show resolved Hide resolved
scripts-dev/make_full_schema.sh Outdated Show resolved Hide resolved
@anoadragon453 anoadragon453 removed this from Review in Homeserver Task Board Dec 5, 2019
Copy link
Member

richvdh left a comment

🚢

@anoadragon453 anoadragon453 merged commit 6676ee9 into develop Dec 11, 2019
22 checks passed
22 checks passed
buildkite/synapse Build #6036 passed (20 minutes, 45 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 46 seconds)
Details
buildkite/synapse/check-style Passed (2 minutes, 12 seconds)
Details
buildkite/synapse/isort Passed (31 seconds)
Details
buildkite/synapse/mypy Passed (25 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (28 seconds)
Details
buildkite/synapse/packaging Passed (32 seconds)
Details
buildkite/synapse/pipeline Passed (4 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (18 minutes, 20 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (7 minutes, 38 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (8 minutes, 45 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (7 minutes, 30 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (17 minutes, 32 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (17 minutes, 57 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (7 minutes, 14 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-5-slash-postgres-9-dot-5 Passed (1 minute, 17 seconds)
Details
buildkite/synapse/synapse-port-db-slash-python-3-dot-7-slash-postgres-11 Passed (1 minute, 17 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (13 minutes, 59 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (12 minutes, 58 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (12 minutes, 7 seconds)
Details
buildkite/synapse/sytest-python-3-dot-7-slash-postgres-11-slash-monolith Passed (11 minutes, 48 seconds)
Details
buildkite/synapse/sytest-python-3-dot-7-slash-postgres-11-slash-workers Passed (11 minutes, 20 seconds)
Details
@anoadragon453 anoadragon453 deleted the anoa/update_full_schema_instructions branch Dec 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.