Skip to content

/synapse

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CI for synapse_port_db #6140

Merged
merged 21 commits into from Oct 28, 2019

Conversation

@babolivier
Copy link
Member

babolivier commented Oct 1, 2019 •
edited

Paired with matrix-org/pipelines#9

This PR adds:

  • a test sqlite database
  • a configuration file for the sqlite database
  • a configuration file for a postgresql database (using the credentials in .buildkite/docker-compose.pyXX.pgXX.yaml)

as well as a new script named .buildkite/scripts/test_synapse_port_db.sh that:

  1. installs Synapse
  2. updates the test sqlite database to the latest schema and runs background updates on it
  3. creates an empty postgresql database
  4. run the synapse_port_db script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step 2 is done via a new script located at scripts-dev/update_database.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
@babolivier babolivier mentioned this pull request Oct 1, 2019
Merged
@babolivier babolivier force-pushed the babolivier/port_db_ci branch 3 times, most recently from 6d062d3 to f370c32 Oct 3, 2019
@babolivier babolivier mentioned this pull request Oct 4, 2019
Merged
@babolivier babolivier marked this pull request as ready for review Oct 4, 2019
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 4, 2019
@babolivier babolivier force-pushed the babolivier/port_db_ci branch from c191072 to 0b68a54 Oct 7, 2019
@erikjohnston

This comment has been minimized.

Sign in to view
Copy link
Member

erikjohnston commented Oct 8, 2019

Can you give us a commit range to review or something please? There are too many commits to figure out whats going on without some help :)
@erikjohnston erikjohnston removed the request for review from matrix-org/synapse-core Oct 8, 2019
@babolivier

This comment has been minimized.

Sign in to view
Copy link
Member Author

babolivier commented Oct 8, 2019 •
edited

Can you give us a commit range to review or something please? There are too many commits to figure out whats going on without some help :)

Right, sorry, I assumed this review would come after the ones on the PRs this one is built on top of. A list of all commits specific to this PR can be seen here, and for maximum clarity this PR only changes the following files:

  • .buildkite/scripts/test_synapse_port_db.sh (creation)
  • .buildkite/postgres-config.yaml (creation)
  • .buildkite/sqlite-config.yaml (creation)
  • .buildkite/test_db.db (creation)
  • scripts-dev/create_postgres_db (creation)
  • scripts-dev/update_database (creation)
  • tox.ini (addition of the synapse_port_db test environment)

None of the other PRs edit these files, so the reviewer can review just the changes to these files, without worrying about whether they've been changed by this PR or another.
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 8, 2019
@richvdh
richvdh reviewed Oct 10, 2019
View changes
Copy link
Member

richvdh left a comment

a few comments; it kinda needs #6102 to land before it gets a proper review.
tox.ini Outdated Show resolved Hide resolved
scripts-dev/create_postgres_db Outdated Show resolved Hide resolved
scripts-dev/update_database Show resolved Hide resolved
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 22, 2019
@richvdh

This comment has been minimized.

Sign in to view
Copy link
Member

richvdh commented Oct 23, 2019

pending merge of #6102
@richvdh richvdh removed the request for review from matrix-org/synapse-core Oct 23, 2019
babolivier added 13 commits Sep 27, 2019
@babolivier
Add script to update a database and run background updates on it
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
0ec5d79
@babolivier
Doc, logging and general cleanup
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
2b32099
@babolivier
Fix copyright
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
4ab1973
@babolivier
Add dev script to create a postgres DB
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
de34656
@babolivier
Add config files and test database
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
2ebd17b
@babolivier
Create tox virtualenv
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
177d55b
@babolivier
Use new stores instead of the DataStore
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
f6d727b
@babolivier
Use the right values for the postgresql configuration file
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
b2cd359
@babolivier
Move update_database script to scripts-dev
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
d8a1938
@babolivier
Changelog
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
c41e5da
@babolivier
Remove db creation script
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
44dd405
@babolivier
Remove warning about trusted key servers
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
0716a90
@babolivier
Fix database upgrade
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
bb3101c
babolivier added 2 commits Oct 22, 2019
@babolivier
Move tox environment instructions to a bash script
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
e9a6787
@babolivier
Remove test data
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
Loading status checks…
05463e4
@babolivier babolivier force-pushed the babolivier/port_db_ci branch from d9efe03 to 05463e4 Oct 23, 2019
babolivier added 2 commits Oct 23, 2019
@babolivier
Remove the tox env
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
Loading status checks…
c228f4f
@babolivier
Fix changelog
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
Loading status checks…
ca61e5d
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 23, 2019
@babolivier
Merge branch 'develop' into babolivier/port_db_ci
Loading status checks…
789cf83
@babolivier babolivier added this to In progress in Homeserver Task Board via automation Oct 23, 2019
@babolivier babolivier moved this from In progress to Review in Homeserver Task Board Oct 23, 2019
@babolivier

This comment has been minimized.

Sign in to view
Copy link
Member Author

babolivier commented Oct 23, 2019

#6102 has been merged; I've rebased the branch on develop so it doesn't pull any more commit than it should.
@richvdh
richvdh reviewed Oct 24, 2019
View changes
changelog.d/6140.bugfix Outdated Show resolved Hide resolved
.buildkite/postgres-config.yaml Show resolved Hide resolved
.buildkite/scripts/test_synapse_port_db.sh Outdated Show resolved Hide resolved
.buildkite/sqlite-config.yaml Show resolved Hide resolved
.buildkite/scripts/test_synapse_port_db.sh Outdated Show resolved Hide resolved
.buildkite/scripts/test_synapse_port_db.sh Outdated Show resolved Hide resolved
scripts-dev/update_database Outdated Show resolved Hide resolved
scripts-dev/update_database Outdated Show resolved Hide resolved
scripts-dev/update_database Outdated Show resolved Hide resolved
scripts-dev/update_database Show resolved Hide resolved
babolivier added 2 commits Oct 24, 2019
@babolivier
Incorporate review
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
2066629
@babolivier
Upgrade the test db to the most recent schema
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
Loading status checks…
dbd751b
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 24, 2019
@richvdh richvdh self-assigned this Oct 24, 2019
@richvdh
richvdh reviewed Oct 28, 2019
View changes
.buildkite/postgres-config.yaml Outdated Show resolved Hide resolved
.buildkite/scripts/test_synapse_port_db.sh Show resolved Hide resolved
.buildkite/scripts/test_synapse_port_db.sh Outdated Show resolved Hide resolved
@richvdh
richvdh reviewed Oct 28, 2019
View changes
scripts-dev/update_database Show resolved Hide resolved
@babolivier
Incorporate review
Verified
This commit was signed with a verified signature.
babolivier Brendan Abolivier
GPG key ID: 1E015C145F1916CD Learn about signing commits
Loading status checks…
60005f5
@babolivier babolivier requested a review from matrix-org/synapse-core Oct 28, 2019
@richvdh
richvdh approved these changes Oct 28, 2019
View changes
Copy link
Member

richvdh left a comment

lgtm!
@babolivier babolivier merged commit 14504ad into develop Oct 28, 2019
18 checks passed
18 checks passed
buildkite/synapse Build #5153 passed (20 minutes, 24 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 37 seconds)
Details
buildkite/synapse/check-style Passed (1 minute, 56 seconds)
Details
buildkite/synapse/isort Passed (39 seconds)
Details
buildkite/synapse/mypy Passed (25 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (16 seconds)
Details
buildkite/synapse/packaging Passed (21 seconds)
Details
buildkite/synapse/pipeline Passed (10 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (17 minutes, 16 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (6 minutes, 50 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (9 minutes, 11 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (6 minutes, 38 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (17 minutes, 45 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (17 minutes, 30 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (6 minutes, 32 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (12 minutes, 51 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (14 minutes, 8 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (13 minutes, 41 seconds)
Details
Homeserver Task Board automation moved this from Review to Done Oct 28, 2019
richvdh added a commit to matrix-org/pipelines that referenced this pull request Oct 28, 2019
@babolivier @richvdh
Add CI jobs for synapse_port_db (#9)
36c9db5 
Paired with matrix-org/synapse#6140

Defines the Synapse CI jobs to run the tox environment added in matrix-org/synapse#6140
@richvdh richvdh mentioned this pull request Oct 28, 2019
Closed
@babolivier babolivier mentioned this pull request Oct 29, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@richvdh richvdh

Assignees

@richvdh richvdh

@babolivier babolivier

Labels
None yet
Projects
Homeserver Task Board
  
Done
Milestone
No milestone
3 participants
@babolivier @erikjohnston @richvdh
You can’t perform that action at this time.