Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MBS-10322: Drop Fabric #1337

Merged
merged 8 commits into from Jan 27, 2020
Merged

MBS-10322: Drop Fabric #1337

merged 8 commits into from Jan 27, 2020

Conversation

yvanzo
Copy link
Contributor

@yvanzo yvanzo commented Dec 20, 2019 •
edited

Summary

Resolve MBS-10322: Port fabfile

Problem

fabfile.py was written for Fabric 1/Python 2 which is retiring in 2020. It was the only piece of Python in this repository, that doesn’t favor Fabric 2/Python 3 which is not 100% feature parity anyway. It was missing hosts discovery for deployment.

Solution

Rewriting tasks as independent scripts, with a few differences:

  • Scripts are self-documented (with HELP string)
  • More errors are handled and have detailed messages
  • Target hosts are deduced from docker server configs

TODO

  • Adding PROD_STANDY to DBDefs example/default
  • Adding MB_POT_DB to extract_pot_db
  • Completing update_potto check these parameters

@yvanzo yvanzo changed the title Drop Fabric MBS-10322: Drop Fabric Dec 31, 2019
@yvanzo
Copy link
Contributor Author

yvanzo commented Jan 8, 2020 •
edited

  • Updated to support Mac OS X (bash 3 only, grep -E, sed -E)
  • Linted using shellcheck

@yvanzo yvanzo added this to the Tiburón Mamá milestone Jan 8, 2020
@yvanzo yvanzo marked this pull request as ready for review January 11, 2020 17:27
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Jan 11, 2020
@yvanzo
Set PROD_STANDY to read the production database
e6616f3 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
mwiencek
mwiencek requested changes Jan 13, 2020
View reviewed changes
po/update_translations.sh Outdated Show resolved Hide resolved
script/update_containers.sh Outdated Show resolved Hide resolved
script/update_containers.sh Outdated Show resolved Hide resolved
script/update_containers.sh Outdated Show resolved Hide resolved
script/database_exists Show resolved Hide resolved
po/update_pot.sh Outdated Show resolved Hide resolved
mwiencek
mwiencek approved these changes Jan 23, 2020
View reviewed changes
Copy link
Member

@mwiencek mwiencek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#1337 h4x

@yvanzo
Copy link
Contributor Author

yvanzo commented Jan 23, 2020

(Just rebased, no code change.)

@yvanzo yvanzo removed the request for review from reosarevok January 27, 2020 20:25
@yvanzo
Copy link
Contributor Author

yvanzo commented Jan 27, 2020

Expanded update_containers.sh doc thanks to @reosarevok’s feedback.

yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Jan 27, 2020
@yvanzo
Set PROD_STANDY to read the production database
2582e43 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
yvanzo and others added 8 commits January 27, 2020 22:02
@yvanzo
MBS-10322: Port 'fab translations' to shell script
954a954 
Replace 'translations' Fabric task with '--commit' option to
'po/update_translations.sh' script.  Small changes in behavior:

- Prevent incidentally committing pre-existing local changes
- No more ask to confirm commit message before committing
- Show commit statistics if any, print a notice otherwise
@yvanzo
MBS-10322: Port 'fab tag' to shell script
66cedb9 
Replace 'tag' Fabric task with new 'script/tag.sh'
@yvanzo
MBS-10322: Port 'fab deploy' to shell script
25a5afa 
Replace 'deploy' Fabric task with new 'script/update_containers.sh'.

Improvements:

- Automatically find hosts to be updated from docker server configs
- Less (1 instead of 3) ssh connections to each host to be updated
- Sleep more (30s instead of 15s) after each updated host
- Update services using script maintained with docker server configs
@yvanzo
Remove obsolete RAWDATA/RAWDATA_SYSTEM comment
d14d8ef
@yvanzo
Fix testing database exists even if remote
77ecc53 
The script 'database_exists' assumes that all databases are on the same
host by using the database configuration 'SYSTEM' to query the table
'pg_database'.  However, this cannot be assumed with the new database
configuration 'PROD_STANDBY' which specifically points to another host.

This patch makes the script 'database_exists' use the tested database
configuration to query the table 'information_schema_catalog_name'.

Note: 'pg_database' may not be readable by user other than 'postgres'.
@mwiencek @yvanzo
Add MB_POT_DB env var to extract_pot_db
627e6f0 
This controls which database in lib/DBDefs.pm is used to extract strings
from, defaulting to READWRITE as before.
@yvanzo
MBS-10322: Port 'fab pot' to shell script
bb2389b 
Replace 'pot' Fabric task with new 'po/update_pot.sh' script and
'--commit' option. It is a derivative of 'po/update_translations.sh'
and has the same behavior:

- Prevent incidentally committing pre-existing local changes
- No more ask to confirm commit message before committing
- Show commit statistics if any, print a notice otherwise
- Test (remote) database connection before querying it

Plus, by Michael: Set MB_POT_DB=PROD_STANDBY in update_pot.sh

The intent of this script is to update pot files using the production
database, so the developer is now required to have a PROD_STANDBY
database configured in lib/DBDefs.pm which does that, via SSH tunnel or
otherwise.
@yvanzo
Replace sed/here-docs with repeated echo
b36ad49
reosarevok
reosarevok approved these changes Jan 27, 2020
View reviewed changes
Copy link
Member

@reosarevok reosarevok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested for a test release, seemed to work fine.

@yvanzo yvanzo merged commit 2dc0354 into metabrainz:master Jan 27, 2020
@yvanzo yvanzo deleted the drop-fabric branch January 27, 2020 22:22
yvanzo added a commit to yvanzo/musicbrainz-server that referenced this pull request Jan 29, 2020
@yvanzo
Require Locale::PO module in develop feature
bb712f5 
Since metabrainz#1337 replaced Fabric, `extract_pot_db` that requires Locale::PO
is no longer run in production, it is run in development setup instead.

This patch makes Locale::PO to be installed using cpanm option
`--with-develop` instead of `--with-feature=production`.
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Feb 14, 2020
@yvanzo
Set PROD_STANDY to read the production database
4b25175 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
@yvanzo yvanzo mentioned this pull request Feb 19, 2020
Merged
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Feb 20, 2020
@yvanzo
Set PROD_STANDY to read the production database
1ee5985 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Mar 19, 2020
@yvanzo
Set PROD_STANDY to read the production database
17ebeaa 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Mar 19, 2020
@yvanzo
Set PROD_STANDY to read the production database
5bd3675 
- 172.17.0.1 is the default gateway of Docker bridge network
- 54321 is an arbitrary port for an SSH tunnel that can be created with:

    ssh -L 172.17.0.1:54321:localhost:65400 pink -N # production slave

It depends on metabrainz/musicbrainz-server#1337
yvanzo added a commit to yvanzo/musicbrainz-docker that referenced this pull request Mar 20, 2020
@yvanzo
Set PROD_STANDY to read the production database
56501b7 
This is a database definition for use by official maintainers only.

Note that standby/slave/secondary database is read-only anyway, thus
setting user to `musicbrainz` or musicbrainz_ro` makes no difference.
It is set to `musicbrainz_ro` as a precaution just in case the SSH
tunnel mistakenly points to a database other than the standby prod.

It relies on metabrainz/musicbrainz-server#1337
yvanzo added a commit to metabrainz/musicbrainz-docker that referenced this pull request Mar 24, 2020
@yvanzo
Set PROD_STANDY to read the production database
a4c27d1 
This is a database definition for use by official maintainers only.

Note that standby/slave/secondary database is read-only anyway, thus
setting user to `musicbrainz` or musicbrainz_ro` makes no difference.
It is set to `musicbrainz_ro` as a precaution just in case the SSH
tunnel mistakenly points to a database other than the standby prod.

It relies on metabrainz/musicbrainz-server#1337
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mwiencek mwiencek mwiencek approved these changes

@reosarevok reosarevok reosarevok approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Tiburón Papá
3 participants
@yvanzo @mwiencek @reosarevok