-
Notifications
You must be signed in to change notification settings - Fork 51
docs: fix stale versions, log paths, and local dev setup #511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
1f932bb
docs: add debugging section to deployment.md
lgelauff 85d0be4
docs: fix MariaDB setup steps in deployment.md
lgelauff 40b157d
docs: note that beta SQLite file may coexist with MariaDB setup
lgelauff 12c56bb
docs(deployment): update Python version from 3.9 to 3.11
lgelauff 4f94d4e
docs(deployment): update Node version from 18 to 20
lgelauff 5a55438
docs(deployment): standardize log paths to logs/ subdirectory
lgelauff aa44c12
docs(dev): clarify local login flow in debug mode
lgelauff eb31a3f
docs(dev): replace hardcoded rdb.py line number with stable reference
lgelauff 707ee29
docs: add CONTRIBUTING.md
lgelauff f9c2b93
Update CONTRIBUTING.md with new contributor guidelines
lgelauff File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Contributing to Montage | ||
|
|
||
| Montage is a small project maintained by volunteers. Contributions are welcome — please read this before opening a PR. | ||
|
|
||
| We follow the [Wikimedia technical spaces Code of Conduct](https://www.mediawiki.org/wiki/Code_of_Conduct). | ||
|
|
||
| ## Getting started | ||
|
|
||
| Check the [open issues](https://github.com/hatnote/montage/issues) for something to pick up. Issues tagged [`good-first-issue`](https://github.com/hatnote/montage/issues?q=is%3Aissue+is%3Aopen+label%3Agood-first-issue) are a good starting point. | ||
|
|
||
| Before writing any code, leave a comment on the issue to let maintainers know you're working on it and to confirm the approach. If no issue exists for what you want to do, open one first. **Do not open a PR without a corresponding issue that has received a response from a maintainer** — especially if you are a new contributor. This avoids wasted effort on both sides. | ||
|
|
||
| See [dev.md](dev.md) for local setup instructions. | ||
|
|
||
| ## Before opening a PR | ||
|
|
||
| Follow the setup instructions in [dev.md](dev.md), including installing pre-commit hooks and running tests locally. Make sure tests pass before opening a PR. | ||
|
|
||
| ## Pull request expectations | ||
|
|
||
| - CI must pass | ||
| - One focused change per PR — link the issue it addresses (`Fixes #NNN`) | ||
| - If your PR adds or changes database columns, include the SQL migration and flag it in the PR description | ||
| - Do not commit screenshots, generated files, or large binaries | ||
|
|
||
| ## Code quality | ||
|
|
||
| - Follow existing patterns in the file you're editing | ||
| - Backend: read `montage/rdb.py` before writing queries — column names are authoritative there. Note that `.filter()` and `.order_by()` return new objects — always reassign | ||
| - Backend: write code that works with both MySQL (production) and SQLite (tests) — no MySQL-only syntax | ||
| - Frontend: use Composition API with `<script setup>` — no Options API | ||
| - All user-facing strings must go in `frontend/src/i18n/en.json` and use `t('key')` — TranslateWiki handles other locales, do not edit them manually | ||
|
|
||
| ## When to open a PR for review | ||
|
|
||
| Only open a PR for review when you consider it ready to merge. If your work is still in progress, open it as a **draft PR** — maintainers will not review draft PRs. | ||
|
|
||
| A PR is ready for review when: | ||
| - It does one thing and does it completely | ||
| - CI passes | ||
| - You have tested it yourself and are confident it works | ||
| - You would be comfortable if it were merged as-is | ||
|
|
||
| Do not open a PR to ask for direction or feedback on an approach — use the issue for that conversation first. | ||
|
|
||
| When you are a new contributor to this codebase (e.g. not part of the maintainer team) we kindly ask you to make a screen recording of the platform with human voice-over that reproduces the issue locally, and shows the change after the implemented changes. Please also make sure to be explicit about your testing approach and the logic used. | ||
|
|
||
| ## Deployment | ||
|
|
||
| Only maintainers have access to the three Toolforge environments: | ||
|
|
||
| | Environment | Purpose | | ||
| |-------------|---------| | ||
| | montage-dev | Flexible test environment — maintainers can deploy any branch here | | ||
| | montage-beta | Tracks `master` — used by volunteers to catch bugs before production | | ||
| | montage-prod | Stable production — promoted manually when beta is stable | | ||
|
|
||
| Contributors do not need Toolforge access. Push your branch to GitHub and open a PR against `master`. A maintainer can then deploy your branch to montage-dev to test it — either in isolation or combined with other PRs to check interaction effects. | ||
|
|
||
| ## Questions | ||
|
|
||
| Leave a comment on the relevant GitHub issue, or reach out on [Commons talk:Montage](https://commons.wikimedia.org/wiki/Commons_talk:Montage). | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,7 +24,7 @@ git clone https://github.com/hatnote/montage.git src | |
|
|
||
| ##### 4. Make the frontend build | ||
| ```bash | ||
| toolforge webservice node18 shell -m 2G | ||
| toolforge webservice node20 shell -m 2G | ||
| cd $HOME/www/python/src/frontend | ||
| npm install | ||
| npm run toolforge:build | ||
|
|
@@ -33,17 +33,25 @@ exit | |
| This will build the vue prod bundle and put in backend's `template` and `static` directory. | ||
|
|
||
| ##### 5. Create your database | ||
| * Get the user name of database (`cat ~/replica.my.cnf`) | ||
| * Open up MariaDB with `sql local` | ||
| * Create a [Toolforge user database](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases) (`create database <user>__<db name>;`), and remember the name for the config | ||
| * Get the username and password from `cat ~/replica.my.cnf` | ||
| * Connect to MariaDB: | ||
| ```bash | ||
| mariadb --defaults-file=~/replica.my.cnf -h tools.db.svc.wikimedia.cloud | ||
| ``` | ||
| * Create a [Toolforge user database](https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#User_databases) with `utf8mb4` charset, and remember the name for the config: | ||
| ```sql | ||
| CREATE DATABASE `<user>__<db name>` DEFAULT CHARACTER SET utf8mb4 DEFAULT COLLATE utf8mb4_unicode_ci; | ||
| EXIT; | ||
| ``` | ||
|
|
||
| ##### 6. Set up the montage config | ||
| * Make a copy of `config.default.yaml` for your environment | ||
| * You may need to update `USER_ENV_MAP` in `montage/utils.py` if you need to add a new environment | ||
| * Add the `oauth_consumer_token` and `oauth_secret_token` | ||
| * Add a `cookie_secret: <your random secret>` | ||
| * Add the `db_url` with your user database name, and the password from `~/.replica.my.cnf` | ||
| * The format is: `mysql://<user>:<password>@tools.labsdb/<db name>?charset=utf8` | ||
| * The format is: `mysql+pymysql://<user>:<password>@tools.db.svc.wikimedia.cloud/<db name>?charset=utf8mb4` | ||
| * Create the log directory: `mkdir -p /data/project/<project>/logs` | ||
| * Add `api_log_path: /data/project/<project>/logs/montage_api.log` | ||
| * Add `replay_log_path: /data/project/<project>/logs/montage_replay.log` | ||
| * Add `labs_db: True` | ||
|
|
@@ -53,20 +61,32 @@ This will build the vue prod bundle and put in backend's `template` and `static` | |
|
|
||
| ##### 7. Creating a virtual environment | ||
| ```bash | ||
| toolforge webservice python3.9 shell | ||
| toolforge webservice python3.11 shell | ||
| python3 -m venv $HOME/www/python/venv | ||
| source $HOME/www/python/venv/bin/activate | ||
| pip install --upgrade pip wheel | ||
| pip install -r $HOME/www/python/src/requirements.txt | ||
| exit | ||
| ``` | ||
|
|
||
| ##### 8. Start the backend service | ||
| ##### 8. Initialise the database schema | ||
| ```bash | ||
| toolforge webservice python3.9 start | ||
| cd $HOME/www/python/src | ||
| source $HOME/www/python/venv/bin/activate | ||
| python3 montage/create_schema.py | ||
| ``` | ||
|
|
||
| ##### 9. Testing of deployment | ||
| If this is an upgrade of an existing deployment (not a fresh install), run the migration SQL instead: | ||
| ```bash | ||
| mariadb --defaults-file=~/replica.my.cnf -h tools.db.svc.wikimedia.cloud <db name> < tools/migrate_prod_db.sql | ||
| ``` | ||
|
|
||
| ##### 9. Start the backend service | ||
| ```bash | ||
| toolforge webservice python3.11 start | ||
| ``` | ||
|
|
||
| ##### 10. Testing of deployment | ||
| * Visit /meta to see the API. Example: https://montage-beta.toolforge.org/meta/ | ||
| * In the top section, you should see that the service was restarted in the last few seconds/minutes. | ||
|
|
||
|
|
@@ -100,7 +120,7 @@ git pull | |
|
|
||
| ##### 4. Make the frontend build | ||
| ```bash | ||
| toolforge webservice node18 shell -m 2G | ||
| toolforge webservice node20 shell -m 2G | ||
| cd $HOME/www/python/src/frontend | ||
| npm install | ||
| npm run toolforge:build | ||
|
|
@@ -110,17 +130,76 @@ exit | |
| ##### 5. (Optional) Install python packages | ||
| If you added new python packages in changes then you have to install them in pod. | ||
| ```bash | ||
| toolforge webservice python3.9 shell | ||
| toolforge webservice python3.11 shell | ||
| source $HOME/www/python/venv/bin/activate | ||
| pip install -r $HOME/www/python/src/requirements.txt | ||
| exit | ||
| ``` | ||
|
|
||
| ##### 8. Restart the backend service | ||
| ```bash | ||
| toolforge webservice python3.9 restart | ||
| toolforge webservice python3.11 restart | ||
| ``` | ||
|
|
||
| ##### 9. Testing of deployment | ||
| * Visit /meta to see the API. Example: https://montage-beta.toolforge.org/meta/ | ||
| * In the top section, you should see that the service was restarted in the last few seconds/minutes. | ||
| * In the top section, you should see that the service was restarted in the last few seconds/minutes. | ||
|
|
||
|
|
||
| --- | ||
|
|
||
|
|
||
| ## Debugging | ||
|
|
||
| ##### Viewing logs | ||
|
|
||
| The uwsgi log is at: | ||
| ```bash | ||
| tail -50 /data/project/montage-beta/uwsgi.log | ||
| ``` | ||
|
|
||
| Note: the log directory is `/data/project/<project>/logs/`, not inside `src/`. | ||
|
|
||
| ##### Running Python / pip commands | ||
|
|
||
| Always run `pip install` and Python diagnostics inside the webservice shell, not the bastion shell. The two environments use different venvs: | ||
|
|
||
| ```bash | ||
| toolforge webservice python3.11 shell | ||
| # venv is activated automatically | ||
| pip install -r ~/www/python/src/requirements.txt | ||
| python3 -c "import montage.app" | ||
| exit | ||
| ``` | ||
|
|
||
| Running `pip` on the bastion shell installs to a different venv and will not affect the running service. | ||
|
|
||
| ##### Restarting the service | ||
|
|
||
| ```bash | ||
| toolforge webservice python3.11 restart | ||
| ``` | ||
|
|
||
| ##### Inspecting the MariaDB database | ||
|
|
||
| Always pass `-h tools.db.svc.wikimedia.cloud` explicitly — there is no local socket on the Toolforge bastion: | ||
|
|
||
| ```bash | ||
| mariadb --defaults-file=~/replica.my.cnf -h tools.db.svc.wikimedia.cloud <db name> | ||
| ``` | ||
|
|
||
| Example queries: | ||
| ```sql | ||
| SELECT COUNT(*) FROM entries; | ||
| DESCRIBE entries; | ||
| ``` | ||
|
|
||
| ##### Inspecting the SQLite database (legacy / dev only) | ||
|
|
||
| Note: montage-beta originally used SQLite and the file (`tmp_montage.db`) may still exist alongside the MariaDB setup. It is no longer used by the running service once the config switches to `mysql+pymysql://`. | ||
|
|
||
| There is no `sqlite3` CLI on Toolforge. Use Python instead: | ||
|
|
||
| ```bash | ||
| python3 -c 'import sqlite3; c=sqlite3.connect("/data/project/montage-beta/www/python/src/tmp_montage.db"); print(c.execute("SELECT COUNT(*) FROM entries").fetchone())' | ||
|
Comment on lines
+199
to
+204
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. if we don't need sqlite, we won't need this caveat :) |
||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that Docker support has gotten pretty good, I don't know if sqlite support is strictly necessary, tbh.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is that also true locally? I like the simpler approach where possible, but would be nice indeed to split that into a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah docker is just for local, so docker compose would be the best way to get production-like env locally now.