-
-
Notifications
You must be signed in to change notification settings - Fork 268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improvements to CI pipeline, workflow organisation #5773
Conversation
eternaltyro
commented
May 8, 2023
- Revert to Python 3.9 <-- Python 3.10 for compatibility
- Breakout frontend and backend checks and builds as their own jobs
- Add a separate job for RDS database backup
- Whittle down the generic "build" job to temporary commands - to be removed later
- Removed Database snapshot cleanup step
- Introduce good workflows - initial draft Separate workflows for production, staging/dev, TM Assisted, and TeachOSM instances.
- Improve branch filters for workflow jobs: Move to workflow level filters.
@eternaltyro the cleanup LGTM |
@eternaltyro I'd like to see how configuration is carried out in the circle-ci app. Can you merge develop into this branch as backend-code-check-black is blocking the entire execution and it was fixed in develop? |
f2da09f
to
cdd4cc3
Compare
@Aadesh-Baral done |
cdd4cc3
to
f6968c9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some processes (mostly build
and deploy-backend
) use the Python 3.7.14, whereas other jobs use Python 3.9. Can Python 3.9 be used for all jobs?
Additionally, since requirements.txt
will eventually be removed, we might need to install backend dependencies via pdm or by generating requirements.txt
with pdm export.
- Revert to Python 3.9 <-- Python 3.10 for compatibility - Breakout frontend and backend checks and builds as their own jobs - Add a separate job for RDS database backup - Whittle down the generic "build" job to temporary commands - to be removed later - Removed Database snapshot cleanup step - Introduce good workflows - initial draft Separate workflows for production, staging/dev, TM Assisted, and TeachOSM instances. - Improve branch filters for workflow jobs: Move to workflow level filters.
Setup a separate temporary database for running test cases in the CI pipeline.
- Move POSTGRES_TEST_DB envvar to primary image - Add PG USER, ENDPOINT vars to primary image - Simplify package installation step into separate runs - Break long commands - Remove the modern flow
2234f8f
to
a83e0db
Compare
- Move from python:3.7.14-node to python:3.9.14-node, later versions use node v18 which is incompatible - Split tasks into their own run steps - Add pdm and disable virtualenv creation
@Aadesh-Baral thanks for the review. Comments below:
They will use Python 3.9.14 image for now. I can move to a later version of 3.9 or even 3.10 once the frontend PR are merged.
This is already included in the changes I made. |
The earlier PR #5184 switched the default TM_LOG_DIR from `logs` to `/home/appuser` which is not present in circleCI. Switched it out for /tmp/logs to make tests work.
Kudos, SonarCloud Quality Gate passed! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM