Improvements to CI pipeline, workflow organisation #5773
Conversation
eternaltyro
commented
May 8, 2023
eternaltyro
commented
- Revert to Python 3.9 <-- Python 3.10 for compatibility
- Breakout frontend and backend checks and builds as their own jobs
- Add a separate job for RDS database backup
- Whittle down the generic "build" job to temporary commands - to be removed later
- Removed Database snapshot cleanup step
- Introduce good workflows - initial draft Separate workflows for production, staging/dev, TM Assisted, and TeachOSM instances.
- Improve branch filters for workflow jobs: Move to workflow level filters.
eternaltyro
added
type: enhancement
eternaltyro
requested review from
dakotabenjamin,
d-rita,
AfiMaameDufie,
HelNershingThapa and
Aadesh-Baral
|
@eternaltyro the cleanup LGTM |
|
@eternaltyro I'd like to see how configuration is carried out in the circle-ci app. Can you merge develop into this branch as backend-code-check-black is blocking the entire execution and it was fixed in develop? |
eternaltyro
force-pushed
the
enhance/circleci-syntactic-sugar
branch
from
f2da09f
to
cdd4cc3
Compare
|
@Aadesh-Baral done |
eternaltyro
force-pushed
the
enhance/circleci-syntactic-sugar
branch
from
cdd4cc3
to
f6968c9
Compare
There was a problem hiding this comment.
Some processes (mostly build and deploy-backend) use the Python 3.7.14, whereas other jobs use Python 3.9. Can Python 3.9 be used for all jobs?
Additionally, since requirements.txt will eventually be removed, we might need to install backend dependencies via pdm or by generating requirements.txt with pdm export.
- Revert to Python 3.9 <-- Python 3.10 for compatibility - Breakout frontend and backend checks and builds as their own jobs - Add a separate job for RDS database backup - Whittle down the generic "build" job to temporary commands - to be removed later - Removed Database snapshot cleanup step - Introduce good workflows - initial draft Separate workflows for production, staging/dev, TM Assisted, and TeachOSM instances. - Improve branch filters for workflow jobs: Move to workflow level filters.
Setup a separate temporary database for running test cases in the CI pipeline.
- Move POSTGRES_TEST_DB envvar to primary image - Add PG USER, ENDPOINT vars to primary image - Simplify package installation step into separate runs - Break long commands - Remove the modern flow
eternaltyro
force-pushed
the
enhance/circleci-syntactic-sugar
branch
from
2234f8f
to
a83e0db
Compare
- Move from python:3.7.14-node to python:3.9.14-node, later versions use node v18 which is incompatible - Split tasks into their own run steps - Add pdm and disable virtualenv creation
|
@Aadesh-Baral thanks for the review. Comments below:
They will use Python 3.9.14 image for now. I can move to a later version of 3.9 or even 3.10 once the frontend PR are merged.
This is already included in the changes I made. |
The earlier PR #5184 switched the default TM_LOG_DIR from `logs` to `/home/appuser` which is not present in circleCI. Switched it out for /tmp/logs to make tests work.
|
Kudos, SonarCloud Quality Gate passed!
|
