Package	Integration Status	Deploy Status	Coverage Status
services/fastapi-ml
services/common-ml
libs/ml
libs/mq
libs/telemetry

Python Monorepo

This is an example of how python structure in monorepo fashion, with the goal of demonstrating:

• Scopes: The folders act as scopes to make sure code artifacts are only visible when they should be. This allows to extract common tasks (e.g. building a C# solution) quickly and maintainers can easier reason about where the error lies.

• The One Version Rule (Atomic Commits). The principle guarantee that you can commit atomically to both of related projects simultaneously. There is no view of the repository where Project A is at Commit #1 but Project B is at Commit #2.

• Big pictures: With everything in one place there is no need to copy code between repositories or to look for infrastructure as code files and documentation.

• Good practice: A monorepo requires teams to work with each other. By merging code only with a MR, teams review each other’s code which breaks silos and improves code quality.

• Tooling required: for handling builds with shared libraries, CI-CD and Coverage Page.

Note

This project drew significant inspiration from dermidgen. However, I discovered that the existing methodology is outdated and doesn't support the use of Poetry within Docker, which is a major let down in the Python ecosystem. Here are the improvements I plan to implement compared to the original repository:

• Poetry playwell with docker container.
• Automated formatting and linting for the Python project.
• Docker-based CI/CD pipeline that allows sharing stages between libraries.
• Test and coverage page for whole monorepo including all subprojects.

Tooling

• Python3 ^3.8: (3.8.16 preferred)

• Poetry ^1.5: (1.5.1 preferred)

• Bash script for CI/CD builds, most conventions was inherited from author of FastAPI.

• Docker for containers.

This example demonstrates a local development approach that allows quick access to changes in shared libraries during the development and testing of individual services.

The tooling used enables seamless integration with CI/CD workflows, ensuring that any changes to shared libraries lead to automatic rebuilding and integration testing of dependent services without any delays. The key component contributing to this functionality is Poetry management.

Furthermore, the entire toolset works harmoniously even within a Docker container, making it well-suited for containerized environments.

General installation and working guide

Note: poetry work best with fresh conda environment, please create new env for this project

To install the monorepo system, please read and follow these steps:

1. Go to your project that you're working on, install environment and provide variable env if required

cd your/project/folder

pip install poetry
poetry install --with dev,lint,test # If you want to package the code, poetry install is enough

2. Before create MR, format code and check lint with these steps

cd your/project/folder

bash scripts/format.sh # Auto format code
bash scripts/lint.sh # Lint check

3. Run test in local by simple command

cd your/project/folder

bash scripts/test.sh