Monorepo setup using Poetry for different classes of projects

[zbitouzakaria]

Hi everyone!

I'm working on setting up a monorepo using Pants and Poetry to manage multiple classes of Python projects, and I would love to gather your feedback on the choices made. We have specific requirements to factor in:

Requirements and Intent

We have two main classes of projects:

• release projects: These are the ones that have deployed artifacts running on our cloud. We want to ease the development of new services by having them neatly organized in a single repo, allowing easy composition of libs for shared usage. We can use visibility rules to ensure a service cannot depend on another service and these can be subdivided in multiple "universes" of dependencies, through resolves. Classic Pants use case.
• sandbox projects: These are of the exploratory kind (e.g some data science projects). We want them isolated since they might use packages with unstable APIs, or just fill a quick need for insights. We don't put them in a separate repo because we believe we can ease the bootstrap of such projects as they might need specific cloud infrastructure, and they could benefit from using libs from our release-* resolves. Usefulness will be limited if they have to comply to our whole universe of dependencies, so they need to be able to pick up just a lib and it's direct dependencies.

I started from poetry#6850 and worked from there.

Monorepo Structure

.
├── lockfiles
│   └── pytest.lock
├── tf
│   ├── modules/ # Common terraform modules
├── py
│   ├── sandbox/
│   │   ├── projects/ # Classic poetry projects
│   │   │   ├── [...]
│   ├── release-core/ # Core resolve
│   │   ├── .venv/ # Poetry virtualenv for the resolve, can be useful for development
│   │   ├── 3rdparty
│   │   │   └── release-core.lock
│   │   ├── libs
│   │   │   └── lib1
│   │   │       ├── org-namespace
│   │   │       │   └── lib/
│   │   │       ├── BUILD
│   │   │       └── pyproject.toml
│   │   ├── projects
│   │   │   ├── app
│   │   │   │   ├── org-namespace
│   │   │   │   │   └── app/
│   │   │   │   ├── terraform/ # Specific terraform modules
│   │   │   │   ├── BUILD
│   │   │   │   └── pyproject.toml
│   │   │   └── cli
│   │   │       ├── org-namespace
│   │   │       │   └── cli/
│   │   │       ├── terraform/ 
│   │   │       ├── BUILD
│   │   │       └── pyproject.toml
│   │   ├── BUILD
│   │   ├── Dockerfile
│   │   ├── Makefile
│   │   ├── poetry.lock
│   │   ├── poetry.toml
│   │   └── pyproject.toml
│   ├── release-pytorch/ # Similar structure to release-core/ (can use libs from release-core/)
│       └── [...]
├── .gitignore
├── pants.toml
└── README.md

The file py/projects/resolve-core/pyproject.toml uses path dependencies references to the relevant projects, and each project describes it's own set of dependencies. This way they can be worked on either in isolation using poetry or using pants goals (the CI ensures pants generate-lockfiles :: doesn't break)

[tool.poetry]
name = "resolve-core"
version = "0.1.0"

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.group.app.dependencies]
namespace-app = { path = "projects/app", develop = true}

[tool.poetry.group.cli.dependencies]
namespace-cli = { path = "projects/cli", develop = true}

[tool.poetry.group.lib.dependencies]
namespace-lib = { path = "libs/lib1", develop = true}

I've tried to implement this (need to do some refactoring then I can share a working repo) and wanted to share a few insights (also had a few questions):

• pants tailor :: I didn't have to modify any of the generated BUILD files to make this work (yet).
• In the project I used as a starting point, the python version requirements was declared under the group [tool.poetry.group.main.dependencies], which poetry seems to support. pants didn't though and understood python as a third-party dependency e.g.

# In py/release-core/pyproject.toml

# This fails when running `pants generate-lockfiles ::`
[tool.poetry.group.main.dependencies]
python = "^3.10"

# This works
[tool.poetry.dependencies]
python = "^3.10"

• pants dependecies /path/to/file fails to detect the dependency when trying to import a relative module using the absolute syntax, have any idea why ? e.g:

# app structure
# .
# ├── org_namespace
# │   └── app
# │       ├── __init__.py
# │       ├── BUILD
# │       ├── main.py
# │       └── test_main.py
# ├── BUILD
# └── pyproject.toml

# In py/release-core/projects/app/namespace/app/test_main.py
from .main import main # Successfully infers the dependency on main.py
from org_namespace.app.main import main # Code works but fails to infer the dependency

• If resolve-pytorch can use libs from resolve-core, would it make sense to put those libs elsewhere ?
• This approach allows us to easily add a dependency using poetry add and leverage it's dependency resolution process. We'll still need to pants generate-lockfiles :: to integrate it into our resolve. But the cool thing is that we can work either with a poetry virtualenv or directly with Pants goals.
• A sandbox project can add a relative path dependency to a lib and get going with poetry, which is nice.

I'm looking for feedback on potential issues with this build setup, any insight or suggestion would be highly appreciated.

Thank you for taking the time to review this!

[huonw]

(Separate comments for easier threading/replies)

pants dependecies /path/to/file fails to detect the dependency when trying to import a relative module using the absolute syntax, have any idea why ? e.g:

This suggests that the "source roots" aren't configured appropriately, so pants isn't aware that org_namespace is the entry point. I'm not sure of the perfect way to configure them based on your directory tree: potentially marker_filenames = ["pyproject.toml"]?

[zbitouzakaria]

Yep you nailed this one 🙏🏼

Ended up defining my source roots this way:

[source]
root_patterns = [
  "/py/release-*/projects/*",
  "/py/libs/*",
]

➜  python-poc-monorepo git:(main) pants roots
py/libs/lib1
py/release-core/projects/app
py/release-core/projects/cli
py/release-django4/projects/django-project

This way it can catch any project, and pants successfully understood all the namespace entries

[huonw]

(Separate comments for easier threading/replies)

This approach allows us to easily add a dependency using poetry add and leverage it's dependency resolution process. We'll still need to pants generate-lockfiles :: to integrate it into our resolve. But the cool thing is that we can work either with a poetry virtualenv or directly with Pants goals.

We do something similar: use poetry to actually manage adding/removing our dependencies, but pants to run our code, and it works fine. We have a test that runs in CI for checking our pants lock file is up to date with our poetry one, so that we can't forget pants generate-lockfiles step and end up with poetry and pants out of sync.

[zbitouzakaria]

Yeah but this way don't you need to also specify the version of the lib ?

[zbitouzakaria]

Ohh I see! I've tried it and you don't need to

[kaos]

Exactly, you can provide any contraints you like, or none. The exact version used will be captured in the lockfile.

[huonw]

Just a warning that has caught us, and why we use poetry in addition to pants: pants currently doesn't natively support "minimal" lockfiles updates, adding a new dep and regenerating the lockfile will regenerate the whole thing, potentially upgrading everything. #15704

In my work repo; that's undesirable, so we have poetry (which does support this better) manage the deps, and export a requirements.txt with everything pinned to feed into pants.

If this doesn't worry you, then you can ignore this.

[zbitouzakaria]

Nice to know, thanks ! This might be undesirable in our case too, I'm not so sure.. It will definitely make it harder to spot if an upgrade or another change broke the build, but on the other hand it will enforce upgrading as we go (and I'm guessing we can still use constraints files to pin the culprits, which is basically what your requirement.txt file is)

[huonw]

(Separate comments for easier threading/replies)

If resolve-pytorch can use libs from resolve-core, would it make sense to put those libs elsewhere ?

Potentially... you may need to package these up as python_distributions so that they can cross (pants) resolves, but I am not sure. AIUI, pants generally works more smoothly with one "big" resolve for all code that imports each other, so that the versions of PyPI deps are pinned across the whole repo, but python_distribution targets may help with this. I don't have personal experience though.

[zbitouzakaria]

Yes for sure, we'll be trying to have one "big" resolve. The thing is I can easily see cases where this mode will fall short and wanted to make sure it's easy to work with multiple resolves. Splitting the resolves per folder would allow for corresponding pants and poetry lock files to live under the same path, and ease future checks of consistency between the two.

I've manage to step up a working version with multiple resolves and it now works like a charm. I didn't need to setup python_distribution, I've used

poetry_requirements(
    name="poetry",
    resolve=parametrize("release-core", "release-django4"),
)

in the BUILD file of the lib. This way I can use it with both resolves fine (it also runs the tests for the libs in both resolves every time)

(My original example was using torch, switched to django because torch and poetry don't get along really well 😅 I might revert to a classic setup.py project if I have to use pytorch)

[zbitouzakaria]

Sharing the working code in case anyone is interested in a similar approach: https://github.com/zbitouzakaria/python-monorepo-pants-poetry, I've had a lot of fun trying to set this up and I'm already blown away by the capabilities of Pants 👏🏻😄

[zbitouzakaria]

Actually, one last question 😛

To completely ignore the projects under sandbox, I've tried multiple approaches:

# In pants.toml

[GLOBAL]

# We could use pants_ignore to ignore sandbox projects but I'm afraid default values
# could change and we'll be introducting subtle bugs in Pants' behaviour
# pants_ignore = [
#   # Default pants_ignore values
#   ".*/",
#   "/dist/",
#   "__pycache__",
#   # Ignore all sandbox projects
#   "py/sandbox/"
# ]

# Are these two options sufficiant ?

build_ignore = [
  # Ignore all sandbox projects
  "py/sandbox/**"
]

[tailor]
ignore_paths = [
  # Ignore all sandbox projects
  "py/sandbox/**"
]

Any idea how to do this properly ?

[huonw]

List options and their defaults are subtle: the right way to do this is pants_ignore.add = ["py/sandbox"], which will append to the default value instead of overwriting, due to the add.

This is not obvious (I had missed it too, as I adopted pants in our repo). There's some more info in https://www.pantsbuild.org/docs/options "add/remove syntax in .toml files" too.

[benjyw]

I'm curious - what are you using Poetry for in this repo? What does it provide in addition to Pants?