Python mono-repo using UV, Pex, and Earthly

This repository contains an example of a monorepo setup using uv and pex for building python executables. The docker builds are handled with Earthly, but could be easily replaced with plain Dockerfiles.

Motivation

UV supports the dev part of python mono-repositories through workspaces.However, when it comes to shipping our code we want to bundle our code in such a way that each docker image only contains the necessary dependencies.

While the related uv proposal for uv bundle is still in the future this repository provides a recipe for how to bundle uv workspaces into executables that can be easily copied into docker images.

How does it work?

The magic of the entire approach relies on two commands:

1. Compile the dependencies of a workspace, e.g. ./server:

uv pip compile pyproject.toml -o dist/requirements.txt

2. Build the pex:

uvx pex \
-r dist/requirements.txt \
-o dist/bin.pex \
-e main \
--python-shebang '#!/usr/bin/env python3' \
--sources-dir=. \
# (optional) Package a full python distribution with the executable 
--scie eager \
--scie-pbs-stripped

The resulting pex file will also include a full python interpreter and is only portable given the same architecture and exec format meaning that if you built this on MacOS it will not work on linux.

Once we have the pex we can easily copy it into a docker image that is custom for each app. Here we do it with earthly, even though you could conceivably also have one Dockerfile per package.

The entire repositories build process can be run with make build-images.

The example repository

The repository consists of a library lib/format that is consumed by two different targets server and cli that each bring their own additional dependencies.

Building

To produce the pex for either target you can run:

scripts/build_pex.sh server

this will create a dist/bin.pex file in the server package folder. Now we can create the image by running earthly +build from that same directory.

Testing

Each workspace maintains its own tests that can be run with uv run pytest. To run all tests you can run the scripts/run_tests.sh script.

Cross-compilation

By default, pex files are not portable between operating systems and architectures. So a pex generated by running the scripts/build_pex.sh command on MacOS will not work on linux amd64.

There are two ways of addressing this:

1. Build the pex on the target platform using your CI (see .github/workflows/release.yaml)
2. Build the pex from within a docker container

The second approach is implemented by running the following script like so:

./scripts/build_pex_in_docker.sh server linux_amd64

Note that this will mount the entire repository into the build container which can be error-prone.

Comparison with other tools

I took this table from here, but it perfectly echoes my sentiment.

While pants (or if you are very experienced Bazel) is the ultimate solution, it has a very steep learning curve and is thus hard to adopt for small to medium-sized teams.

On the other end of the spectrum poetry provides a good development experience, but it relies on many non-standard features and lacks good support for mono-repositories. UV being the long awaited messiah of the python eco-system not only supports mono-repos via workspaces, but also provides a global lock file and top performance.

	Poetry	UV	Pants
Simplicity	😃	😃	😭
Single lock file	😭	😍	🚀
CI/CD	🤨	🤨	🙂
Docker builds	🤔	😀	😭➡️🙂
Speed	🤮	🥰	😌
Caching	🤷	🤷	🥰
Reproducability	🤷	🤷	🥰
Verdict	Woefully inadequate	Happy medium	Too complicated

Overall, the recipe in this repository lacks many important aspects that are implemented by a "real" mono-repository build tool such as Pants or Bazel. Instead, it provides a low-lift first step towards the right direction.

In fact, once you have this setup pants is not too far off since it also uses a global lock file and packages all python executables using pex.