Improve performance in the Python runtime module

[helderco]

This PR adds uv support, using it by default to install python packages.

The need for speed

The main motivation is performance. Some quick numbers:

Without uv:

• python -m venv → ≈2.08s
• pip install -r lock → ≈4s (cold)
• pip install -e sdk → ≈6.30s/3.56s (cold/warm)

With uv:

• uv venv → ≈0.03s
• uv pip install -r lock → ≈0.9s (cold)
• uv pip install -e sdk → ≈1.36s/0.76s (cold/warm)

These were not properly benchmarked, but the speed is real and uv does unlock a few things:

• It’s a single binary that joins virtualenv, pip and pip-tools so it’s cheap to install.
• Creating a virtual env is free now (but otherwise with pip, not creating a virtual env and just using the system install).
• No longer need to worry about setting up a cache volume for the virtual env with a cache key that would have to be unique for each module. 😶‍🌫️
• We can now generate a pinned requirements.lock file on dagger init --sdk=python by default.
• Installing from the lock file, with --no-deps is faster.
• The global cache in uv is instantly reusable for other modules, making it faster for new dagger init --sdk=python!

There were other efforts at improving performance beyond uv too:

• More intelligent pipeline setup to avoid cache invalidation.
• All API calls done concurrently once, in the beginning of execution.
• Insignificant difference between Codegen and RuntimeModule functions to reuse as much layer cache as possible.
• Installing dependencies first, sources last.
• Compiling bytecode during install, so it can be reused in the final exec /runtime step.
• Favoring mounts over copies.
• Smart and dynamic pipeline creation based on an initial .Entries() fetch.
• Simpler generation from template without doing an exec with sed.

Configurability

uv is still very new, but it’s already production ready and being adopted in the community like wild fire 🔥. But, in case there’s any conflict or issue, I took the opportunity to introduce configurability into the runtime module. There’s more configurations to come, but for now, these are available:

• New [tool.dagger] table in pyproject.toml to add dagger specific configuration:
  • use-uv = false to disable uv and use pip instead.
  • uv-version = "==0.1.27" to override the default pinned uv version
  • base-image = "mypython:3.12" to use a different base image¹ (e.g., add system packages, certificates, etc).
• .python-version file, as used by pyenv and rye, to use a different Python version:
  • Can also be used to pin it and avoid an automatic upgrade with a new version of dagger.
  • With rye, can be created with rye pin 3.12.
• requirements.lock file, if exists, will assume it has all dependencies in it.
  • This file will be created automatically for new modules.

For more advanced use cases, the pipelines have been more granularly composed to allow reusability in a custom SDK module that uses this one as a dependency, in order to customize some part of the process. Without this, you’d had to duplicate most of the code.

Note that adding configurability adds an overhead, but I feel it's necessary. And there's a flag to disable it if doing a custom SDK use case.

Default template

The default template has a couple of changes:

• In pyproject.toml:
  • Included dependencies = [] to make it clearer how to add custom python dependencies
  • Added the hatchling build backed for better defaults
• Code created in src/main/__init__.py instead of src/main.py.
  • Fixes Python modules: dagger init should create code from template in src/main/__init__.py instead of src/main.py #6863
  • Hatchling doesn’t find it if not a package, otherwise it requires an additional setting to tell it.
• A requirements.lock file is created, with pinned versions for SDK dependencies at the time that the new module is created.

Reproducibility

Summarizing from above, you have stronger guarantees for a reproducible environment, with:

• The requirements.lock file
• The pinned python version in .python-version
• The pinned uv version in uv-version config
• Hashes in the lock file for verifying the downloaded packages
  • Pip checks the hashes. uv doesn’t yet, but will.

What's next?

There's more follow-up configuration and performance gains but need to do more profiling and tracing will help with that.

Footnotes

1. Use at own risk since we can’t guarantee support for custom deviations. For best compatibility extend from the default image that dagger uses: python:3.11-slim. May need to watch out when the default image is updated. ↩

[gerhard]

Love the PR description - it's world class 🥇🚀

[helderco]

Added missing tests.