Rye + Docker

[santigandolfo]

Has anybody tried creating a multistage Dockerfile that uses Rye?

Im trying to do so, but I don't see a way to either export a requirements.txt file (as suggested in FastAPI's poetry docs https://fastapi.tiangolo.com/deployment/docker/) or to directly copy the packages (as suggested in PDMs docs https://pdm.fming.dev/latest/usage/advanced/#use-pdm-in-a-multi-stage-dockerfile) because it uses virtualenvs instead of pypackages

[jankatins]

You can probably copy the whole venv, at least that's what we do with poetry.

[ischaojie]

You can actually use the lock file generated by rye (which has the same format as requirements.txt).

[takashi-yoneya]

I have written a Dockerfile as follows.

Since rye exports requirements.lock by default, use this file to install the package with pip install.

However, since an error occurs if the file is left as is, I removed unnecessary lines.

WORKDIR /backend
COPY ./requirements.lock /backend/
RUN sed '/-e/d' requirements.lock > requirements.txt
RUN pip install -r requirements.txt

[abdulhaq-e]

Note to those unlucky people who just happen to be using a dependency with a url containing "-e" (looking at you @abdulhaq-e):

rye requirement file has a line near the beginning where it says -e file:. which causes package installers to stop. The purpose of sed '/-e/d' requirements.lock > requirements.txt is to remove that line. However, if you have any other -e in the file like I did then it will remove that as well and if you're like me, it will be the last thing to think of and you will only realize what's happening after several hours :D.

A better pattern to provide for sed would be /^-e/d, the command will become:

RUN sed '/^-e/d' requirements.lock > requirements.txt

This will only delete -e found in the beginning of a line.

[helderco]

Poetry has a --no-root (Installing dependencies only) option for this. The requirements.txt format doesn't make that simple, but instead of adding -e file:. to the lock file, it can be done implicitly by rye (when not virtual), and then you could use a similar --no-root option to disable it.

EDIT: Formalized this suggestion in:

• Don't include the project in the lock file in favor of installing it implicitly by default #895

[luisfrodriguezr]

@mitsuhiko Can we add this answer to the documentation?

[ischaojie]

Now I using uv to generate requirements.txt from projects:

[tool.rye.scripts]
dev = "uvicorn app:app --port 8000 --reload"
requirements = "uv pip compile pyproject.toml -o requirements.txt"

[astronautas]

@ischaojie Thanks for a nifty workaround!

[Code-Hex]

I tried to write Dockerfile for the multi-staging build. Here is example one:

ARG PYTHON_BASE_IMAGE='python'

FROM ${PYTHON_BASE_IMAGE}:3.11 AS rye

# Prevents Python from writing pyc files.
ENV PYTHONDONTWRITEBYTECODE=1

# Keeps Python from buffering stdout and stderr to avoid situations where
# the application crashes without emitting any logs due to buffering.
ENV PYTHONUNBUFFERED=1

ENV PYTHONPATH="/workspace/src:$PYTHONPATH"

# The virtual environment is created in the working directory where rye is run
# so the development and production environments must be in the same directory respectively.
WORKDIR /workspace

RUN \
  --mount=type=cache,target=/var/lib/apt/lists \
  --mount=type=cache,target=/var/cache/apt/archives \
  apt-get update \
  && apt-get install -y --no-install-recommends build-essential

ENV RYE_HOME="/opt/rye"
ENV PATH="$RYE_HOME/shims:$PATH"

# RYE_INSTALL_OPTION is required to build.
# See: https://github.com/mitsuhiko/rye/issues/246
RUN curl -sSf https://rye-up.com/get | RYE_NO_AUTO_INSTALL=1 RYE_INSTALL_OPTION="--yes" bash

# Download dependencies as a separate step to take advantage of Docker's caching.
# Leverage a bind mount to some files to avoid having to copy them into
# into this layer.
RUN --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    --mount=type=bind,source=requirements.lock,target=requirements.lock \
    --mount=type=bind,source=requirements-dev.lock,target=requirements-dev.lock \
    --mount=type=bind,source=.python-version,target=.python-version \
    --mount=type=bind,source=README.md,target=README.md \
    rye sync --no-dev --no-lock

RUN . .venv/bin/activate

# Stage for development.
# The development environment assumes a devcontainer and the environment is
# closed inside the container, so you don't need to be aware of the virtual environment
FROM rye AS dev


RUN --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    --mount=type=bind,source=requirements.lock,target=requirements.lock \
    --mount=type=bind,source=requirements-dev.lock,target=requirements-dev.lock \
    --mount=type=bind,source=.python-version,target=.python-version \
    --mount=type=bind,source=README.md,target=README.md \
    rye sync --no-lock

# Stage for production
FROM rye AS run

# Create a non-privileged user that the app will run under.
# See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user
ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    appuser

# Switch to the non-privileged user to run the application.
USER appuser

COPY . .

ENTRYPOINT ["python3", "./src/main.py"]

The directory structure we are assuming looks like this. (This is a fairly simplified version.)

.
├── .devcontainer
│   ├── devcontainer.json
│   ├── docker-compose.yml
│   └── postCreateCommand.sh
├── src      # source code directory for the application
├── tests   # test code directory
├── Dockerfile
└── docker-compose.yml

docker-compose.yml in the project root

version: "3.9"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: run # to specify the multi-stage build target
    ports:
      - 8080:8080

devcontainer.json

{
  "name": "rye",
  "dockerComposeFile": [
    "../docker-compose.yml",
    "docker-compose.yml"
  ],
  "service": "app",
  "workspaceFolder": "/workspace",
  "postCreateCommand": "bash .devcontainer/postCreateCommand.sh",
  "userEnvProbe": "loginInteractiveShell",
  "customizations": {
    "vscode": {
      "extensions": [
        "ms-python.python",
        "charliermarsh.ruff",
        "bungcip.better-toml",
        "ms-azuretools.vscode-docker"
      ]
    }
  }
}

docker-compose.yml in .devcontainer directory

version: "3.9"
services:
  app:
    container_name: app-dev
    build:
      args:
       - PYTHON_BASE_IMAGE=mcr.microsoft.com/vscode/devcontainers/python
      target: ${TARGET_STAGE:-dev} # override to use dev stage
    command: sleep infinity
    volumes:
      - .:/workspace:cached

---

Thoughts

• I don't believe a Python virtual environment is necessary because it confines the dependencies and application code within the container.
• Whenever I run the curl https://rye-up.com/get line, cpython@3.10 is installed, which I believe is unnecessary. I want the .python-version one to be installed. (Perhaps there is something wrong with the way I wrote it.)

[watany-dev]

Here's another way I was able to put it together. As an example of using rye add, I'm importing pandas to verify its installation.

# Use multi-stage builds
FROM ubuntu:jammy-20230522 AS build

# Set up non-root user
ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/home/ryeuser" \
    --shell "/bin/bash" \
    --uid "${UID}" \
    ryeuser

# Update and install dependencies
RUN apt-get update && apt-get install -y \
    curl \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Set working directory
USER ryeuser
WORKDIR /home/ryeuser

# Set Rye environment variables
ENV RYE_HOME="/home/ryeuser/.rye"
ENV PATH="$RYE_HOME/shims:$PATH"

# Install Rye
RUN curl -sSf https://rye-up.com/get | RYE_NO_AUTO_INSTALL=1 RYE_INSTALL_OPTION="--yes" bash

RUN rye init app && cd app \
    && rye add pandas \
    && rye sync

# Second stage to create a lean production image
FROM ubuntu:jammy-20230522
COPY --from=build /home/ryeuser /home/ryeuser

# Set up non-root user
ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    ryeuser

# Switch to non-root user
USER ryeuser
WORKDIR /home/ryeuser/app

# Set Rye environment variables
ENV RYE_HOME="/home/ryeuser/.rye"
ENV PATH="$RYE_HOME/shims:$PATH"
ENTRYPOINT ["rye", "run", "python", "-c", "import pandas"]

[ekusiadadus]

Thanks to @Code-Hex, I tried writing Dockerfile for FastAPI with rye.

ARG PYTHON_BASE_IMAGE='python'

FROM ${PYTHON_BASE_IMAGE}:3.11 AS rye

ENV PYTHONDONTWRITEBYTECODE=1

ENV PYTHONUNBUFFERED=1

ENV PYTHONPATH="/app/src:$PYTHONPATH"

WORKDIR /app

RUN apt-get update \
  && apt-get install -y --no-install-recommends build-essential

ENV RYE_HOME="/opt/rye"
ENV PATH="$RYE_HOME/shims:$PATH"

RUN curl -sSf https://rye-up.com/get | RYE_NO_AUTO_INSTALL=1 RYE_INSTALL_OPTION="--yes" bash

COPY pyproject.toml requirements.lock requirements-dev.lock .python-version README.md ./

RUN rye sync --no-dev --no-lock

RUN . .venv/bin/activate

FROM rye AS dev

RUN rye sync --no-lock

FROM rye AS run

ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/nonexistent" \
    --shell "/sbin/nologin" \
    --no-create-home \
    --uid "${UID}" \
    appuser

USER appuser

COPY src src

ENTRYPOINT ["/app/.venv/bin/uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

[SyedUmerHasan]

This works great with Fast API, only issue I find with this was big docker build size.
Around 2GB. I think we can make it better by changing base image from python to ubuntu:jammy
As rye is gonna install cpython itself so there would be 2 python versions in a build.

[zhuxining]

Why requirements.lock must has a row "-e file:." ?

use dockerfile:

……
COPY ./requirements.lock .
RUN pip install --no-cache-dir --upgrade -r requirements.lock
……

when docker build, has error:

Obtaining file:///. (from -r requirements.lock (line 9))
ERROR: file:///. (from -r requirements.lock (line 9)) does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
The command '/bin/sh -c pip install --no-cache-dir --upgrade -r requirements.lock' returned a non-zero code: 1

[zhuxining]

mark a solution
#239 (comment)

[tsimoshka]

The possible approach with rye build. Works great for workspace packages:

ARG PYTHON_VERSION
FROM python:${PYTHON_VERSION}-slim as base

ARG UID=10001
RUN adduser \
    --disabled-password \
    --gecos "" \
    --home "/work" \
    --shell "/sbin/nologin" \
    --uid "${UID}" \
    appuser

WORKDIR /work

FROM base as build

# dependencies for rye installation
RUN \
    --mount=type=cache,target=/var/lib/apt/lists \
    --mount=type=cache,target=/var/cache/apt/archives \
        apt-get update \
        && apt-get install -y --no-install-recommends \
            curl \
            ca-certificates \
            build-essential

# use the appuser in the build stage
# to allow re-using pip cache in run stage
USER appuser

ENV RYE_HOME="/work/rye"
ENV PATH="$RYE_HOME/shims:$PATH"

RUN curl -sSf https://rye-up.com/get | RYE_INSTALL_OPTION="--yes" bash

RUN --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    --mount=type=bind,source=requirements.lock,target=requirements.lock \
    --mount=type=bind,source=requirements-dev.lock,target=requirements-dev.lock \
    --mount=type=bind,source=.python-version,target=.python-version \
    --mount=type=bind,source=README.md,target=README.md \
    --mount=type=bind,source=src1,target=src1 \
    # you can add as many source foders as you want
    # for workspace packages:
    --mount=type=bind,source=src2,target=src2 \
    # this on mounts the pip cache 
    # to build faster on subsequent builds
    --mount=type=cache,target=/work.cache/pip,uid=$UID \
    # install proeject prod deps
    rye sync --no-dev --no-lock \
    # build the weels for all packages in workspace
    # and plase them into /dist/work directory
    && rye build --all --wheel \
    # edit requirements.lock file to remove editable packages from it
    # and place it in the same /work/dist directory
    && sed '/-e/d' requirements.lock > dist/requirements.lock


# production container
FROM base as run

USER appuser

# add bin user folder
# to have access to [project.scripts]
# installed by your packages
ENV PATH=/work/.local/bin:$PATH

# mount pip cache to reuse dependencies already downloaded in the first step
# mount dist directory to have access to modified requirements.lock file and built wheels
# install dependencies for requirements.lock file (pip cache will be used from the previous stage, so now new downloads should happen)
# and wheels directly from directory
RUN --mount=type=cache,target=/work/.cache/pip,uid=$UID \
    --mount=type=bind,from=build,source=/work/dist,target=/work/dist \
    python -m pip install -r /work/dist/requirements.lock /work/dist/*.whl

to build use:

docker build --build-arg PYTHON_VERSION=$(cat .python-version) --progress=plain .

The one small issue with this approach is a bit slow rye installation (downloading CPython in internals bootstrapping) on the first run, or when the layer cache is busted. Would be nice to have some way to either use the already available Python in a container or to use --mount=bind caches to reuse this Python between different projects maybe.

Also, it would be nice to be able to generate a wheelhouse with rye from the .venv with all the dependencies to mount it in the final container and install everything with pip install --no-index /work/wheelhouse/*.whl

[spookyuser]

ily

[bluss]

You now have RYE_TOOLCHAIN to use an existing python

[mostthingsweb]

One thing to remember is to add .venv/ to your .dockerignore, otherwise if you are using rye outside of Docker, then the existing .venv/ will screw up the build.

[spookyuser]

good point! i think this may have actually been happening to me

[mostthingsweb]

If anyone wants to build Rye from source, this is what I came up with:

FROM rust:latest as rust_builder
RUN cargo install --git https://github.com/mitsuhiko/rye rye

FROM debian:bookworm as app
WORKDIR /app
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

ENV RYE_HOME="/app/rye"
ENV PATH="$RYE_HOME/shims:$PATH"
COPY --from=rust_builder /usr/local/cargo/bin/rye /usr/local/bin/
COPY pyproject.toml .
RUN rye self install --yes && \
    rye sync

COPY . .

# blah blah blah

RUN python3 myapp.py

[spookyuser]

so when you use the python3 at the bottom it'll have access to all the stuff rye installed?

[mostthingsweb]

Yes, because the 'python3' that is found is the rye shim (which we injected into PATH using ENV).

[spookyuser]

awesome!

[spookyuser]

Oh btw i also needed a RUN apt-get update && apt-get install ca-certificates -y for the last builder

[Igetin]

Here’s a multi-stage distroless example that I’m using with Podman:

# Use debian:12-slim as the distroless image is based on it as well
FROM debian:12-slim AS build-setup

ENV PYTHONDONTWRITEBYTECODE=1

WORKDIR /build

ARG RYE_VERSION="0.16.0"
ARG RYE_BASE_URL="https://github.com/mitsuhiko/rye/releases/download/$RYE_VERSION"
ARG RYE_PACKAGE_NAME="rye-x86_64-linux.gz"
RUN /bin/bash <<END_RUN
set -o nounset -o errexit -o pipefail

# Install distro dependencies
apt-get update
apt-get install --no-install-suggests --no-install-recommends --yes \
    python3-venv python3-pip curl

# Install Rye
curl --show-error --location --progress-bar --remote-name-all \
    --write-out "%{filename_effective}: %{http_code}\n" \
    "$RYE_BASE_URL/$RYE_PACKAGE_NAME" "$RYE_BASE_URL/$RYE_PACKAGE_NAME.sha256"
echo "\$(<rye-x86_64-linux.gz.sha256)" "$RYE_PACKAGE_NAME" | sha256sum --check -
gunzip --verbose --stdout "$RYE_PACKAGE_NAME" > /opt/rye
chmod --verbose +x /opt/rye

# Prepare a virtual environment
/usr/bin/python3 -m venv /venv
/venv/bin/pip install --no-cache-dir --upgrade pip wheel
END_RUN


FROM build-setup AS build-venv

# Copy the source code and build configuration
COPY pyproject.toml requirements.lock /build/
COPY src /build/src

# Build a wheel and install it in the virtual environment
RUN /bin/bash <<END_RUN
set -o nounset -o errexit -o pipefail
/opt/rye build --wheel
/venv/bin/pip install /build/dist/myapp*.whl
END_RUN


# This refers to a Python 3.11 non-root image
FROM gcr.io/distroless/python3-debian12@sha256:27d2d6afcfb109e4c147449d4af957f71cb770196527d0da1d1d92b9680b0daa

COPY --from=build-venv /venv /app/.venv

USER nonroot

# Makes sure we get timely logs from containers
ENV PYTHONUNBUFFERED 1

ENTRYPOINT [ "/app/.venv/bin/python3.11", "-m", "myapp" ]