Optimize Docker Layers While Ignoring Pyproject.toml Changes (except dependencies)

[GuyGooL5]

Hi everyone, I'm facing a situation where I have a Dockerfile that is quite well optimized regarding layers.
My only problem is this line

# ... installed poetry and stuff
COPY pyproject.toml poetry.lock ./
RUN poetry install --no-root
# ... rest of the dockerfile incl' copying the project and installing it

It works fine and all but every time I'm bumping the version of my project in pyproject.toml this whole layer is getting invalidated and being re-run.
This is really annoying since most of the work is downloading packages. I don't want to use global installs (I'm using in-project virtualenv for staged-builds).

My question is that if is there a good practice of versioning poetry-managed packages outside of pyproject.toml file, or even managing the dependencies in a different way.

All I want to have is a sanely versioned project and an expensive layer cached and invalidated only when really necessary.

[dimbleby]

https://python-poetry.org/docs/faq/#poetry-busts-my-docker-cache-because-it-requires-me-to-copy-my-source-files-in-before-installing-3rd-party-dependencies

[dimbleby]

too bad, then.

You can't expect to change pyproject.toml without invalidating the layer that includes pyproject.toml; and that is always going to come before the things that rely on pyprojec.toml

[GuyGooL5]

Guess there's no way of avoiding it 😕

[adriangb]

My advice: don't version your services via a version in pyproject.toml.

[atemate]

don't version your services via a version in pyproject.toml.

I don't think it's possible, version attribute is required in pyproject.toml in a package mode

[adriangb]

You can set that to 0.0.0 or whatever. What matters is the tag on your container images, the version in your logs/traces/metrics, etc. For that I recommend you use the git sha.

[MarcelWilson]

AFAIK, the only real thing you can do is ensure all the steps in the dockerfile that can be cached occur first and run the poetry install step as late as possible.

Also, a little trick I learned is I didn't need to copy my whole project into the dockerfile (thus incurring a new cache layer with every change). I use a volume mapping to point to the local directory where my project resides. I still have to copy the pyproject.toml & lock but everything else sits outside the image. The image is only the runtime environment. YMMV of course.

I use docker compose, but you could accomplish the same thing with commandline when running the image.

services:
    name_of_your_service:
        build:
            context: .
            dockerfile: wherever_you_keep_your_dockerfiles/Dockerfile
        image: name_of_your_image
        volumes:
            # This is the mapping.  localhost_path:image_path
            # ./ is the root of the project on the localhost
            # /localfolder is an alias of the directory within the image.
            - ./:/localfolder

COPY pyproject.toml poetry.lock* ./
RUN poetry install

WORKDIR /localfolder/tests
# from here you can still execute scripts in the ./tests without ever having to copy them into the image

[GuyGooL5]

I did all optimizations possible, except that neat trick using volumes. Although that's a good tip, unfortunately, it still doesn't solve my initial problem, which I guess is not that big of a problem considering the effort.

[BCarley]

You could use mounted caches?

RUN --mount=type=cache,target=/root/.cache poetry config cache-dir /root/.cache && \
        poetry install --without=dev --no-root --no-directory

[sbrunner]

I think that this thread also provides some good recommendations:
https://github.com/orgs/python-poetry/discussions/1879