Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
How to use pipenv with multistage docker builds? #3160
I'm exploring on how to use pipenv with multi-stage docker builds. In the nutshell, the idea is to "compile" stuff in base image and only copy the resulting artifacts to the final image.
With Python is gets tricky, since you need to copy package dependencies as well.
I've checked out several ideas and looks like
FROM alpine AS builder # Install your gcc, python3-dev, etc. here apk add --no-cache python3 COPY . /src/ WORKDIR /src ENV PYROOT /pyroot RUN PYTHONUSERBASE=$PYROOT pip3 install --user -r requirements.txt RUN PYTHONUSERBASE=$PYROOT pip3 install --user . # The final image FROM alpine apk add --no-cache python3 ENV PYROOT /pyroot COPY --from=builder $PYROOT/lib/ $PYROOT/lib/
The problem is that pipenv disregards
I found a workaround by using
Any other ideas?
Apologies if the original intent was not clear enough.
All I want is to install my package and its dependencies into a separate directory instead of python's site-packages; and then to copy that directory into the final docker image.
I'll try to elaborate on the problem: I want to keep my final, "production" docker image as small as possible. Therefore I need to build/install my app dependencies in earlier stage.
Pipenv unsets user settings because they are incompatible with virtualenv settings. I understand the approach you’re taking, but maybe you can say more about what problem you are trying to solve? You want the wheels and sdists or whatever? If so, on Linux they’re stored in
While it's nice to know about this option, I don't see show does it help me. Site packages from vanilla venv weigh 12MB (because of pip and setuptools)
What I'm trying to do is to run
and easily collect results of only the last two lines. So another venv does not help me. I need to "split" the installation paths within the venv.
Do you think PEEP that suggest obeying PIP* options if
Why not just specify your library dependencies in your package metadata and just add uninstall steps for pip/setuptools via
OK, I'll try once again to explain. Bear with me please.
Here is what would've wanted to do (in dockerfile):
# Install all stuff in our system so we can install setup.py for our app later on RUN pipenv install --deploy --system --dev # Install our app LOCKED dependencies aside RUN PYTHONUSERBASE=/pyroot pipenv install --deploy --system # Install the app aside as well - all dependencies already present, so it's just the app RUN PYTHONUSERBASE=/pyroot pip install --user . # Take pyroot to the next docker image stage
This look very clear and "human" (as in pipenv's motto) to me :)
Other solutions that require fiddling with virtualenv look error-prone IMHO. I've tried
Uninstall does not remove dependencies
Whatever we conclude in this discussion, I think there should be PEEP/doc explaining the recommended way to use pipenv with docker multi-stage.
Here is what works:
I'm happy now.
I mean I understand the steps you are attempting. I am trying to understand why there is a strict constraint on not including incidental dependencies and why you have to fish entry points out.
Historically we had documentation on using
We'd like to have our docker images as lean as possible. One of the reasons that we have some systems running over mobile line internet, so when pushing upgrade for 20 microservices, those megabytes start to add up. On alpine, pip + setuptools alone weigh 10MB. Compare it to alpine itself (5MB) + python3 (40MB) and that's over 20% increase. Another reason is security - I want to include only what's really used by my app, to minimize attack surface. It's not just me freaking out. It seems where the whole industry is going. With Go, they compile statically and bundle docker image that even does not have a shell. Distroless is another example.
My setup.py installs entrypoints that will go to venv's
I'm not sure what you mean. Can you please elaborate? Which artifacts are not preserved exactly?
Even not during build stage?
$ mkdir /pyroot; chown appinstall:appinstall /pyroot $ PIP_USER=1 PYTHONUSERBASE=/pyroot su-exec appinstall:appinstall pipenv install --system --deploy $ ls -lah /pyroot/ total 16 drwxr-xr-x 4 appinsta appinsta 4.0K Nov 6 02:25 . drwxr-xr-x 19 root root 4.0K Nov 6 02:23 .. drwxr-xr-x 2 appinsta appinsta 4.0K Nov 6 02:25 bin drwxr-xr-x 3 appinsta appinsta 4.0K Nov 6 02:25 lib