How to use pipenv with multistage docker builds? #3160
Comments
Apologies if the original intent was not clear enough. All I want is to install my package and its dependencies into a separate directory instead of python's site-packages; and then to copy that directory into the final docker image. Since I'll try to elaborate on the problem: I want to keep my final, "production" docker image as small as possible. Therefore I need to build/install my app dependencies in earlier stage. |
Pipenv unsets user settings because they are incompatible with virtualenv settings. I understand the approach you’re taking, but maybe you can say more about what problem you are trying to solve? You want the wheels and sdists or whatever? If so, on Linux they’re stored in |
@techalchemy My comment appeared before yours for some reason (github still recovers?). Just making sure you got notification. |
Ah okay, you may be able to just use |
While it's nice to know about this option, I don't see show does it help me. Site packages from vanilla venv weigh 12MB (because of pip and setuptools) What I'm trying to do is to run
and easily collect results of only the last two lines. So another venv does not help me. I need to "split" the installation paths within the venv. Do you think PEEP that suggest obeying PIP* options if |
Why not just specify your library dependencies in your package metadata and just add uninstall steps for pip/setuptools via |
FOUND IT!!!Here is what works: In action:
Hooray! I'm happy now. |
I mean I understand the steps you are attempting. I am trying to understand why there is a strict constraint on not including incidental dependencies and why you have to fish entry points out. Historically we had documentation on using |
We'd like to have our docker images as lean as possible. One of the reasons that we have some systems running over mobile line internet, so when pushing upgrade for 20 microservices, those megabytes start to add up. On alpine, pip + setuptools alone weigh 10MB. Compare it to alpine itself (5MB) + python3 (40MB) and that's over 20% increase. Another reason is security - I want to include only what's really used by my app, to minimize attack surface. It's not just me freaking out. It seems where the whole industry is going. With Go, they compile statically and bundle docker image that even does not have a shell. Distroless is another example.
My setup.py installs entrypoints that will go to venv's
I'm not sure what you mean. Can you please elaborate? Which artifacts are not preserved exactly?
Even not during build stage? $ mkdir /pyroot; chown appinstall:appinstall /pyroot
$ PIP_USER=1 PYTHONUSERBASE=/pyroot su-exec appinstall:appinstall pipenv install --system --deploy
$ ls -lah /pyroot/
total 16
drwxr-xr-x 4 appinsta appinsta 4.0K Nov 6 02:25 .
drwxr-xr-x 19 root root 4.0K Nov 6 02:23 ..
drwxr-xr-x 2 appinsta appinsta 4.0K Nov 6 02:25 bin
drwxr-xr-x 3 appinsta appinsta 4.0K Nov 6 02:25 lib |
where to copy these /bin and /lib dirs, to reach it from an other build stage? @haizaar Can you publish a working Dockerfile? (without private code, of course) |
@derPuntigamer It's all here: https://tech.zarmory.com/2018/09/docker-multi-stage-builds-for-python-app.html (scroll down for pipenv version). Questions are welcome. |
@haizaar thats only for alpine it seems. will try work it out for deb/ubuntu, the pipenv user info should be enough to go on though |
Can you elaborate what you mean? Why pipenv steps for Ubuntu should be any different? |
can confirm this works on the non-alpine I only needed a simple script on top of the pipenv deps, I added an initial
many thanks for the hard work here @haizaar! |
I would suggest also to add
|
Are there anything unresolved on this topic? It seems to me that everything in recent comments is working. |
LGTM. |
I'll revert my last comment - |
Since running into #4432 which is possibly related to #4453 I have changed my multistage docker builds to utilize the venv method described here: https://sourcery.ai/blog/python-docker/ This does come with the drawbacks that @haizaar had mentioned of including unneeded additional files in the final image, however it appears to be the most stable approach, with using |
Good day,
I'm exploring on how to use pipenv with multi-stage docker builds. In the nutshell, the idea is to "compile" stuff in base image and only copy the resulting artifacts to the final image.
With Python is gets tricky, since you need to copy package dependencies as well.
I've checked out several ideas and looks like
pip install --user
together with settingPYTHONUSERBASE
is the simplest ways to install dependencies to a side directory, e.g.:(The full story)
The problem is that pipenv disregards
PYTHONUSERBASE
:I found a workaround by using
pipenv lock -r
and then installingrequirements.txt
as in my original idea, but I'm not sure this is the best way to go, particularly if I have custom (private)source
s defined in my Pipefile - I don't want to replicate their configuration into pip.Any other ideas?
The text was updated successfully, but these errors were encountered: