build time only -v option #14080

Open
zrml opened this Issue Jun 21, 2015 · 179 comments

Comments

Projects
None yet
@zrml

zrml commented Jun 21, 2015

As suggested by @cpuguy83 in docker#3156
here is the use case for a flexible -v option at build time.

When building a Docker image I need to install a database and an app. It's all wrapped up in two tarballs: 1 for the DB and 1 for the App that needs to be installed in it (schema, objects, static data, credentials etc.). The whole solution is then run via a shell script that handles several shell variables and tune OS credentials and other things accordingly.
When I explode the above tarball (or use the Dockerfile ADD directive) the whole thing bloats up to about 1.5GB(!). Not ideal as you can immagine.

I would like to have this '-v /distrib/ready2installApp:/distrib' directive still possible (as it is today in the Dockerfile)
but

I would like to disassociate the declarative build process (infrastructure as code) from the container run-time deployable artifact. I do not want to have to deal with the dead weight of 1.5GB that I do not need.

Could we have an --unmount-volume option that I can run at the end of the Dockerfile?
or
Given how Volume works right now in a Dockerfile, maybe we need a new Dockerfile directive for a temporary volume that people use while installing? I think the Puppet example supplied by @fatherlinux was on a similar line...
or
Whatever you guys can think of.
The objective is avoiding to have to carry around all that dead weight that is useless for a deployed App or Service. However that dead weight is necessary @install-time. Not Everybody has a simple "yum install" from the official repositories. :)

thank you very much

@tpires

This comment has been minimized.

Show comment
Hide comment
@tpires

tpires Jun 25, 2015

I'm looking for a similar solution.

Problem

Recently the enterprise I work enabled Zscaler proxy with SSL inspection, which implies having certificates installed and some environment variables set during build.

A temporarily solution was to create a new Dockerfile with certificates and environment variables set. But that doesn't seem reasonable, in a long term view.

So, my first thought was set a transparent proxy with HTTP and HTTPS, but again I need to pass a certificate during build.

The ideal scenario is with the same Dockerfile, I would be able to build my image on my laptop, at home, and enterprise.

Possible solution

# Enterprise
$ docker build -v /etc/ssl:/etc/ssl -t myimage .

# Home
$ docker build -t myimage .

tpires commented Jun 25, 2015

I'm looking for a similar solution.

Problem

Recently the enterprise I work enabled Zscaler proxy with SSL inspection, which implies having certificates installed and some environment variables set during build.

A temporarily solution was to create a new Dockerfile with certificates and environment variables set. But that doesn't seem reasonable, in a long term view.

So, my first thought was set a transparent proxy with HTTP and HTTPS, but again I need to pass a certificate during build.

The ideal scenario is with the same Dockerfile, I would be able to build my image on my laptop, at home, and enterprise.

Possible solution

# Enterprise
$ docker build -v /etc/ssl:/etc/ssl -t myimage .

# Home
$ docker build -t myimage .
@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Jul 7, 2015

I have a slightly different use case for this feature - Caching packages which are downloaded / updated by the ASP.Net 5 package manager. The package manager manages its own cache folder so ultimately I just need a folder which I can re-use between builds.

I.e:

docker build -v /home/dokku/cache/dnx/packages:/opt/dnx/packages -t "dokku/aspnettest" .

yngndrw commented Jul 7, 2015

I have a slightly different use case for this feature - Caching packages which are downloaded / updated by the ASP.Net 5 package manager. The package manager manages its own cache folder so ultimately I just need a folder which I can re-use between builds.

I.e:

docker build -v /home/dokku/cache/dnx/packages:/opt/dnx/packages -t "dokku/aspnettest" .
@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Jul 16, 2015

@yngndrw what you propose would be OK for me too, i.e, we need to mount extra resources at build time that would not be necessary at run time as they have been installed in the container.

FWIW I saw somewhere in these pages somebody saying something along the line of (and I hope I'm paraphrasing it right) "resolve your compilation issue on a similar host machine then just install the deployable artifact or exe in the container".
I'm afraid it's not that simple guys. At times, I need to install in /usr/bin but I also need to edit some config file. I check for the OS I'm running on, the kernel params I need to tune, the files I need to create depending on variables or manifesto build files. There are many dependencies that are just not satisfied with a simple copy of a compiled product.

I re-state what I said when I open the issue: there is a difference between a manifest declaration file and its process and the run-time of an artifact.
If we truly believe in infrastructure-as-code and furthermore in immutable infrastructure, that Docker itself is promoting further & I like it btw, then this needs to be seriously considered IMO (see the bloating in post 1 herewith)

Thank you again

zrml commented Jul 16, 2015

@yngndrw what you propose would be OK for me too, i.e, we need to mount extra resources at build time that would not be necessary at run time as they have been installed in the container.

FWIW I saw somewhere in these pages somebody saying something along the line of (and I hope I'm paraphrasing it right) "resolve your compilation issue on a similar host machine then just install the deployable artifact or exe in the container".
I'm afraid it's not that simple guys. At times, I need to install in /usr/bin but I also need to edit some config file. I check for the OS I'm running on, the kernel params I need to tune, the files I need to create depending on variables or manifesto build files. There are many dependencies that are just not satisfied with a simple copy of a compiled product.

I re-state what I said when I open the issue: there is a difference between a manifest declaration file and its process and the run-time of an artifact.
If we truly believe in infrastructure-as-code and furthermore in immutable infrastructure, that Docker itself is promoting further & I like it btw, then this needs to be seriously considered IMO (see the bloating in post 1 herewith)

Thank you again

@fatherlinux

This comment has been minimized.

Show comment
Hide comment
@fatherlinux

fatherlinux Aug 16, 2015

Another use case that is really interesting is upgrading software. There are times, like with FreeIPA, you should really test with a copy of the production data to makes sure that all of the different components can cleanly upgrade. You still want to do the upgrade in a "build" environment. You want the production copy of the data to live somewhere else so that when you move the new upgraded versions of the containers into production, they can mound the exact data that you did the upgrade on.

Another example, would be Satellite/Spacewalk which changes schema often and even changed databases from Oracle to Postgresql at version 5.6 (IIRC).

There are many, many scenarios when I temporarily need access to data while doing an upgrade of software in a containerized build, especially with distributed/micro services....

Another use case that is really interesting is upgrading software. There are times, like with FreeIPA, you should really test with a copy of the production data to makes sure that all of the different components can cleanly upgrade. You still want to do the upgrade in a "build" environment. You want the production copy of the data to live somewhere else so that when you move the new upgraded versions of the containers into production, they can mound the exact data that you did the upgrade on.

Another example, would be Satellite/Spacewalk which changes schema often and even changed databases from Oracle to Postgresql at version 5.6 (IIRC).

There are many, many scenarios when I temporarily need access to data while doing an upgrade of software in a containerized build, especially with distributed/micro services....

@fatherlinux

This comment has been minimized.

Show comment
Hide comment
@fatherlinux

fatherlinux Aug 16, 2015

Essentially, I am now forced to do a manual upgrade by running a regular container with a -v bind mount, then doing a "docker commit." I cannot understand why the same capability wouldn't be available with an automated Dockerfile build?

Essentially, I am now forced to do a manual upgrade by running a regular container with a -v bind mount, then doing a "docker commit." I cannot understand why the same capability wouldn't be available with an automated Dockerfile build?

@stevenschlansker

This comment has been minimized.

Show comment
Hide comment
@stevenschlansker

stevenschlansker Aug 19, 2015

Seconding @yngndrw pointing out caching: the exact same reasoning applies to many popular projects such as Maven, npm, apt, rpm -- allowing a shared cache can dramatically speed up builds, but must not make it into the final image.

Seconding @yngndrw pointing out caching: the exact same reasoning applies to many popular projects such as Maven, npm, apt, rpm -- allowing a shared cache can dramatically speed up builds, but must not make it into the final image.

@NikonNLG

This comment has been minimized.

Show comment
Hide comment
@NikonNLG

NikonNLG Aug 19, 2015

I agree with @stevenschlansker. It can be many requirements for attach cache volume, or some kind of few data gigabytes, which must present (in parsed state) on final image, but not as raw data.

I agree with @stevenschlansker. It can be many requirements for attach cache volume, or some kind of few data gigabytes, which must present (in parsed state) on final image, but not as raw data.

@wjordan

This comment has been minimized.

Show comment
Hide comment
@wjordan

wjordan Aug 20, 2015

I've also been bitten by the consistent resistance to extending docker build to support the volumes that can be used by docker run. I have not found the 'host-independent builds' mantra to be very convincing, as it only seems to make developing and iterating on Docker images more difficult and time-consuming when you need to re-download the entire package repository every time you rebuild an image.

My initial use case was a desire to cache OS package repositories to speed up development iteration. A workaround I've been using with some success is similar to the approach suggested by @fatherlinux, which is to just give up wrestling with docker build and the Dockerfile altogether, and start from scratch using docker run on a standard shell script followed by docker commit.

As a bit of an experiment, I extended my technique into a full-fledged replacement for docker build using a bit of POSIX shell scripting: dockerize.

If anyone wants to test out this script or the general approach, please let me know if it's interesting or helpful (or if it works at all for you). To use, put the script somewhere in your PATH and add it as a shebang for your build script (the #! thing), then set relevant environment variables before a second shebang line marking the start of your Docker installation script.

FROM, RUNDIR, and VOLUME variables will be automatically passed as arguments to docker run.
TAG, EXPOSE, and WORKDIR variables will be automatically passed as arguments to docker commit.

All other variables will be evaluated in the shell and passed as environment arguments to docker run, making them available within your build script.

For example, this script will cache and reuse Alpine Linux packages between builds (the VOLUME mounts a home directory to CACHE, which is then used as a symlink for the OS's package repository cache in the install script):

#!/usr/bin/env dockerize
FROM=alpine
TAG=${TAG:-wjordan/my-image}
WORKDIR=/var/cache/dockerize
CACHE=/var/cache/docker
EXPOSE=3001
VOLUME="${HOME}/.docker-cache:${CACHE} ${PWD}:${WORKDIR}:ro /tmp"
#!/bin/sh
ln -s ${CACHE}/apk /var/cache/apk
ln -s ${CACHE}/apk /etc/apk/cache
set -e
apk --update add gcc g++ make libc-dev python
[...etc etc build...]

wjordan commented Aug 20, 2015

I've also been bitten by the consistent resistance to extending docker build to support the volumes that can be used by docker run. I have not found the 'host-independent builds' mantra to be very convincing, as it only seems to make developing and iterating on Docker images more difficult and time-consuming when you need to re-download the entire package repository every time you rebuild an image.

My initial use case was a desire to cache OS package repositories to speed up development iteration. A workaround I've been using with some success is similar to the approach suggested by @fatherlinux, which is to just give up wrestling with docker build and the Dockerfile altogether, and start from scratch using docker run on a standard shell script followed by docker commit.

As a bit of an experiment, I extended my technique into a full-fledged replacement for docker build using a bit of POSIX shell scripting: dockerize.

If anyone wants to test out this script or the general approach, please let me know if it's interesting or helpful (or if it works at all for you). To use, put the script somewhere in your PATH and add it as a shebang for your build script (the #! thing), then set relevant environment variables before a second shebang line marking the start of your Docker installation script.

FROM, RUNDIR, and VOLUME variables will be automatically passed as arguments to docker run.
TAG, EXPOSE, and WORKDIR variables will be automatically passed as arguments to docker commit.

All other variables will be evaluated in the shell and passed as environment arguments to docker run, making them available within your build script.

For example, this script will cache and reuse Alpine Linux packages between builds (the VOLUME mounts a home directory to CACHE, which is then used as a symlink for the OS's package repository cache in the install script):

#!/usr/bin/env dockerize
FROM=alpine
TAG=${TAG:-wjordan/my-image}
WORKDIR=/var/cache/dockerize
CACHE=/var/cache/docker
EXPOSE=3001
VOLUME="${HOME}/.docker-cache:${CACHE} ${PWD}:${WORKDIR}:ro /tmp"
#!/bin/sh
ln -s ${CACHE}/apk /var/cache/apk
ln -s ${CACHE}/apk /etc/apk/cache
set -e
apk --update add gcc g++ make libc-dev python
[...etc etc build...]
@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Aug 24, 2015

So, after meeting the French contingent :) from Docker at MesoCon last week (it was a pleasure guys) I was made aware they have the same issue in-house and they developed a hack that copies over to a new slim image what they need.
I'd say that hacks are note welcome in the enterprise world ;) and this request should be properly handled.
Thank you for listening guys...

zrml commented Aug 24, 2015

So, after meeting the French contingent :) from Docker at MesoCon last week (it was a pleasure guys) I was made aware they have the same issue in-house and they developed a hack that copies over to a new slim image what they need.
I'd say that hacks are note welcome in the enterprise world ;) and this request should be properly handled.
Thank you for listening guys...

@yngndrw yngndrw referenced this issue in aspnet/aspnet-docker Sep 9, 2015

Closed

Dockerfile caching #59

@raine

This comment has been minimized.

Show comment
Hide comment
@raine

raine Sep 17, 2015

I'm also in favor of adding build-time -v flag to speed up builds by sharing a cache directory between them.

raine commented Sep 17, 2015

I'm also in favor of adding build-time -v flag to speed up builds by sharing a cache directory between them.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Sep 17, 2015

@yngndrw I don't understand why you closed two related issues. I read your #59 issue and I don't see how this relates to this. In some cases containers become super-bloated when it's not needed at run-time. Please read the 1st post.
I hope I'm not missing something here... as it has been a long day :-o

zrml commented Sep 17, 2015

@yngndrw I don't understand why you closed two related issues. I read your #59 issue and I don't see how this relates to this. In some cases containers become super-bloated when it's not needed at run-time. Please read the 1st post.
I hope I'm not missing something here... as it has been a long day :-o

@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Sep 17, 2015

@zrml Issue aspnet/aspnet-docker#59 was related to the built-in per-layer caching that docker provides during a build to all docker files, but this current issue is subtly different as we are talking about using host volumes to provide dockerfile-specific caching which is dependent on the dockerfile making special use of the volume. I closed issue aspnet/aspnet-docker#59 as it is not specifically related to the aspnet-docker project / repository.

The other issue that I think you're referring to is issue progrium/dokku#1231, which was regarding the Dokku processes explicitly disabling the built-in docker layer caching. Michael made a change to Dokku in order to allow this behaviour to be configurable and this resolved the issue in regards to the Dokku project / repository, so that issue was also closed.

There is possibly still a Docker-related issue that is outstanding (I.e. Why was Docker not handling the built-in layer caching as I expected in issue aspnet/aspnet-docker#59), but I haven't had a chance to work out why that is and confirm if it's still happening. If it is still an issue, then a new issue for this project / repository should be raised for it as it is distinct from this current issue.

yngndrw commented Sep 17, 2015

@zrml Issue aspnet/aspnet-docker#59 was related to the built-in per-layer caching that docker provides during a build to all docker files, but this current issue is subtly different as we are talking about using host volumes to provide dockerfile-specific caching which is dependent on the dockerfile making special use of the volume. I closed issue aspnet/aspnet-docker#59 as it is not specifically related to the aspnet-docker project / repository.

The other issue that I think you're referring to is issue progrium/dokku#1231, which was regarding the Dokku processes explicitly disabling the built-in docker layer caching. Michael made a change to Dokku in order to allow this behaviour to be configurable and this resolved the issue in regards to the Dokku project / repository, so that issue was also closed.

There is possibly still a Docker-related issue that is outstanding (I.e. Why was Docker not handling the built-in layer caching as I expected in issue aspnet/aspnet-docker#59), but I haven't had a chance to work out why that is and confirm if it's still happening. If it is still an issue, then a new issue for this project / repository should be raised for it as it is distinct from this current issue.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Sep 18, 2015

@yngndrw exactly, so we agree this is different and known @docker.com so I'm re-opening it if you don't mind... well I cannot. Do you mind, please?
I'd like to see some comments from our colleagues in SF at least before we close it

BTW I was asked by @cpuguy83 to open a user case and explain it all, from log #3156

zrml commented Sep 18, 2015

@yngndrw exactly, so we agree this is different and known @docker.com so I'm re-opening it if you don't mind... well I cannot. Do you mind, please?
I'd like to see some comments from our colleagues in SF at least before we close it

BTW I was asked by @cpuguy83 to open a user case and explain it all, from log #3156

@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Sep 18, 2015

@zrml I'm not sure I follow - Is it aspnet/aspnet-docker#59 that you want to re-open ? It isn't an /aspnet/aspnet-docker issue so I don't think it's right to re-open that issue. It should really be a new issue on /docker/docker, but would need to be verified and would need re-producible steps generating first.

yngndrw commented Sep 18, 2015

@zrml I'm not sure I follow - Is it aspnet/aspnet-docker#59 that you want to re-open ? It isn't an /aspnet/aspnet-docker issue so I don't think it's right to re-open that issue. It should really be a new issue on /docker/docker, but would need to be verified and would need re-producible steps generating first.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Sep 18, 2015

no, no.. this one #14080 that you closed yesterday.

zrml commented Sep 18, 2015

no, no.. this one #14080 that you closed yesterday.

@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Sep 18, 2015

This issue is still open ?

yngndrw commented Sep 18, 2015

This issue is still open ?

@flomotlik flomotlik referenced this issue in codeship/codeship-tool-examples Sep 19, 2015

Merged

Updating the deployment example to use volumes, not configure #10

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Sep 21, 2015

@yngndrw I believe I mis-read the red "closed" icon. Apologies.

zrml commented Sep 21, 2015

@yngndrw I believe I mis-read the red "closed" icon. Apologies.

@lukaso

This comment has been minimized.

Show comment
Hide comment
@lukaso

lukaso Sep 22, 2015

Heartily agree that build time -v would be a huge help.

Build caching is one use case.

Another use case is using ssh keys at build time for building from private repos without them being stored in the layer, eliminating the need for hacks (though well engineered) such as this one: https://github.com/dockito/vault

lukaso commented Sep 22, 2015

Heartily agree that build time -v would be a huge help.

Build caching is one use case.

Another use case is using ssh keys at build time for building from private repos without them being stored in the layer, eliminating the need for hacks (though well engineered) such as this one: https://github.com/dockito/vault

@btrepp

This comment has been minimized.

Show comment
Hide comment
@btrepp

btrepp Oct 8, 2015

I'm commenting here because this is hell in a corporate world.
We have a SSL intercepting proxy, while I can direct traffic through it, heaps of projects assume they have good SSL connections, so they die horribly.

Even though my machine (and thus the docker builder) trusts the proxy, docker images don't.
Worst still the best practice is now to use curl inside the container, so that is painful, I have to modify Dockerfiles to make them even build. I could mount the certificates with a -v option, and be happy.

This being said. Its less the fault of docker, more the fault of package managers using https when they should be using a system similar to how apt-get works. As that is still secure and verifyable, and also cacheable by a http proxy.

btrepp commented Oct 8, 2015

I'm commenting here because this is hell in a corporate world.
We have a SSL intercepting proxy, while I can direct traffic through it, heaps of projects assume they have good SSL connections, so they die horribly.

Even though my machine (and thus the docker builder) trusts the proxy, docker images don't.
Worst still the best practice is now to use curl inside the container, so that is painful, I have to modify Dockerfiles to make them even build. I could mount the certificates with a -v option, and be happy.

This being said. Its less the fault of docker, more the fault of package managers using https when they should be using a system similar to how apt-get works. As that is still secure and verifyable, and also cacheable by a http proxy.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Oct 8, 2015

@btrepp thank you for another good use case.

zrml commented Oct 8, 2015

@btrepp thank you for another good use case.

@btrepp

This comment has been minimized.

Show comment
Hide comment
@btrepp

btrepp Oct 9, 2015

I can think of another situation.

One of the things I would like to do with my dockerfiles is not ship the build tools with the "compiled" docker file. There's no reason a C app needs gcc, nor a ruby app need bundler in the image, but using docker build currently while have this.

An idea I've had is specifying a dockerfile, that runs multiple docker commands when building inside it. Psuedo-ish dockerfiles below.

Docker file that builds others

FROM dockerbuilder
RUN docker build -t docker/builder myapp/builder/Dockerfile
RUN docker run -v /app:/app builder
RUN docker build -t btrepp/myapplication myapp/Dockerfile

btrepp/myapplication dockerfile

FROM debian:jessie+sayrubyruntime
ADD . /app //(this is code thats been build using the builder dockerfile
ENTRYPOINT ["rails s"]

Here we have a temporary container that does all the bundling install/package management and any build scripts, but it produces the files that the runtime container needs.

The runtime container then just adds the results of this, meaning it shouldn't need much more than ruby installed. In the case of say GCC or even better statically linked go, we may not need anything other than the core OS files to run.

That would keep the docker images super light.

Issue here is that the temporary builder container would go away at the end, meaning it would be super expensive without the ability to load a cache of sorts, we would be grabbing debian:jessie a whole heap of times.

I've seen people to certain techniques like this, but using external http servers to add the build files. I would prefer to keep it all being build by docker. Though there is possibly a way of using a docker image to do this properly. Using run and thus being able to mount volumes.

btrepp commented Oct 9, 2015

I can think of another situation.

One of the things I would like to do with my dockerfiles is not ship the build tools with the "compiled" docker file. There's no reason a C app needs gcc, nor a ruby app need bundler in the image, but using docker build currently while have this.

An idea I've had is specifying a dockerfile, that runs multiple docker commands when building inside it. Psuedo-ish dockerfiles below.

Docker file that builds others

FROM dockerbuilder
RUN docker build -t docker/builder myapp/builder/Dockerfile
RUN docker run -v /app:/app builder
RUN docker build -t btrepp/myapplication myapp/Dockerfile

btrepp/myapplication dockerfile

FROM debian:jessie+sayrubyruntime
ADD . /app //(this is code thats been build using the builder dockerfile
ENTRYPOINT ["rails s"]

Here we have a temporary container that does all the bundling install/package management and any build scripts, but it produces the files that the runtime container needs.

The runtime container then just adds the results of this, meaning it shouldn't need much more than ruby installed. In the case of say GCC or even better statically linked go, we may not need anything other than the core OS files to run.

That would keep the docker images super light.

Issue here is that the temporary builder container would go away at the end, meaning it would be super expensive without the ability to load a cache of sorts, we would be grabbing debian:jessie a whole heap of times.

I've seen people to certain techniques like this, but using external http servers to add the build files. I would prefer to keep it all being build by docker. Though there is possibly a way of using a docker image to do this properly. Using run and thus being able to mount volumes.

@fatherlinux

This comment has been minimized.

Show comment
Hide comment
@fatherlinux

fatherlinux Oct 14, 2015

Here is another example. Say I want to build a container for systemtap that has all of the debug symbols for the kernel in it (which are Yuuuuge). I have to mount the underlying /lib/modules so that the yum command knows which RPMs to install.

Furthermore, maybe I would rather have these live somewhere other than in the 1.5GB image (from the debug symbols)

I went to write a Dockerfile, then realize it was impossible :-(

docker run --privileged -v /lib/modules:/lib/modules --tty=true --interactive=true rhel7/rhel-tools /bin/bash
yum --enablerepo=rhel-7-server-debug-rpms install kernel-debuginfo-$(uname -r) kernel-devel-$(uname -r)
docker ps -a
CONTAINER ID        IMAGE                     COMMAND             CREATED             STATUS                        PORTS               NAMES
52dac30dc495        rhel7/rhel-tools:latest   "/bin/bash"         34 minutes ago      Exited (0) 15 minutes ago                         dreamy_thompson
docker commit dreamy_thompson stap:latest

https://access.redhat.com/solutions/1420883

Here is another example. Say I want to build a container for systemtap that has all of the debug symbols for the kernel in it (which are Yuuuuge). I have to mount the underlying /lib/modules so that the yum command knows which RPMs to install.

Furthermore, maybe I would rather have these live somewhere other than in the 1.5GB image (from the debug symbols)

I went to write a Dockerfile, then realize it was impossible :-(

docker run --privileged -v /lib/modules:/lib/modules --tty=true --interactive=true rhel7/rhel-tools /bin/bash
yum --enablerepo=rhel-7-server-debug-rpms install kernel-debuginfo-$(uname -r) kernel-devel-$(uname -r)
docker ps -a
CONTAINER ID        IMAGE                     COMMAND             CREATED             STATUS                        PORTS               NAMES
52dac30dc495        rhel7/rhel-tools:latest   "/bin/bash"         34 minutes ago      Exited (0) 15 minutes ago                         dreamy_thompson
docker commit dreamy_thompson stap:latest

https://access.redhat.com/solutions/1420883

@jeremyherbert

This comment has been minimized.

Show comment
Hide comment
@jeremyherbert

jeremyherbert Nov 19, 2015

I'd like to repeat my use case here from #3949 as that bug has been closed for other reasons.

I'd really like to sandbox proprietary software in docker. It's illegal for me to host it anywhere, and the download process is not realistically (or legally) able to be automated. In total, the installers come to about 22GB (and they are getting bigger with each release). I think it's silly to expect that this should be copied into the docker image at build time.

I'd like to repeat my use case here from #3949 as that bug has been closed for other reasons.

I'd really like to sandbox proprietary software in docker. It's illegal for me to host it anywhere, and the download process is not realistically (or legally) able to be automated. In total, the installers come to about 22GB (and they are getting bigger with each release). I think it's silly to expect that this should be copied into the docker image at build time.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Nov 19, 2015

Any news in this needed feature?
thank you

zrml commented Nov 19, 2015

Any news in this needed feature?
thank you

@GordonTheTurtle

This comment has been minimized.

Show comment
Hide comment
@GordonTheTurtle

GordonTheTurtle Nov 30, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@vad

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@vad

@simonvanderveldt

This comment has been minimized.

Show comment
Hide comment
@simonvanderveldt

simonvanderveldt Dec 22, 2015

+1 for this feature!

Another use case is using ssh keys at build time for building from private repos without them being stored in the layer, eliminating the need for hacks (though well engineered) such as this one: https://github.com/dockito/vault

This is our usecase as well (ssh keys rendered using tmpfs on the host in this case).

+1 for this feature!

Another use case is using ssh keys at build time for building from private repos without them being stored in the layer, eliminating the need for hacks (though well engineered) such as this one: https://github.com/dockito/vault

This is our usecase as well (ssh keys rendered using tmpfs on the host in this case).

@simonvanderveldt

This comment has been minimized.

Show comment
Hide comment
@simonvanderveldt

simonvanderveldt Dec 23, 2015

Another usecase for this is for a local cache of the node_modules directory on a CI server to reduce build-times.
npm install is very slow and even in the current "best" case where the package.json is ADDed to the image, npm install is run and only then are the actual project sources added and built on changes to package.json all dependencies have to be redownloaded again.

See npm/npm#8836 for an issue about this on the Node/npm side.

Another usecase for this is for a local cache of the node_modules directory on a CI server to reduce build-times.
npm install is very slow and even in the current "best" case where the package.json is ADDed to the image, npm install is run and only then are the actual project sources added and built on changes to package.json all dependencies have to be redownloaded again.

See npm/npm#8836 for an issue about this on the Node/npm side.

@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Dec 23, 2015

Related aspnet-docker issue regarding slow package restoration and the resulting image size of caching the current packages in the layer. Would be much better to use a mounted volume for the caching of the package.
aspnet/aspnet-docker#123

This isn't a language-specific issue, it will affect many people given that package managers are now an accepted standard.

yngndrw commented Dec 23, 2015

Related aspnet-docker issue regarding slow package restoration and the resulting image size of caching the current packages in the layer. Would be much better to use a mounted volume for the caching of the package.
aspnet/aspnet-docker#123

This isn't a language-specific issue, it will affect many people given that package managers are now an accepted standard.

@qoke

This comment has been minimized.

Show comment
Hide comment
@qoke

qoke Jan 24, 2016

The OP has nailed the issue on the head, in that "docker build -v" would greatly help decoupling the build process from the runtime environment.

I've seen several projects which now build "Mulberry harbours" which are then used to build the actual docker that is then pushed/distributed. This is overly complex from both an administration and compute resource perspective, which in turn translates to slower CI and unit testing, and overall a less productive development workflow.

qoke commented Jan 24, 2016

The OP has nailed the issue on the head, in that "docker build -v" would greatly help decoupling the build process from the runtime environment.

I've seen several projects which now build "Mulberry harbours" which are then used to build the actual docker that is then pushed/distributed. This is overly complex from both an administration and compute resource perspective, which in turn translates to slower CI and unit testing, and overall a less productive development workflow.

@btrepp

This comment has been minimized.

Show comment
Hide comment
@btrepp

btrepp Mar 16, 2016

I've been thinking about this, and the other option I can think of is the ability to mark layers as "src" layers.

Something along the lines of those layers only being accessible during a docker build, but not pulled in the resulting image file.

This way docker can cache earlier layers/images, temporary build artifacts, but these aren't required to utilize the final image.

Eg.

FROM ubuntu
RUN apt-get install gcc
ADDPRIVATE . /tmp/src <--these can be cached by docker locally
RUNPRIVATE make     <-- basically these layers become scoped to the current build process/dockerfile
RUN make install <--result of this layer is required.

Of course this means you would need to know what you are doing better, as you could very well leave critical files out.

@yngndrw
A much better solution for situations like netcore would be for them to not use HTTPS for package management, then its trivial to set up iptables+squid to have a transparent caching proxy for docker builds.My personal opinion is that these package managers should up their game, they are terrible to use in corporate environments due to ssl resigning, whereas things such as apt-get work perfectly fine and are already cacheable with iptables+squid for docker.

I can also see a downside to using build time volumes, dockerfiles won't be as reproducible, and it's going to require extra setup outside of docker build -t btrepp/myapp ., It's also going to make automated builds on dockerhub difficult.

btrepp commented Mar 16, 2016

I've been thinking about this, and the other option I can think of is the ability to mark layers as "src" layers.

Something along the lines of those layers only being accessible during a docker build, but not pulled in the resulting image file.

This way docker can cache earlier layers/images, temporary build artifacts, but these aren't required to utilize the final image.

Eg.

FROM ubuntu
RUN apt-get install gcc
ADDPRIVATE . /tmp/src <--these can be cached by docker locally
RUNPRIVATE make     <-- basically these layers become scoped to the current build process/dockerfile
RUN make install <--result of this layer is required.

Of course this means you would need to know what you are doing better, as you could very well leave critical files out.

@yngndrw
A much better solution for situations like netcore would be for them to not use HTTPS for package management, then its trivial to set up iptables+squid to have a transparent caching proxy for docker builds.My personal opinion is that these package managers should up their game, they are terrible to use in corporate environments due to ssl resigning, whereas things such as apt-get work perfectly fine and are already cacheable with iptables+squid for docker.

I can also see a downside to using build time volumes, dockerfiles won't be as reproducible, and it's going to require extra setup outside of docker build -t btrepp/myapp ., It's also going to make automated builds on dockerhub difficult.

@zrml

This comment has been minimized.

Show comment
Hide comment
@zrml

zrml Mar 16, 2016

@btrepp: I like your suggestion. I could even live for my use cases with an hardcoded (I know it's generally a bad thing) TMP dir that Docker tells us about so that they know when they build the final artifact from all the layers that they can forget/leave out the one mounted on /this_is_the_tmp_explosion_folder_that_will_be_removed_from_your_final_container_image
easy enough....

zrml commented Mar 16, 2016

@btrepp: I like your suggestion. I could even live for my use cases with an hardcoded (I know it's generally a bad thing) TMP dir that Docker tells us about so that they know when they build the final artifact from all the layers that they can forget/leave out the one mounted on /this_is_the_tmp_explosion_folder_that_will_be_removed_from_your_final_container_image
easy enough....

@yngndrw

This comment has been minimized.

Show comment
Hide comment
@yngndrw

yngndrw Mar 24, 2016

@btrepp I quite like your source layer idea.

However regarding package managers not using SSL, I would have to disagree.

If you were to want to cache packages like that, then you should probably use a (Local) private package feed instead which mirrors the official source. Reverting to HTTP seems like a bad idea to me, especially given that a lot of package managers don't seem to sign their packages and therefore rely on HTTPS.

yngndrw commented Mar 24, 2016

@btrepp I quite like your source layer idea.

However regarding package managers not using SSL, I would have to disagree.

If you were to want to cache packages like that, then you should probably use a (Local) private package feed instead which mirrors the official source. Reverting to HTTP seems like a bad idea to me, especially given that a lot of package managers don't seem to sign their packages and therefore rely on HTTPS.

@sergle

This comment has been minimized.

Show comment
Hide comment
@sergle

sergle Mar 24, 2016

There is a tool grammarly/rocker which can be used while this issue is not yet fixed.

sergle commented Mar 24, 2016

There is a tool grammarly/rocker which can be used while this issue is not yet fixed.

@BryanHunt

This comment has been minimized.

Show comment
Hide comment
@BryanHunt

BryanHunt Oct 5, 2017

A solution we are about to try is to build an image by building our shared libraries and retaining the dependency cache. This image would then become our build image for our apps. It's not ideal, but we think it's worth a try.

A solution we are about to try is to build an image by building our shared libraries and retaining the dependency cache. This image would then become our build image for our apps. It's not ideal, but we think it's worth a try.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 7, 2017

Member

There is an issue though, whenever the list of dependencies is changed COPY pom.xml in the stage 0 image causes the layer to be ditched out and thus the whole cache is gone. That meant that whenever a developer change anything in the pom (a comment, a 1kBytes dependency) the whole cache gotta be redownloaded again.

@hashar note that the COPY --from feature is not limited to build-stages; from the Dockerfile reference:

Optionally COPY accepts a flag --from=<name|index> that can be used to set the source location to a previous build stage (created with FROM .. AS <name>) that will be used instead of a build context sent by the user. The flag also accepts a numeric index assigned for all previous build stages started with FROM instruction. In case a build stage with a specified name can’t be found an image with the same name is attempted to be used instead.

This allows you to build an image for your dependencies, tag it, and use that to copy your dependencies from. For example:

FROM maven
WORKDIR /usr/src/app
# /root/.m2 is a volume :(
ENV MAVEN_OPTS=-Dmaven.repo.local=../m2repo/
COPY pom.xml .
# v2.8 doesn't work :(
RUN mvn -B -e -C -T 1C org.apache.maven.plugins:maven-dependency-plugin:3.0.2:go-offline
COPY . .
RUN mvn -B -e -o -T 1C verify
docker build -t dependencies:1.0.0 .

And specify using the dependencies:1.0.0 image for your dependencies;

FROM openjdk
COPY --from=dependencies:1.0.0 /usr/src/app/target/*.jar ./

Or (just a very basic example to test);

$ mkdir example && cd example
$ touch dep-one.jar dep-two.jar dep-three.jar

$ docker build -t dependencies:1.0.0 . -f -<<'EOF'
FROM scratch
COPY . /usr/src/app/target/
EOF

$ docker build -t myimage -<<'EOF'
FROM busybox
RUN mkdir /foo
COPY --from=dependencies:1.0.0 /usr/src/app/target/*.jar /foo/
RUN ls -la /foo/
EOF

In the output of the build, you'll see:

Step 4/4 : RUN ls -la /foo/
 ---> Running in 012a8dbef91d
total 8
drwxr-xr-x    1 root     root          4096 Oct  7 13:27 .
drwxr-xr-x    1 root     root          4096 Oct  7 13:27 ..
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-one.jar
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-three.jar
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-two.jar
 ---> 71fc7f4b8802
Member

thaJeztah commented Oct 7, 2017

There is an issue though, whenever the list of dependencies is changed COPY pom.xml in the stage 0 image causes the layer to be ditched out and thus the whole cache is gone. That meant that whenever a developer change anything in the pom (a comment, a 1kBytes dependency) the whole cache gotta be redownloaded again.

@hashar note that the COPY --from feature is not limited to build-stages; from the Dockerfile reference:

Optionally COPY accepts a flag --from=<name|index> that can be used to set the source location to a previous build stage (created with FROM .. AS <name>) that will be used instead of a build context sent by the user. The flag also accepts a numeric index assigned for all previous build stages started with FROM instruction. In case a build stage with a specified name can’t be found an image with the same name is attempted to be used instead.

This allows you to build an image for your dependencies, tag it, and use that to copy your dependencies from. For example:

FROM maven
WORKDIR /usr/src/app
# /root/.m2 is a volume :(
ENV MAVEN_OPTS=-Dmaven.repo.local=../m2repo/
COPY pom.xml .
# v2.8 doesn't work :(
RUN mvn -B -e -C -T 1C org.apache.maven.plugins:maven-dependency-plugin:3.0.2:go-offline
COPY . .
RUN mvn -B -e -o -T 1C verify
docker build -t dependencies:1.0.0 .

And specify using the dependencies:1.0.0 image for your dependencies;

FROM openjdk
COPY --from=dependencies:1.0.0 /usr/src/app/target/*.jar ./

Or (just a very basic example to test);

$ mkdir example && cd example
$ touch dep-one.jar dep-two.jar dep-three.jar

$ docker build -t dependencies:1.0.0 . -f -<<'EOF'
FROM scratch
COPY . /usr/src/app/target/
EOF

$ docker build -t myimage -<<'EOF'
FROM busybox
RUN mkdir /foo
COPY --from=dependencies:1.0.0 /usr/src/app/target/*.jar /foo/
RUN ls -la /foo/
EOF

In the output of the build, you'll see:

Step 4/4 : RUN ls -la /foo/
 ---> Running in 012a8dbef91d
total 8
drwxr-xr-x    1 root     root          4096 Oct  7 13:27 .
drwxr-xr-x    1 root     root          4096 Oct  7 13:27 ..
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-one.jar
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-three.jar
-rw-r--r--    1 root     root             0 Oct  7 13:26 dep-two.jar
 ---> 71fc7f4b8802
@dustinlacewell-wk

This comment has been minimized.

Show comment
Hide comment
@dustinlacewell-wk

dustinlacewell-wk Oct 9, 2017

I don't know if anyone has mentioned this use-case yet (I briefly searched the page) but mounting an SSH auth socket into the build container would make utilizing dependencies that deployed via private git repositories much easier. There would be less of a need for boilerplate inside the Dockerfile regarding copying around keys in non-final build stages, etc.

I don't know if anyone has mentioned this use-case yet (I briefly searched the page) but mounting an SSH auth socket into the build container would make utilizing dependencies that deployed via private git repositories much easier. There would be less of a need for boilerplate inside the Dockerfile regarding copying around keys in non-final build stages, etc.

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@AkihiroSuda

AkihiroSuda Oct 9, 2017

Member

buildkit has native support for git
https://github.com/moby/buildkit

Member

AkihiroSuda commented Oct 9, 2017

buildkit has native support for git
https://github.com/moby/buildkit

@kinnalru

This comment has been minimized.

Show comment
Hide comment
@kinnalru

kinnalru Nov 30, 2017

Solving.
Create bash script(~/bin/docker-compose or like):

#!/bin/bash

trap 'kill $(jobs -p)' EXIT
socat TCP-LISTEN:56789,reuseaddr,fork UNIX-CLIENT:${SSH_AUTH_SOCK} &

/usr/bin/docker-compose $@

And in Dockerfile using socat:

...
ENV SSH_AUTH_SOCK /tmp/auth.sock
...
  && apk add --no-cache socat openssh \
  && /bin/sh -c "socat -v UNIX-LISTEN:${SSH_AUTH_SOCK},unlink-early,mode=777,fork TCP:172.22.1.11:56789 &> /dev/null &" \
  && bundle install \
...
or any other ssh commands will works

Then run docker-compose build

Solving.
Create bash script(~/bin/docker-compose or like):

#!/bin/bash

trap 'kill $(jobs -p)' EXIT
socat TCP-LISTEN:56789,reuseaddr,fork UNIX-CLIENT:${SSH_AUTH_SOCK} &

/usr/bin/docker-compose $@

And in Dockerfile using socat:

...
ENV SSH_AUTH_SOCK /tmp/auth.sock
...
  && apk add --no-cache socat openssh \
  && /bin/sh -c "socat -v UNIX-LISTEN:${SSH_AUTH_SOCK},unlink-early,mode=777,fork TCP:172.22.1.11:56789 &> /dev/null &" \
  && bundle install \
...
or any other ssh commands will works

Then run docker-compose build

@nigelgbanks

This comment has been minimized.

Show comment
Hide comment
@nigelgbanks

nigelgbanks Jan 8, 2018

To throw another use case on the pile. I use Docker for Windows to generate a filesystem for building embedded linux system in one container and I would like to share this with other containers during their build step. I interact with this container changing configuration and rebuilding etc so performing the build in a Dockerfile and using multi-stage builds isn't a really good fit as I would lose incremental builds. I want to cache my previous build artefacts as it takes about 1.5 hours to do a clean build. Due to the way Windows deals with symbolic links I can't do my build into a host mounted volume so I use named volumes. Ideally I would like to share these named volumes in the build steps of my other images; as at the moment I have to create a tar of the build output (about 4gb) then do a docker copy to make it available on the windows host for subsequent builds.

To throw another use case on the pile. I use Docker for Windows to generate a filesystem for building embedded linux system in one container and I would like to share this with other containers during their build step. I interact with this container changing configuration and rebuilding etc so performing the build in a Dockerfile and using multi-stage builds isn't a really good fit as I would lose incremental builds. I want to cache my previous build artefacts as it takes about 1.5 hours to do a clean build. Due to the way Windows deals with symbolic links I can't do my build into a host mounted volume so I use named volumes. Ideally I would like to share these named volumes in the build steps of my other images; as at the moment I have to create a tar of the build output (about 4gb) then do a docker copy to make it available on the windows host for subsequent builds.

@wtayyeb

This comment has been minimized.

Show comment
Hide comment
@wtayyeb

wtayyeb Jan 9, 2018

In case of python, when we pip install package it and its dependencies are downloaded to a cache folder and then installed to site-packages.
As a good practice we use pip --no-cache-dir install package to not store rubbish/cache in current layer. But for best practice it is desired to put the cache folder out of build context. so build time -v will help.
some users above mentioned to use COPY . /somewhere/in/container/ it is OK for user app or files but not for cache. because COPY creates one more layer as its own and removing caches in later layers will not be useful. other bad side effect is if cache changed when we use COPY the context changed and following layers will invalidate and forced to rebuild.

wtayyeb commented Jan 9, 2018

In case of python, when we pip install package it and its dependencies are downloaded to a cache folder and then installed to site-packages.
As a good practice we use pip --no-cache-dir install package to not store rubbish/cache in current layer. But for best practice it is desired to put the cache folder out of build context. so build time -v will help.
some users above mentioned to use COPY . /somewhere/in/container/ it is OK for user app or files but not for cache. because COPY creates one more layer as its own and removing caches in later layers will not be useful. other bad side effect is if cache changed when we use COPY the context changed and following layers will invalidate and forced to rebuild.

@manishtomar

This comment has been minimized.

Show comment
Hide comment
@manishtomar

manishtomar Jan 9, 2018

@wtayyeb If you have Dockerfile which runs pip install ... only when requirements file changes then build time -v doesn't seem that important since requirements do not change as often as applications do when building.

manishtomar commented Jan 9, 2018

@wtayyeb If you have Dockerfile which runs pip install ... only when requirements file changes then build time -v doesn't seem that important since requirements do not change as often as applications do when building.

@ibukanov

This comment has been minimized.

Show comment
Hide comment
@ibukanov

ibukanov Jan 9, 2018

@wtayyeb You can use multi-stage Dockerfile to have both the cache and a lean image. That is, use an installer image to install python into some directory and then for your final image use COPY --from to transfer only the necessary python files without any installation artifacts or even pip itself.

ibukanov commented Jan 9, 2018

@wtayyeb You can use multi-stage Dockerfile to have both the cache and a lean image. That is, use an installer image to install python into some directory and then for your final image use COPY --from to transfer only the necessary python files without any installation artifacts or even pip itself.

@wtayyeb

This comment has been minimized.

Show comment
Hide comment
@wtayyeb

wtayyeb Jan 9, 2018

@manishtomar, Thanks, Yes and No! In clean case all dependencies are downloaded again and build and converted to wheels and cached, then installed into destination environment. So if one put requirements in there that is one time job. But if one tiny dependency is updated, all the dependencies must to re-downloaded, re-build and re-wheeled and re-cached to be usable.
When using a CI to build and test your libraries and your applications in a matrix of several jobs, multiply above work in number of concurrent jobs in your CI server and will get iowait raising to more than 3s and load average above 15 even with SSDs. (these numbers are real for 2 concurrent builds and app with ~20 dependencies) I think pip cache is doing it in the right way, avoiding re-downloading, re-building and re-wheeling the ready packages. and without bind -v we loose time and server resources.

@ibukanov, Thanks. I am using multi-stage Dockerfile for building my apps packages and use them later. It would help if I have only one Dockerfile and want to build it several times, but what if there be several Dockerfiles and each one is build against a python version (2.7,3.6 for now) and also have several c-extensions that need to be build for selected base image? what about above paragraph?

wtayyeb commented Jan 9, 2018

@manishtomar, Thanks, Yes and No! In clean case all dependencies are downloaded again and build and converted to wheels and cached, then installed into destination environment. So if one put requirements in there that is one time job. But if one tiny dependency is updated, all the dependencies must to re-downloaded, re-build and re-wheeled and re-cached to be usable.
When using a CI to build and test your libraries and your applications in a matrix of several jobs, multiply above work in number of concurrent jobs in your CI server and will get iowait raising to more than 3s and load average above 15 even with SSDs. (these numbers are real for 2 concurrent builds and app with ~20 dependencies) I think pip cache is doing it in the right way, avoiding re-downloading, re-building and re-wheeling the ready packages. and without bind -v we loose time and server resources.

@ibukanov, Thanks. I am using multi-stage Dockerfile for building my apps packages and use them later. It would help if I have only one Dockerfile and want to build it several times, but what if there be several Dockerfiles and each one is build against a python version (2.7,3.6 for now) and also have several c-extensions that need to be build for selected base image? what about above paragraph?

@thedrow

This comment has been minimized.

Show comment
Hide comment
@thedrow

thedrow Feb 6, 2018

@thaJeztah You suggestion is great and it will save us some time, however in the case of build caches we really don't want to have to copy anything from the other image.
Why can't we access another image without copying it?

thedrow commented Feb 6, 2018

@thaJeztah You suggestion is great and it will save us some time, however in the case of build caches we really don't want to have to copy anything from the other image.
Why can't we access another image without copying it?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 6, 2018

Member

@thedrow my example was with the features that are currently there; have a look at the RUN --mount proposal (#32507), which may be a closer fit to your use case

Member

thaJeztah commented Feb 6, 2018

@thedrow my example was with the features that are currently there; have a look at the RUN --mount proposal (#32507), which may be a closer fit to your use case

@kbaegis

This comment has been minimized.

Show comment
Hide comment
@kbaegis

kbaegis Feb 23, 2018

Reading the above thread I see a large number of people trying to find kludges to fix a basic functionality gap in the docker build process. I see no compelling arguments from the basis of portability without necessarily conflating host mounts with image mounts- arguments which are frankly specious and lazy.

I am also a gentoo container user and was redirected from #3156 which is a completely valid use case for this missing functionality.

All I really want is the ability to mount the contents of another image at build-time so that I don't bloat my images.

kbaegis commented Feb 23, 2018

Reading the above thread I see a large number of people trying to find kludges to fix a basic functionality gap in the docker build process. I see no compelling arguments from the basis of portability without necessarily conflating host mounts with image mounts- arguments which are frankly specious and lazy.

I am also a gentoo container user and was redirected from #3156 which is a completely valid use case for this missing functionality.

All I really want is the ability to mount the contents of another image at build-time so that I don't bloat my images.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 23, 2018

Member

@kbaegis sounds like an exact match with the feature that's proposed in #32507

Member

thaJeztah commented Feb 23, 2018

@kbaegis sounds like an exact match with the feature that's proposed in #32507

@kbaegis

This comment has been minimized.

Show comment
Hide comment
@kbaegis

kbaegis Feb 24, 2018

Sure. That one's only been an unimplemented P3 in the backlog for one year rather than 3 years.

It looks like https://github.com/projectatomic/buildah is actually going to outstrip docker build pretty quickly here for this basic functionality. I think I'm just going to switch my pipeline over once that happens.

kbaegis commented Feb 24, 2018

Sure. That one's only been an unimplemented P3 in the backlog for one year rather than 3 years.

It looks like https://github.com/projectatomic/buildah is actually going to outstrip docker build pretty quickly here for this basic functionality. I think I'm just going to switch my pipeline over once that happens.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 24, 2018

Member

@kbaegis what did you come here to add to this discussion? You described a use-case that exactly matches a different proposal;

All I really want is the ability to mount the contents of another image at build-time so that I don't bloat my images.

It’s open-source, things don’t come to existence magically.

Member

thaJeztah commented Feb 24, 2018

@kbaegis what did you come here to add to this discussion? You described a use-case that exactly matches a different proposal;

All I really want is the ability to mount the contents of another image at build-time so that I don't bloat my images.

It’s open-source, things don’t come to existence magically.

@kbaegis

This comment has been minimized.

Show comment
Hide comment
@kbaegis

kbaegis Feb 24, 2018

What am I looking to add to the discussion?

Succinctly that I'm moving on from this toolset. I'm sure that's valuable information for the development team as I'm sure I'm not alone there.

The glacial speed and low priority for supporting this use case (and any reliable workaround that provides this functionality) has forced me onto other tools and that I'm abandoning this build pipeline due to missing functionality.

kbaegis commented Feb 24, 2018

What am I looking to add to the discussion?

Succinctly that I'm moving on from this toolset. I'm sure that's valuable information for the development team as I'm sure I'm not alone there.

The glacial speed and low priority for supporting this use case (and any reliable workaround that provides this functionality) has forced me onto other tools and that I'm abandoning this build pipeline due to missing functionality.

@draeath

This comment has been minimized.

Show comment
Hide comment
@draeath

draeath Feb 27, 2018

I've got a (rehash, I'm sure) use case to add. #32507 may suit this better.

I'm building a docker image for some bioinformatics pipelines. A few of the tools require some databases to be present prior to their compilation/installation (please don't ask, it's not my code). These databases weigh in at a lovely 30gb minimum.

During runtime, I certainly intend for those databases to be mounted -v volumes. Unfortunately, I cannot do this during the build process without "baking" them in, resulting in a rather obscenely sized image.

draeath commented Feb 27, 2018

I've got a (rehash, I'm sure) use case to add. #32507 may suit this better.

I'm building a docker image for some bioinformatics pipelines. A few of the tools require some databases to be present prior to their compilation/installation (please don't ask, it's not my code). These databases weigh in at a lovely 30gb minimum.

During runtime, I certainly intend for those databases to be mounted -v volumes. Unfortunately, I cannot do this during the build process without "baking" them in, resulting in a rather obscenely sized image.

@lig

This comment has been minimized.

Show comment
Hide comment
@lig

lig Feb 28, 2018

@draeath take a look at the https://github.com/grammarly/rocker . It already supports a lovely MOUNT instruction.

lig commented Feb 28, 2018

@draeath take a look at the https://github.com/grammarly/rocker . It already supports a lovely MOUNT instruction.

@fatherlinux

This comment has been minimized.

Show comment
Hide comment
@fatherlinux

fatherlinux Feb 28, 2018

@draeath also, check out Buildah, it supports mounts by default because it is set up more like a programming tool. Also supports mounts with a Dockerfile:

https://github.com/projectatomic/buildah

@draeath also, check out Buildah, it supports mounts by default because it is set up more like a programming tool. Also supports mounts with a Dockerfile:

https://github.com/projectatomic/buildah

@draeath

This comment has been minimized.

Show comment
Hide comment
@draeath

draeath Feb 28, 2018

Thank you both @fatherlinux and @lig - this will help me get my task done. I still think I shouldn't have to stray outside the project to do it, though, and would still love to see this and #32507 implemented ;)

draeath commented Feb 28, 2018

Thank you both @fatherlinux and @lig - this will help me get my task done. I still think I shouldn't have to stray outside the project to do it, though, and would still love to see this and #32507 implemented ;)

@ymc-dabe ymc-dabe referenced this issue in docker/docker-py Mar 1, 2018

Closed

build method is missing a volumes argument #1498

@smacdonald-miov

This comment has been minimized.

Show comment
Hide comment
@smacdonald-miov

smacdonald-miov Mar 1, 2018

I've come here via some googling to ask for the same feature, volumes at 'docker build' time, not 'docker run' time.

We have an embedded system that contains a CPU. The manufacturer provides tooling to compose a system image, and then transfer the image into the CPU. This tooling is 3rd party to me and I cannot change it. The manufacturer is also unlikely to alter it at my request.

I want to build a docker image that does a first pass "build the firmware image", and then be able to spawn containers that just push the firmware image to the fresh-off-the-line PCBs. A Dockerfile might look like:
----------[ Cut Here ]----------
FROM base-image as builder
COPY src src
RUN build-src

FROM base-image as flasher
COPY --from=builder build-artifacts
RUN cpu-build-and-flash --build-only
----------[ Cut Here ]----------
Unfortunately, the cpu-build-and-flash step requires access to the target device via USB bus, even though it's not going to push the firmware image to the device. Thus I need to take the '-v /dev/usb/bus:/dev/usb/bus' from the 'docker run' command and have it in the build instead.

It's clear that this isn't currently possible.

The workaround I'm going ahead with is to manually create a flashing image by 'docker container commit'ing a container to an image. I'd much rather just mount the USB bus at build time.

I've come here via some googling to ask for the same feature, volumes at 'docker build' time, not 'docker run' time.

We have an embedded system that contains a CPU. The manufacturer provides tooling to compose a system image, and then transfer the image into the CPU. This tooling is 3rd party to me and I cannot change it. The manufacturer is also unlikely to alter it at my request.

I want to build a docker image that does a first pass "build the firmware image", and then be able to spawn containers that just push the firmware image to the fresh-off-the-line PCBs. A Dockerfile might look like:
----------[ Cut Here ]----------
FROM base-image as builder
COPY src src
RUN build-src

FROM base-image as flasher
COPY --from=builder build-artifacts
RUN cpu-build-and-flash --build-only
----------[ Cut Here ]----------
Unfortunately, the cpu-build-and-flash step requires access to the target device via USB bus, even though it's not going to push the firmware image to the device. Thus I need to take the '-v /dev/usb/bus:/dev/usb/bus' from the 'docker run' command and have it in the build instead.

It's clear that this isn't currently possible.

The workaround I'm going ahead with is to manually create a flashing image by 'docker container commit'ing a container to an image. I'd much rather just mount the USB bus at build time.

@kbaegis

This comment has been minimized.

Show comment
Hide comment
@kbaegis

kbaegis Mar 2, 2018

Update for any who are interested: I've recently rebuilt my entire pipe successfully with buildah. I've currently got the two build pipelines running in parallel and the oci/buildah pipeline is generating smaller images (specifically removing /usr/portage in my case by masking it with another mount).

kbaegis commented Mar 2, 2018

Update for any who are interested: I've recently rebuilt my entire pipe successfully with buildah. I've currently got the two build pipelines running in parallel and the oci/buildah pipeline is generating smaller images (specifically removing /usr/portage in my case by masking it with another mount).

lig referenced this issue in grammarly/rocker Mar 12, 2018

@socketpair

This comment has been minimized.

Show comment
Hide comment
@socketpair

socketpair Apr 28, 2018

And finally this feature is here: docker/docker-py#1498

And finally this feature is here: docker/docker-py#1498

@graingert

This comment has been minimized.

Show comment
Hide comment
@graingert

graingert Apr 28, 2018

Contributor
Contributor

graingert commented Apr 28, 2018

@mcattle

This comment has been minimized.

Show comment
Hide comment
@mcattle

mcattle May 17, 2018

I'd also like to see this feature (with write capabilities) so that a unit test results file can be exported during the multistage build process in a CI pipeline. To keep with the spirit of build portability, if the -v switch was not provided, the file would simply be written internally within the test image at that stage.

The ideal goal is to build once, test once, and still have the results file given to the host system, even in the event (and especially in the event) that tests fail, stopping the build.

mcattle commented May 17, 2018

I'd also like to see this feature (with write capabilities) so that a unit test results file can be exported during the multistage build process in a CI pipeline. To keep with the spirit of build portability, if the -v switch was not provided, the file would simply be written internally within the test image at that stage.

The ideal goal is to build once, test once, and still have the results file given to the host system, even in the event (and especially in the event) that tests fail, stopping the build.

@shanselman

This comment has been minimized.

Show comment
Hide comment
@shanselman

shanselman May 18, 2018

Yes please. All day.

Yes please. All day.

@hoffa

This comment has been minimized.

Show comment
Hide comment
@hoffa

hoffa May 21, 2018

Not entirely relevant, but we're migrating a part of our deployment infrastructure and needed a way to copy files from an image after build. The following did the trick:

docker build -t x .
ID=$(docker create x)
docker cp $ID:/package.deb .
docker rm $ID

hoffa commented May 21, 2018

Not entirely relevant, but we're migrating a part of our deployment infrastructure and needed a way to copy files from an image after build. The following did the trick:

docker build -t x .
ID=$(docker create x)
docker cp $ID:/package.deb .
docker rm $ID
@TqrHsn

This comment has been minimized.

Show comment
Hide comment
@TqrHsn

TqrHsn May 23, 2018

It should have already been added when multistage docker file was introduced. Eventually everyone is gonna face this issue as soon as they are gonna start running unit tests as a stage in multistage docker file specially in case of CI build pipelines. We are also facing this issue where we have to publish unit test reports to VSTS. Already applying the workaround @hoffa has mentioned. But after all it is a workaround and making the things complicated.

TqrHsn commented May 23, 2018

It should have already been added when multistage docker file was introduced. Eventually everyone is gonna face this issue as soon as they are gonna start running unit tests as a stage in multistage docker file specially in case of CI build pipelines. We are also facing this issue where we have to publish unit test reports to VSTS. Already applying the workaround @hoffa has mentioned. But after all it is a workaround and making the things complicated.

@ajbouh

This comment has been minimized.

Show comment
Hide comment
@ajbouh

ajbouh May 23, 2018

Should we make a different issue for people that want/need build-time volumes for a build cache?

ajbouh commented May 23, 2018

Should we make a different issue for people that want/need build-time volumes for a build cache?

@AkihiroSuda

This comment has been minimized.

Show comment
Hide comment
@ajbouh

This comment has been minimized.

Show comment
Hide comment
@ajbouh

ajbouh May 24, 2018

ajbouh commented May 24, 2018

@chrisaxiom chrisaxiom referenced this issue in axiomzen/zest May 25, 2018

Open

Zest 2.0 #43

@chuckadams

This comment has been minimized.

Show comment
Hide comment
@chuckadams

chuckadams Jun 17, 2018

While you can't add volumes at build-time, you can add hosts, so I now build all my docker images with something like --add-host yum-mirror:$MIRROR_IP which serves up a yum mirror which my build images then detect via a wrapper around yum. Handy when my project changes dependencies many times a day and I'm offline or on a bad connection (part of the project involves updating and cleaning up its many deps).

I find Docker's resistance to solving this problem infuriating.

chuckadams commented Jun 17, 2018

While you can't add volumes at build-time, you can add hosts, so I now build all my docker images with something like --add-host yum-mirror:$MIRROR_IP which serves up a yum mirror which my build images then detect via a wrapper around yum. Handy when my project changes dependencies many times a day and I'm offline or on a bad connection (part of the project involves updating and cleaning up its many deps).

I find Docker's resistance to solving this problem infuriating.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jun 17, 2018

Contributor

Experimental support for buildkit was recently merged, with that comes with an option to RUN --mount=<opts> <command>.

Contributor

cpuguy83 commented Jun 17, 2018

Experimental support for buildkit was recently merged, with that comes with an option to RUN --mount=<opts> <command>.

@glensc

This comment has been minimized.

Show comment
Hide comment
@glensc

glensc Jun 17, 2018

Contributor
Contributor

glensc commented Jun 17, 2018

@agolomoodysaada

This comment has been minimized.

Show comment
Hide comment
@agolomoodysaada

agolomoodysaada Jun 18, 2018

@glensc @cpuguy83 When can we expect a release for this merged feature?

agolomoodysaada commented Jun 18, 2018

@glensc @cpuguy83 When can we expect a release for this merged feature?

@ekovacs

This comment has been minimized.

Show comment
Hide comment

ekovacs commented Jun 22, 2018

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment