Let Dockerfile ADD pick file from another docker image

[ndeloof]

I use Dockerfile to build my project from sources. But I don't want my production Docker image to include sources, build and test tools, so I have a second Dockerfile for this purpose to COPY or ADD the built artifact that I need to extract some way from my first "build-from-sources" Dockerfile.

This feature request is about letting ADD command support a docker:// scheme so I can point to a file/directory from another docker image

ADD docker://build-from-sources:latest/home/build/target/myapp.war /var/app

[borromeotlhs]

When you build from source in your docker file, add a line to CPU the build
artifacts to some internal directory to your source building container.
Then, using the link feature and the fact that you exposed that directory
with run's -w option, you can easily might the post build artifacts into
any container that depends on them.

See http://docs.docker.com/userguide/dockervolumes/ for a step by step
example.

R/
TJ
On Sep 11, 2014 2:30 AM, "Nicolas De loof" notifications@github.com wrote:

I use Dockerfile to build my project from sources. But I don't want my
production Docker image to include sources, build and test tools, so I have
a second Dockerfile for this purpose to COPY or ADD the built artifact that
I need to extract some way from my first "build-from-sources" Dockerfile.

This feature request is about letting ADD command support a docker://
scheme so I can point to a file/directory from another docker image

ADD docker://build-from-sources:latest/home/build/target/myapp.war /var/app

—
Reply to this email directly or view it on GitHub
#7992.

[ndeloof]

Thanks @borromeotlhs for quick reply, I just don't understand the proposed tip.
So you suggest my "build-from-source" Dockerfile group all to-be-exported artifact on a specific directory, ok. I don't see how the working directory flag -w would impact exposed directory. I don't understand how volumes can help here. Volume sonly apply to containers, here I'm asking for a feature about Docker images building, without any running container.

[borromeotlhs]

Well, I firstly want to point out, you would use the -v option, not -w.
Apologies if I mistyped.

Besides that, the reason it'd be really convenient to use docker containers
to build your source, and then expose the artifacts to another container
are many. Off the top, you can ensure that the process to build your
source remains constant ( save for the syncing to a git repo, which should
also be constant save for passing in a newer tag ). You can then also, if
your concerned about packaging the app or binary that you built, it ends up
providing you with a lot of interesting ways to deliver and deploy the
built artifact as you'll be packaging the artifacts up in order to use
them, the artifacts end up being already containerized. You've just, by
insisting on containerizing your workflow, enable quick deployment of a
containerized app!

Hope that helps.
On Sep 11, 2014 2:54 AM, "Nicolas De loof" notifications@github.com wrote:

Thanks @borromeotlhs https://github.com/borromeotlhs for quick reply, I
just don't understand the proposed tip.
So you suggest my "build-from-source" Dockerfile group all to-be-exported
artifact on a specific directory, ok. I don't see how the working directory
flag -w would impact exposed directory. I don't understand how volumes can
help here. Volume sonly apply to containers, here I'm asking for a feature
about Docker images building, without any running container.

—
Reply to this email directly or view it on GitHub
#7992 (comment).

[ndeloof]

What I'm suggesting here is a "trusted build" compliant way to get a docker image reusing some elements from another one, not a workaround. I only know some of them, including -v, or just running docker cp to retrieve built artifacts. But then I need to use host filesystem as a temporary directory during my build process, and I can't guarantee the production image has been built with artifact from a specific build-from-source docker image.

[borromeotlhs]

If your container that built your source exists ( you didn't docker rmi it) and the build artifacts are stored within it, you can easily start the container back up at the exact commit, name it, expose the volume, and then run your other container to grab the exact build artifact from the trusted container ( which,BTW, is all stored on host via aufs,btrfs, etc. anyway )

[ndeloof]

Sure it's actually stored by host, but you don't get the idea. If I want to use trusted builds service, I can't rely on some manual steps to run container and do such things. I'm ok this is a fully valid workaround (this is the way I do today), but doing this I rely on the command I run on docker host to get my production container include the expected artifact, not on specified build-from-source docker image tag I would declare in my production Dockerfile. My issue here is about trusting the produced production image and traceability from source.

[wking]

On Thu, Sep 11, 2014 at 03:52:43PM -0700, Nicolas De loof wrote:

What I'm suggesting here is a "trusted build" compliant way to get a
docker image reusing some elements from another one, not a
workaround.

For previous discussions of this sort of “pure Dockerfile build
without bundling the build tooling”, see #3156 and #3949. For an
official image that's doesn't contain it's build toolchain, see the
rootfs.tar gymnastics used for busybox 1. Since #3949 is still
open, I think this issue should be closed in favor of #3949.

[ndeloof]

#3949 VOLUME-FROM-IMG is indeed addressing a comparable use-case, #3949 (comment) describes this exact suggestion.

[ndeloof]

Reading #3949 with more care, the suggested feature is to mount volumes from another image, but not make them persistent in the produced image. This let you access some third party stuff required to build a Dockerfile but that you want to exclude from produced image. This can help for my use-case but with a distinct approach.

[prologic]

/cc @therealprologic

[SvenDowideit]

see #7115, #7149 and #8021 too.

[vbatts]

See #4933 too

[unclejack]

The ADD instruction is probably not going to be extended any further. ADD has quite a bit of magic in it for extracting tarballs already and adding more magic isn't desirable.

Functionality like what's being discussed here is indeed useful and needed in many cases, but I think the existing open issues are going to implement something which adds this functionality.

I'll close this issue in favor of #7115, #7149 and #8021.

[ndeloof]

This makes me wonder how ADD is configured to support various protocols. Seems it natively support http/https/ftp, which other protocols ? Can I plug more ?

[unclejack]

@ndeloof It's not wired to do something like that, it just has the code to handle those. However, the problem is that it's too magical right now. It handles remote files and local files. In additional to fetching remote files and local files, it's also extracting local tarballs.

Making it add files from another image would add even more magic and would make that code even worse. That's why we've introduced COPY, to start cleaning things up and maybe get rid of ADD at some point in the future.

[jasimmk]

The functionality can be implemented using COPY command

COPY --from=hadolint/hadolint /bin/hadolint .