Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let Dockerfile ADD pick file from another docker image #7992

Closed
ndeloof opened this issue Sep 11, 2014 · 16 comments
Closed

Let Dockerfile ADD pick file from another docker image #7992

ndeloof opened this issue Sep 11, 2014 · 16 comments
Labels
area/builder kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny

Comments

@ndeloof
Copy link
Contributor

ndeloof commented Sep 11, 2014

I use Dockerfile to build my project from sources. But I don't want my production Docker image to include sources, build and test tools, so I have a second Dockerfile for this purpose to COPY or ADD the built artifact that I need to extract some way from my first "build-from-sources" Dockerfile.

This feature request is about letting ADD command support a docker:// scheme so I can point to a file/directory from another docker image

ADD docker://build-from-sources:latest/home/build/target/myapp.war /var/app
@borromeotlhs
Copy link

When you build from source in your docker file, add a line to CPU the build
artifacts to some internal directory to your source building container.
Then, using the link feature and the fact that you exposed that directory
with run's -w option, you can easily might the post build artifacts into
any container that depends on them.

See http://docs.docker.com/userguide/dockervolumes/ for a step by step
example.

R/
TJ
On Sep 11, 2014 2:30 AM, "Nicolas De loof" notifications@github.com wrote:

I use Dockerfile to build my project from sources. But I don't want my
production Docker image to include sources, build and test tools, so I have
a second Dockerfile for this purpose to COPY or ADD the built artifact that
I need to extract some way from my first "build-from-sources" Dockerfile.

This feature request is about letting ADD command support a docker://
scheme so I can point to a file/directory from another docker image

ADD docker://build-from-sources:latest/home/build/target/myapp.war /var/app


Reply to this email directly or view it on GitHub
#7992.

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 11, 2014

Thanks @borromeotlhs for quick reply, I just don't understand the proposed tip.
So you suggest my "build-from-source" Dockerfile group all to-be-exported artifact on a specific directory, ok. I don't see how the working directory flag -w would impact exposed directory. I don't understand how volumes can help here. Volume sonly apply to containers, here I'm asking for a feature about Docker images building, without any running container.

@borromeotlhs
Copy link

Well, I firstly want to point out, you would use the -v option, not -w.
Apologies if I mistyped.

Besides that, the reason it'd be really convenient to use docker containers
to build your source, and then expose the artifacts to another container
are many. Off the top, you can ensure that the process to build your
source remains constant ( save for the syncing to a git repo, which should
also be constant save for passing in a newer tag ). You can then also, if
your concerned about packaging the app or binary that you built, it ends up
providing you with a lot of interesting ways to deliver and deploy the
built artifact as you'll be packaging the artifacts up in order to use
them, the artifacts end up being already containerized. You've just, by
insisting on containerizing your workflow, enable quick deployment of a
containerized app!

Hope that helps.
On Sep 11, 2014 2:54 AM, "Nicolas De loof" notifications@github.com wrote:

Thanks @borromeotlhs https://github.com/borromeotlhs for quick reply, I
just don't understand the proposed tip.
So you suggest my "build-from-source" Dockerfile group all to-be-exported
artifact on a specific directory, ok. I don't see how the working directory
flag -w would impact exposed directory. I don't understand how volumes can
help here. Volume sonly apply to containers, here I'm asking for a feature
about Docker images building, without any running container.


Reply to this email directly or view it on GitHub
#7992 (comment).

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 11, 2014

What I'm suggesting here is a "trusted build" compliant way to get a docker image reusing some elements from another one, not a workaround. I only know some of them, including -v, or just running docker cp to retrieve built artifacts. But then I need to use host filesystem as a temporary directory during my build process, and I can't guarantee the production image has been built with artifact from a specific build-from-source docker image.

@borromeotlhs
Copy link

If your container that built your source exists ( you didn't docker rmi it) and the build artifacts are stored within it, you can easily start the container back up at the exact commit, name it, expose the volume, and then run your other container to grab the exact build artifact from the trusted container ( which,BTW, is all stored on host via aufs,btrfs, etc. anyway )

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 11, 2014

Sure it's actually stored by host, but you don't get the idea. If I want to use trusted builds service, I can't rely on some manual steps to run container and do such things. I'm ok this is a fully valid workaround (this is the way I do today), but doing this I rely on the command I run on docker host to get my production container include the expected artifact, not on specified build-from-source docker image tag I would declare in my production Dockerfile. My issue here is about trusting the produced production image and traceability from source.

@wking
Copy link

wking commented Sep 11, 2014

On Thu, Sep 11, 2014 at 03:52:43PM -0700, Nicolas De loof wrote:

What I'm suggesting here is a "trusted build" compliant way to get a
docker image reusing some elements from another one, not a
workaround.

For previous discussions of this sort of “pure Dockerfile build
without bundling the build tooling”, see #3156 and #3949. For an
official image that's doesn't contain it's build toolchain, see the
rootfs.tar gymnastics used for busybox 1. Since #3949 is still
open, I think this issue should be closed in favor of #3949.

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 12, 2014

#3949 VOLUME-FROM-IMG is indeed addressing a comparable use-case, #3949 (comment) describes this exact suggestion.

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 12, 2014

Reading #3949 with more care, the suggested feature is to mount volumes from another image, but not make them persistent in the produced image. This let you access some third party stuff required to build a Dockerfile but that you want to exclude from produced image. This can help for my use-case but with a distinct approach.

@prologic
Copy link
Contributor

/cc @therealprologic

@SvenDowideit
Copy link
Contributor

see #7115, #7149 and #8021 too.

@vbatts
Copy link
Contributor

vbatts commented Sep 23, 2014

See #4933 too

@unclejack
Copy link
Contributor

The ADD instruction is probably not going to be extended any further. ADD has quite a bit of magic in it for extracting tarballs already and adding more magic isn't desirable.

Functionality like what's being discussed here is indeed useful and needed in many cases, but I think the existing open issues are going to implement something which adds this functionality.

I'll close this issue in favor of #7115, #7149 and #8021.

@ndeloof
Copy link
Contributor Author

ndeloof commented Sep 26, 2014

This makes me wonder how ADD is configured to support various protocols. Seems it natively support http/https/ftp, which other protocols ? Can I plug more ?

@unclejack
Copy link
Contributor

@ndeloof It's not wired to do something like that, it just has the code to handle those. However, the problem is that it's too magical right now. It handles remote files and local files. In additional to fetching remote files and local files, it's also extracting local tarballs.

Making it add files from another image would add even more magic and would make that code even worse. That's why we've introduced COPY, to start cleaning things up and maybe get rid of ADD at some point in the future.

@jasimmk
Copy link

jasimmk commented Mar 27, 2020

The functionality can be implemented using COPY command

COPY --from=hadolint/hadolint /bin/hadolint .

@thaJeztah thaJeztah added area/builder kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny labels Oct 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/builder kind/feature Functionality or other elements that the project doesn't currently have. Features are new and shiny
Projects
None yet
Development

No branches or pull requests

9 participants