ADD enhancement: allow adding from another image

[vbatts]

Rather than adding an additional buildfile instruction, this enhancement
will use a mock-URI prefix of "image://", followed by a image name, and
a ":" suffix. (i.e. image://[registry[:port]][:tag]: )

Such that if you have a container that has a built artifact, you can
copy it over to a new/different image.

FROM vbatts/minimal-runtime

ADD image://vbatts/fat-buildtime:/build/server /server

CMD "/server"

Docker-DCO-1.1-Signed-off-by: Vincent Batts vbatts@redhat.com (github: vbatts)

[vbatts]

@shykes I expect this mock-URI will have overlap with the notion of provenance, but I would like to get this conversation started.

[vbatts]

@shykes I expect this mock-URI will have overlap with the notion of provenance, but I would like to get this conversation started.

[jamtur01]

This definitely needs docs.

[jamtur01]

This definitely needs docs.

[vbatts]

@jamtur01 true fact. I was hoping to iron out the functionality, before having to iterate constantly over the docs while determining the layout of it.

[vbatts]

@jamtur01 true fact. I was hoping to iron out the functionality, before having to iterate constantly over the docs while determining the layout of it.

[SvenDowideit]

@vbatts - on the other hand, if you write the simplest docs you can, we all know what you intended, and can figure out the weird ideas.

my first weird q - what happens when the ADDed path is a tgz - is it 'consistently' expanded?

if i read the tests right, you're only testing the parsing? those will be interesting integration tests :)

(I want this so very much)

[SvenDowideit]

@vbatts - on the other hand, if you write the simplest docs you can, we all know what you intended, and can figure out the weird ideas.

my first weird q - what happens when the ADDed path is a tgz - is it 'consistently' expanded?

if i read the tests right, you're only testing the parsing? those will be interesting integration tests :)

(I want this so very much)

[vbatts]

@SvenDowideit I just pushed some docs, but not sure whether to enumerate examples like this in the docs. Like

ADD image://test:/etc.tar.gz /mnt

will result in the file 'etc.tar.gz' being created as a file called 'mnt', or where

ADD image://test:/etc.tar.gz /mnt/

will result in the file 'etc.tar.gz' being copied to '/mnt/etc.tar.gz'

[vbatts]

@SvenDowideit I just pushed some docs, but not sure whether to enumerate examples like this in the docs. Like

ADD image://test:/etc.tar.gz /mnt

will result in the file 'etc.tar.gz' being created as a file called 'mnt', or where

ADD image://test:/etc.tar.gz /mnt/

will result in the file 'etc.tar.gz' being copied to '/mnt/etc.tar.gz'

[SvenDowideit]

@vbatts yeah, I'm not suggesting to enumerate all examples in the docs - the tgz one is 'special' this just means we're treating image:// as non-local. Perhaps the easy solution is to change the docs where it says If <src> is a local tar archive in a to If <src> is a tar archive in the uploaded context, in a.

@jamtur01 second opinion?

ooo: I suddenly realize - can you make the last example a little more explicit please? If I understand right, ADD image://my.registry.lan:5000/vbatts/fat-buildtime:stable:/build/runtime/bin/myapp / would grab that path from the local copy (possibly outdated), or if it does not we just error out?

ie, there isn't some magic in the registry that will just send it the one file it wants (which imo would be an awesome feature :) )

And lastly, I wonder how we can hook this together with being able to use multiple FROM statements - ie ADD image://2imagesbeforethis:/path /binaries :) (no, not a request for this PR)

@crosbymichael @shykes

[SvenDowideit]

@vbatts yeah, I'm not suggesting to enumerate all examples in the docs - the tgz one is 'special' this just means we're treating image:// as non-local. Perhaps the easy solution is to change the docs where it says If <src> is a local tar archive in a to If <src> is a tar archive in the uploaded context, in a.

@jamtur01 second opinion?

ooo: I suddenly realize - can you make the last example a little more explicit please? If I understand right, ADD image://my.registry.lan:5000/vbatts/fat-buildtime:stable:/build/runtime/bin/myapp / would grab that path from the local copy (possibly outdated), or if it does not we just error out?

ie, there isn't some magic in the registry that will just send it the one file it wants (which imo would be an awesome feature :) )

And lastly, I wonder how we can hook this together with being able to use multiple FROM statements - ie ADD image://2imagesbeforethis:/path /binaries :) (no, not a request for this PR)

@crosbymichael @shykes

[vbatts]

@SvenDowideit regarding ADD image://my.registry.lan:5000/vbatts/fat-buildtime:stable:/build/runtime/bin/myapp /, if my.registry.lan:5000/vbatts/fat-buildtime:stable was not found locally, then it would attempt to pull it. If the image is outdated, then you'd get outdated content. Same as docker run, right?

As for magically copying a path, from an image, on a remote registry... not yet :-)

[vbatts]

@SvenDowideit regarding ADD image://my.registry.lan:5000/vbatts/fat-buildtime:stable:/build/runtime/bin/myapp /, if my.registry.lan:5000/vbatts/fat-buildtime:stable was not found locally, then it would attempt to pull it. If the image is outdated, then you'd get outdated content. Same as docker run, right?

As for magically copying a path, from an image, on a remote registry... not yet :-)

[vbatts]

though, i'm still hoping to hear from @shykes and/or @crosbymichael on the layout of the image:// [mock] URI

[vbatts]

though, i'm still hoping to hear from @shykes and/or @crosbymichael on the layout of the image:// [mock] URI

[gabrtv]

+1. Moving build artifacts to a minimal runtime is a really important feature outside the world of interpreted languages. The ADD instruction seems the right place for this functionality. I'm "meh" on the image URI since it seems a little verbose for Dockerfile syntax, but I don't really know a good alternative.

If this gets the OK from the maintainers (I hope it does soon!) I'm happy to help with testing.

[gabrtv]

+1. Moving build artifacts to a minimal runtime is a really important feature outside the world of interpreted languages. The ADD instruction seems the right place for this functionality. I'm "meh" on the image URI since it seems a little verbose for Dockerfile syntax, but I don't really know a good alternative.

If this gets the OK from the maintainers (I hope it does soon!) I'm happy to help with testing.

[pnasrat]

This is not IPv6 compatible. Why not simply use url.Parse from "net/url"

http://play.golang.org/p/kClKARcxHc

[vbatts]

I worked with url.Parse, but to find an image schema that could satisfy referencing a path, and all the permutations of the docker image reference (registry:port/namespace/image:tag), it would best be a bunch of query parameters. Which is even more ugly.

I had not considered the ipv6 address as a registry name. Though this logic should still hold up for an ipv6 address as well.

[pnasrat]

OK can you add tests with a range of rfc 2732 ipv6 literals - eg [2060::1]

The case I think won't work is an ipv6 address with no port

[vbatts]

@pnasrat i've added several tests, and the parsing looks good.

[vbatts]

@pnasrat i've added several tests, and the parsing looks good.

[pmorie]

@vbatts It would be great if this could be used with docker run as well - for example, you could have an image that does its own incremental build without a Dockerfile.

[pmorie]

@vbatts It would be great if this could be used with docker run as well - for example, you could have an image that does its own incremental build without a Dockerfile.

[vbatts]

@pmorie oh interesting, though a more general purpose --add <src>:<dest> would be good, then if it were to support this as well. (something for a separate PR)

[vbatts]

@pmorie oh interesting, though a more general purpose --add <src>:<dest> would be good, then if it were to support this as well. (something for a separate PR)

[vbatts]

@shykes and @crosbymichael what are the next steps on this PR?

[vbatts]

@shykes and @crosbymichael what are the next steps on this PR?

[vbatts]

ah @proppy has already proposed an --add flag in #3578

[vbatts]

ah @proppy has already proposed an --add flag in #3578

[vbatts]

@shykes can you please weigh in on this? We are needing for this for an incrementally image build workflow. (Having and --add flag for docker run would sweeten the deal)

[vbatts]

@shykes can you please weigh in on this? We are needing for this for an incrementally image build workflow. (Having and --add flag for docker run would sweeten the deal)

[aigarius]

+1 on wanting this, one possible simplification - just skip the registry specification. It can be fine to force people to pull the image from whatever registry they want explicitly before building (if the image is not available in their default registry). This will leave just image://namespace/image:tag/folder/file and avoid all the hostname lookup and IPv4/v6 logic.

[aigarius]

+1 on wanting this, one possible simplification - just skip the registry specification. It can be fine to force people to pull the image from whatever registry they want explicitly before building (if the image is not available in their default registry). This will leave just image://namespace/image:tag/folder/file and avoid all the hostname lookup and IPv4/v6 logic.

[alexlarsson]

I wrote this blog entry: http://blogs.gnome.org/alexl/2014/06/12/building-clean-docker-images/

In it i use a two-Dockerfile setup to build a clean image. The first container just builds rpms and at the end sets up a yum repo with the result. We then start this container exposing the results with lightttp. The cool thing about this is that we can avoid ADDing the rpms to the target image so that we can then install it, thus avoiding a useless layer in the final image that has the rpms.

[alexlarsson]

I wrote this blog entry: http://blogs.gnome.org/alexl/2014/06/12/building-clean-docker-images/

In it i use a two-Dockerfile setup to build a clean image. The first container just builds rpms and at the end sets up a yum repo with the result. We then start this container exposing the results with lightttp. The cool thing about this is that we can avoid ADDing the rpms to the target image so that we can then install it, thus avoiding a useless layer in the final image that has the rpms.

[alexlarsson]

The above would be a lot nicer if docker build could use linked containers though. Without it I had to hack it by using the host ip and a global port map.

[alexlarsson]

The above would be a lot nicer if docker build could use linked containers though. Without it I had to hack it by using the host ip and a global port map.

[proppy]

@alexlarsson I'm curious what you think of #5715, it allow things like: docker build; docker run | docker build.

[proppy]

@alexlarsson I'm curious what you think of #5715, it allow things like: docker build; docker run | docker build.

[shykes]

@vbatts it looks like the primary use case is to build images where the build and run environments
don't have the same base image.

I think instead of adding a way for images to cross-reference each other in ADD, the problem will
be solved by 2 specific improvements:

1. Nested builds: #7115

2. Squashing of build dependencies: #6906

Note the 2 new keywords, IN and PUBLISH. The syntax is still up in the air,

With the 2 new keywords proposed in #7115 (IN and PUBLISH), you can compose the contents of multiple images into a
complex build, without having to make a special case in the ADD syntax.

I'm going to close this in favor of #7115. If you feel it doesn't solve your problem, would you mind leaving a comment
there? I'm open to changes and improvements.

Thanks!

[shykes]

@vbatts it looks like the primary use case is to build images where the build and run environments
don't have the same base image.

I think instead of adding a way for images to cross-reference each other in ADD, the problem will
be solved by 2 specific improvements:

1. Nested builds: #7115

2. Squashing of build dependencies: #6906

Note the 2 new keywords, IN and PUBLISH. The syntax is still up in the air,

With the 2 new keywords proposed in #7115 (IN and PUBLISH), you can compose the contents of multiple images into a
complex build, without having to make a special case in the ADD syntax.

I'm going to close this in favor of #7115. If you feel it doesn't solve your problem, would you mind leaving a comment
there? I'm open to changes and improvements.

Thanks!

[TomasTomecek]

I think this is an excellent idea! @shykes any updates what's docker strategy in this area (different build and runtime env)? There are no new comments in #7115 and #6906 for months.

[TomasTomecek]

I think this is an excellent idea! @shykes any updates what's docker strategy in this area (different build and runtime env)? There are no new comments in #7115 and #6906 for months.

[thaJeztah]

I think the multi-stage builds (implemented in #31257, #32063, and #32496) largely cover this;

FROM node AS build-env
ADD ./ /app
WORKDIR /app
RUN npm install
RUN npm run build

FROM nginx AS prod-env
COPY --from=build-env /app /var/www/html

Using that Dockerfile, running this command would only perform the steps up to the build-env stage, and tag it as development;

docker build -t development --target build-env .

Whereas omitting the --target would build all stages, and tag the last stage as production

docker build -t production .

There's two open proposals in this area that may be relevant; #32507 (COPY --mount), and #30449 (Proposal: mount from image) - which is for runtime

[thaJeztah]

I think the multi-stage builds (implemented in #31257, #32063, and #32496) largely cover this;

FROM node AS build-env
ADD ./ /app
WORKDIR /app
RUN npm install
RUN npm run build

FROM nginx AS prod-env
COPY --from=build-env /app /var/www/html

Using that Dockerfile, running this command would only perform the steps up to the build-env stage, and tag it as development;

docker build -t development --target build-env .

Whereas omitting the --target would build all stages, and tag the last stage as production

docker build -t production .

There's two open proposals in this area that may be relevant; #32507 (COPY --mount), and #30449 (Proposal: mount from image) - which is for runtime