build: add multi-stage build support

[tonistiigi]

fixes #31067
fixes #31892
depends on #31236

[tonistiigi]

@johnstep @jhowardmsft Do you know what may be the problem with windows test. TestBuildMultiStageArg passes so it shouldn't be completely broken.

[tonistiigi]

@johnstep @jhowardmsft Do you know what may be the problem with windows test. TestBuildMultiStageArg passes so it shouldn't be completely broken.

[thaJeztah]

ping @johnstep @jhowardmsft PTAL

[thaJeztah]

ping @johnstep @jhowardmsft PTAL

[anusha-ragunathan]

I tested the PR while building a lean network plugin and it works quite nicely! Design and testing LGTM.

[anusha-ragunathan]

I tested the PR while building a lean network plugin and it works quite nicely! Design and testing LGTM.

[duglin]

A slight variant/alternative mentioned here: #31067 (comment)

[duglin]

A slight variant/alternative mentioned here: #31067 (comment)

[tonistiigi]

note to self: make sure old context is closed

[tonistiigi]

I've not rebased this since #31236 needs to be merged before anyway.

[tonistiigi]

I've not rebased this since #31236 needs to be merged before anyway.

[tonistiigi]

Based on #31067 (comment) I will reimplement this with COPY --context

[tonistiigi]

Based on #31067 (comment) I will reimplement this with COPY --context

[simonferquel]

I think we need to explicitly forbid building / (or c:\) and /windows (or c:\windows) out of a Windows image. That would make the image very large, and might even fail because Windows has a concept of "exclusive openness" of files, that might make some system files not copyable

[simonferquel]

I think we need to explicitly forbid building / (or c:\) and /windows (or c:\windows) out of a Windows image. That would make the image very large, and might even fail because Windows has a concept of "exclusive openness" of files, that might make some system files not copyable

[tonistiigi]

I have updated this PR to the new COPY --context=n syntax. Named addressing should be in a separate PR.

PTAL @dnephin @dmcgowan @simonferquel @philtay

Could I move it to code-review now as the design was discussed in the previous implementations and a decision was made in the favor of this?

[tonistiigi]

I have updated this PR to the new COPY --context=n syntax. Named addressing should be in a separate PR.

PTAL @dnephin @dmcgowan @simonferquel @philtay

Could I move it to code-review now as the design was discussed in the previous implementations and a decision was made in the favor of this?

[philtay]

Looks good. Thinking about the "zero case", I'm not sure if context=0 actually makes sense. Probably a simple COPY, without the context flag, should imply context=0. If we give to the initial context a number now (i.e. zero), we'll be forced to give it a name later for consistency (following named contexts PR). While it's easy (syntax-wise) to give the user the ability to name contexts from 1 onward, I don't see an elegant way to let them specify a name for the initial context.

Moreover it's debatable if a case like this one should really fail.

FROM busybox
COPY --context=0 foo bar

Even if we're clearly not in a nested build situation, the context=0 flag refers to the initial context. Thus, in theory, it should work. Omitting a number/name for the initial context would solve these little ambiguities and it feels (at least to me) more consistent with the actual behavior (i.e. you can COPY only from the initial context without having to specify a flag).

[philtay]

Looks good. Thinking about the "zero case", I'm not sure if context=0 actually makes sense. Probably a simple COPY, without the context flag, should imply context=0. If we give to the initial context a number now (i.e. zero), we'll be forced to give it a name later for consistency (following named contexts PR). While it's easy (syntax-wise) to give the user the ability to name contexts from 1 onward, I don't see an elegant way to let them specify a name for the initial context.

Moreover it's debatable if a case like this one should really fail.

FROM busybox
COPY --context=0 foo bar

Even if we're clearly not in a nested build situation, the context=0 flag refers to the initial context. Thus, in theory, it should work. Omitting a number/name for the initial context would solve these little ambiguities and it feels (at least to me) more consistent with the actual behavior (i.e. you can COPY only from the initial context without having to specify a flag).

[tonistiigi]

@philtay The numbers mean the index of a build block that is defined by FROM. Accessing user files is different and doesn't require any flags. I think there is no confusion understanding that numbers start with 0. I'll change the error message to the case you pointed out so it clearly states that this copy would mean source and destination are already the same. After the naming is implemented in the next step the indexes would be minimally used anyway.

[tonistiigi]

@philtay The numbers mean the index of a build block that is defined by FROM. Accessing user files is different and doesn't require any flags. I think there is no confusion understanding that numbers start with 0. I'll change the error message to the case you pointed out so it clearly states that this copy would mean source and destination are already the same. After the naming is implemented in the next step the indexes would be minimally used anyway.

[philtay]

@tonistiigi I misunderstood the meaning of context=0, thanks for the explanation. LGTM.

[philtay]

@tonistiigi I misunderstood the meaning of context=0, thanks for the explanation. LGTM.

[AkihiroSuda]

How about calling it --from-context for clarity?

[AkihiroSuda]

How about calling it --from-context for clarity?

[tonistiigi]

@AkihiroSuda That is bit too verbose to my taste. I'd rather prefer --from then but it may be confused with FROM.

I have updated this PR so it doesn't depend on any other PRs. PTAL.

[tonistiigi]

@AkihiroSuda That is bit too verbose to my taste. I'd rather prefer --from then but it may be confused with FROM.

I have updated this PR so it doesn't depend on any other PRs. PTAL.

[StefanScherer]

Great. I've seen the tweets today, but first didn't understand why people are so happy about it. After reading #31067 I understood the benefit.
Some other captains discussed about the COPY --from=0 ... as it looks very low level.
Coincidentally I've seen some slides about Rocker (https://github.com/grammarly/rocker#exportimport) and their approach how to copy artifacts from a previous layer.

FROM google/golang:1.4
ADD . /src
WORKDIR /src
RUN CGO_ENABLED=0 go build -a -installsuffix cgo -v -o rocker.o rocker.go
EXPORT rocker.o

FROM busybox
IMPORT rocker.o /bin/rocker
CMD ["/bin/rocker"]

As we haven't seen options to Dockerfile instructions I just wonder if the final syntax could more look like this?

[StefanScherer]

Great. I've seen the tweets today, but first didn't understand why people are so happy about it. After reading #31067 I understood the benefit.
Some other captains discussed about the COPY --from=0 ... as it looks very low level.
Coincidentally I've seen some slides about Rocker (https://github.com/grammarly/rocker#exportimport) and their approach how to copy artifacts from a previous layer.

FROM google/golang:1.4
ADD . /src
WORKDIR /src
RUN CGO_ENABLED=0 go build -a -installsuffix cgo -v -o rocker.o rocker.go
EXPORT rocker.o

FROM busybox
IMPORT rocker.o /bin/rocker
CMD ["/bin/rocker"]

As we haven't seen options to Dockerfile instructions I just wonder if the final syntax could more look like this?

[StefanScherer]

Just seen #32063, much better ❤️

[StefanScherer]

Just seen #32063, much better ❤️

[dserodio]

I'm confused, what is the syntax that was merged in this PR?

[dserodio]

I'm confused, what is the syntax that was merged in this PR?

[alexisvincent]

Is it possible to run the image from a particular stage?

For instance if I have build/dev tools in a "builder" stage. That I want to use interactively.

e.g.

docker run my-image --stage=builder some-useful-binary

[alexisvincent]

Is it possible to run the image from a particular stage?

For instance if I have build/dev tools in a "builder" stage. That I want to use interactively.

e.g.

docker run my-image --stage=builder some-useful-binary

[dnephin]

--target=

[dnephin]

--target=