Add multi-stage builds support #495

bertinatto · 2018-02-24T11:46:51Z

This PR adds multi-stage builds support to buildah.

Addresses #440.

rhatdan · 2018-02-24T11:51:24Z

TomSweeneyRedHat · 2018-02-24T16:28:23Z

Travis CI isn't happy @bertinatto

not ok 13 bud-from-multiple-files-two-froms
# (in test file ./bud.bats, line 51)
#   `cmp $root/Dockerfile1 ${TESTSDIR}/bud/from-multiple-files/Dockerfile1.scratch' failed with status 2

bertinatto · 2018-02-24T23:14:05Z

Right, I have a question about this failing test (bud-from-multiple-files-two-froms).

In a multi-stage build, each FROM statement will result in a different stage (i.e., container). I believe this specific test made sense so far, but from now on I'd assume that files added to a stage shouldn't end up in the final stage.

What do you think about this?

rhatdan · 2018-02-26T12:34:26Z

I am no expert on this. but isn't multi-stage the equivalent of doing two different Dockerfile, where the second Dockerfile from is from the artifact of the first Dockerfile. Anything that is in the image created by the first Dockerfile would end up in the second unless it was at some point removed?

kbaegis · 2018-02-26T12:50:53Z

@rhatdan Here's an example:
https://github.com/gentoo/gentoo-docker-images/blob/master/portage.Dockerfile

This is all based around the flexibility of needing to use one images utilities to bootstrap the toolchain for another (i.e. - using alpine gpg to validate or busybox to copy/untar).

bertinatto · 2018-02-26T20:28:30Z

@rhatdan, AFAIU the idea is that we can selectively copy artifacts from one stage to another, leaving the final image without unnecessary artifacts.

Using the example of #440, the first FROM starts a new stage with the base image golang where the binary grafeas-server is built. The second stage is built using alpine as base image and only the file grafeas-server is copied to it, leaving behind the Go SDK.

rhatdan · 2018-02-26T23:09:47Z

Do these workloads work with this patch?

bertinatto · 2018-02-27T20:23:51Z

Yup, but unfortunately multi-stage builds support changes the way buildah (and imagebuilder) work with multiple Dockerfiles (i.e., multiple FROM statements).

rhatdan · 2018-03-08T22:01:49Z

@nalind PTAL

rhatdan · 2018-03-18T11:19:52Z

@ipbabble @TomSweeneyRedHat Could you try out a couple of Dockerfiles against this to make sure it works fairly well.

I would like to get this merged along with SHELL Support so we can cut a new version of Buildah.

TomSweeneyRedHat · 2018-03-27T00:09:04Z

FWIW, I tested using both of the below Dockerfiles and these worked. I tried a couple of other more complex Dockerfiles that I found on the net and they had failures, but the same failure happened in Docker too. I don't believe it was a Buildah issue. If you've a pointer to other multi-stage Dockerfiles, happy to try them.

FROM ubuntu
RUN whoami
FROM ubuntu
RUN echo hi

and

FROM golang:1.7.3
WORKDIR /go/src/github.com/alexellis/href-counter/
RUN go get -d -v golang.org/x/net/html  
COPY app.go .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

FROM alpine:latest  
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=0 /go/src/github.com/alexellis/href-counter/app .
CMD ["./app"]

nalind · 2018-04-03T19:00:51Z

imagebuildah/build.go

@@ -113,6 +113,8 @@ type BuildOptions struct {
 // Executor is a buildah-based implementation of the imagebuilder.Executor
 // interface.
 type Executor struct {
+	name                           string


Please add a field for the numeric index of the stage which this Executor is responsible for building. We'll need that information to ensure that COPY instructions that try to reference this stage or a later stage, can be rejected.

Done, although I think it was rejecting invalid stages before.

nalind · 2018-04-03T19:00:53Z

imagebuildah/build.go

@@ -329,6 +343,12 @@ func (b *Executor) Copy(excludes []string, copies ...imagebuilder.Copy) error {
 		for _, src := range copy.Src {
 			if strings.HasPrefix(src, "http://") || strings.HasPrefix(src, "https://") {
 				sources = append(sources, src)
+			} else if len(copy.From) > 0 {
+				if other, ok := b.named[copy.From]; ok {


We'll need to check that the stage number for other is less than the one for b.

nalind · 2018-04-03T19:00:56Z

imagebuildah/build.go

+	}
+	var stageExecutor *Executor
+	for _, stage := range stages {
+		stageExecutor = b.withName(stage.Name)


This should take the index of the stage as an argument, since whether a name for the stage is specified with AS or not, its contents still need to be accessible by referring to the stage using its numeric index, and we want to know where a given Executor is in the list of them. Converting the number to a string in withName() and using it as another key for adding copied to the named map, since the values are pointers, should be fine.

Done. Thanks for catching this.

I checked the behaviour of docker and it does accept both indexes and names. On the other hand, imagebuilder does not. We might report this.

nalind · 2018-04-03T19:02:32Z

Apologies for the delay. A couple of notes about stage ordering and numbers, but the rest looks really good. Thanks!

rh-atomic-bot · 2018-04-03T22:49:53Z

☔ The latest upstream changes (presumably 5ce8009) made this pull request unmergeable. Please resolve the merge conflicts.

rhatdan · 2018-04-04T01:12:52Z

@bertinatto Needs a rebase.

bertinatto · 2018-04-05T07:24:29Z

@nalind, thanks for the review. I'll apply your suggestions as soon as I have time (probably over the weekend, if that's not a problem).

@rhatdan, will do, thanks!

ipbabble · 2018-04-05T19:27:10Z

I've been traveling and haven't been able to take a look at this. Sorry. I've been doing som eof my own testing of complex Dockerfiles. William Henry Senior Consulting Software Engineer Red Hat Colorado, USA.

…

On Thu, Apr 5, 2018 at 1:24 AM, Fábio Bertinatto ***@***.***> wrote: @nalind <https://github.com/nalind>, thanks for the review. I'll apply your suggestions as soon as I have time (probably over the weekend, if that's not a problem). @rhatdan <https://github.com/rhatdan>, will do, thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#495 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPpgZe3tN_RcxG5ARGHbCRlQnVwhoAPks5tlcavgaJpZM4SR3Us> .

nalind · 2018-04-10T18:59:19Z

LGTM pending CI.

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com>

bertinatto · 2018-04-10T19:57:07Z

I had to close/reopen to retest. All green now.

nalind · 2018-04-10T21:04:26Z

Thanks!
@rh-atomic-bot r+

rh-atomic-bot · 2018-04-10T21:04:27Z

📌 Commit b5ef4be has been approved by nalind

rh-atomic-bot · 2018-04-10T21:04:31Z

⚡ Test exempted: pull fully rebased and already tested.

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com> Closes: #495 Approved by: nalind

rhatdan · 2018-04-11T05:43:28Z

Time for a new build? I think this could be buildah 1.0? @nalind @TomSweeneyRedHat WDYT

bertinatto · 2018-04-11T06:30:36Z

My vote doesn't count, but it would be awesome to have buildah 1.0!

TomSweeneyRedHat · 2018-04-11T11:21:42Z

On 04/11/2018 01:43 AM, Daniel J Walsh wrote: Time for a new build? I think this could be buildah 1.0? @nalind <https://github.com/nalind> @TomSweeneyRedHat <https://github.com/TomSweeneyRedHat> WDYT — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#495 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYbWBRJkNd9F4NiftznvUOnxXMgF5nZSks5tnZgBgaJpZM4SR3Us>.

+1

boaz0 · 2018-04-11T11:27:08Z

/me being excited! 😀

bertinatto force-pushed the multi_stage branch from 8ccfb2c to 8a96829 Compare February 24, 2018 22:49

bertinatto force-pushed the multi_stage branch 2 times, most recently from f807c99 to ecec5e2 Compare February 25, 2018 07:56

bertinatto changed the title ~~[WIP] Add multi-stage builds support~~ Add multi-stage builds support Feb 25, 2018

bertinatto force-pushed the multi_stage branch from ecec5e2 to 0da653b Compare February 28, 2018 19:35

nalind reviewed Apr 3, 2018

View reviewed changes

bertinatto force-pushed the multi_stage branch 5 times, most recently from 64a78ab to 847aff2 Compare April 10, 2018 17:56

bertinatto added 3 commits April 10, 2018 21:09

Add multi-stage builds support

82744e7

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com>

Add multi-stage builds tests

b5ff69f

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com>

Allow referencing stages as index and names

b5ef4be

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com>

bertinatto force-pushed the multi_stage branch from 847aff2 to b5ef4be Compare April 10, 2018 19:09

bertinatto closed this Apr 10, 2018

bertinatto reopened this Apr 10, 2018

bertinatto closed this Apr 10, 2018

bertinatto reopened this Apr 10, 2018

rh-atomic-bot closed this in a16c3cd Apr 10, 2018

rh-atomic-bot pushed a commit that referenced this pull request Apr 10, 2018

Add multi-stage builds tests

d46d16c

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com> Closes: #495 Approved by: nalind

rh-atomic-bot pushed a commit that referenced this pull request Apr 10, 2018

Allow referencing stages as index and names

6f7d05b

Signed-off-by: Fabio Bertinatto <fbertina@redhat.com> Closes: #495 Approved by: nalind

bertinatto mentioned this pull request Apr 11, 2018

support for multiple FROM instructions #440

Closed

jeremyeder mentioned this pull request Sep 12, 2018

Utilize multi-stage build to reduce docker image size kubernetes-sigs/node-feature-discovery#131

Merged

khrm mentioned this pull request Jan 31, 2019

Support Docker builds fabric8io/osio-pipeline#67

Closed

github-actions bot added the locked - please file new issue/PR label Oct 26, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multi-stage builds support #495

Add multi-stage builds support #495

bertinatto commented Feb 24, 2018 •

edited

rhatdan commented Feb 24, 2018

TomSweeneyRedHat commented Feb 24, 2018

bertinatto commented Feb 24, 2018

rhatdan commented Feb 26, 2018

kbaegis commented Feb 26, 2018

bertinatto commented Feb 26, 2018

rhatdan commented Feb 26, 2018

bertinatto commented Feb 27, 2018

rhatdan commented Mar 8, 2018

rhatdan commented Mar 18, 2018

TomSweeneyRedHat commented Mar 27, 2018

nalind Apr 3, 2018

bertinatto Apr 10, 2018

nalind Apr 3, 2018

bertinatto Apr 10, 2018

nalind Apr 3, 2018

bertinatto Apr 10, 2018

nalind commented Apr 3, 2018

rh-atomic-bot commented Apr 3, 2018

rhatdan commented Apr 4, 2018

bertinatto commented Apr 5, 2018

ipbabble commented Apr 5, 2018 via email

nalind commented Apr 10, 2018

bertinatto commented Apr 10, 2018

nalind commented Apr 10, 2018

rh-atomic-bot commented Apr 10, 2018

rh-atomic-bot commented Apr 10, 2018

rhatdan commented Apr 11, 2018

bertinatto commented Apr 11, 2018

TomSweeneyRedHat commented Apr 11, 2018 via email

boaz0 commented Apr 11, 2018

Add multi-stage builds support #495

Add multi-stage builds support #495

Conversation

bertinatto commented Feb 24, 2018 • edited

rhatdan commented Feb 24, 2018

TomSweeneyRedHat commented Feb 24, 2018

bertinatto commented Feb 24, 2018

rhatdan commented Feb 26, 2018

kbaegis commented Feb 26, 2018

bertinatto commented Feb 26, 2018

rhatdan commented Feb 26, 2018

bertinatto commented Feb 27, 2018

rhatdan commented Mar 8, 2018

rhatdan commented Mar 18, 2018

TomSweeneyRedHat commented Mar 27, 2018

nalind Apr 3, 2018

Choose a reason for hiding this comment

bertinatto Apr 10, 2018

Choose a reason for hiding this comment

nalind Apr 3, 2018

Choose a reason for hiding this comment

bertinatto Apr 10, 2018

Choose a reason for hiding this comment

nalind Apr 3, 2018

Choose a reason for hiding this comment

bertinatto Apr 10, 2018

Choose a reason for hiding this comment

nalind commented Apr 3, 2018

rh-atomic-bot commented Apr 3, 2018

rhatdan commented Apr 4, 2018

bertinatto commented Apr 5, 2018

ipbabble commented Apr 5, 2018 via email

nalind commented Apr 10, 2018

bertinatto commented Apr 10, 2018

nalind commented Apr 10, 2018

rh-atomic-bot commented Apr 10, 2018

rh-atomic-bot commented Apr 10, 2018

rhatdan commented Apr 11, 2018

bertinatto commented Apr 11, 2018

TomSweeneyRedHat commented Apr 11, 2018 via email

boaz0 commented Apr 11, 2018

bertinatto commented Feb 24, 2018 •

edited