Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tighter integration with Docker multi-stage builds #90

Merged
merged 1 commit into from Mar 15, 2018

Conversation

Projects
None yet
2 participants
@mumoshu
Copy link
Collaborator

mumoshu commented Mar 14, 2018

Point multiple steps to the same Dockerfile, but with different stage by specifying target in build.yaml, so that:

  • You can leverage Docker's native multi-stage build feature for theoretically faster assets propagation due to no docker cp between the nodes habitus and docker-daemon are running respectively.
  • An easiger migration path for usual Docker users used to multi-stage builds. They benefit from habitus' advanced features like build secrets without splitting Dockerfile

The implementation of splitting Dockerfile for a targeted build is inspired by the implementation of Docker's.
But this one is slightly more naive than the original, due to fsouza/go-dockerclient doesn't support targeted build or injecting Dockerfile content rather than the filepath to it.
This implementation, although it is native, seemed better than greatly hacking fsouzo/go-dockerclient or incorporating the official docker-client just for this use-case.

This is verified manually by running habitus with the new example at exampls/multistage/build.yml:

2018/03/14 18:50:47 ▶ Using '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/build.yml' as build file
2018/03/14 18:50:47 ▶ Collecting artifact information
2018/03/14 18:50:47 ▶ Building 3 steps
2018/03/14 18:50:47 ▶ Step 1 - base, image-name = 'base'
2018/03/14 18:50:47 ▶ Step 2 - compilation, image-name = 'compilation'
2018/03/14 18:50:47 ▶ Step 3 - runtime, image-name = 'runtime'
2018/03/14 18:50:47 ▶ Step 1 - Build for base
2018/03/14 18:50:47 ▶ Step 1 - Building base from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 1 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 1 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 1 - Building the base image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/2 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/2 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Successfully built 960b4976bfa7
Successfully tagged base:latest
2018/03/14 18:50:47 ▶ Step 2 - Build for compilation
2018/03/14 18:50:47 ▶ Step 2 - Building compilation from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 2 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 2 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 2 - Building the compilation image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/4 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/4 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Step 3/4 : FROM base aS compile
 ---> 960b4976bfa7
Step 4/4 : RUN echo 'compiling'
 ---> Running in a6a55d38773c
compiling
 ---> 598ac9b1ef0b
Removing intermediate container a6a55d38773c
Successfully built 598ac9b1ef0b
Successfully tagged compilation:latest
2018/03/14 18:50:47 ▶ Step 3 - Build for runtime
2018/03/14 18:50:47 ▶ Step 3 - Building runtime from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 3 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 3 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 3 - Building the runtime image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/6 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/6 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Step 3/6 : FROM base aS compile
 ---> 960b4976bfa7
Step 4/6 : RUN echo 'compiling'
 ---> Using cache
 ---> 598ac9b1ef0b
Step 5/6 : FROM base AS runtime
 ---> 960b4976bfa7
Step 6/6 : RUN echo 'runtime'
 ---> Using cache
 ---> 7c6308850176
Successfully built 7c6308850176
Successfully tagged runtime:latest
2018/03/14 18:50:48 ▶ Step 1 - Removing unwanted image base
2018/03/14 18:50:48 ▶ Step 2 - Removing unwanted image compilation

Resolves #89

@mumoshu mumoshu force-pushed the mumoshu:enhance-docker-native-multi-stage-build branch 2 times, most recently from 5f56acd to 7bffda5 Mar 14, 2018

@khash

khash approved these changes Mar 14, 2018

@khash

This comment has been minimized.

Copy link
Member

khash commented Mar 14, 2018

This is a great addition to Habitus @mumoshu Thank you for your help and implementing this. May I ask you to add this to the documentation (gh-pages) for others so they can use it please?

Also amending the documentation with your changes regarding build arguments would be very useful for others.

@khash

This comment has been minimized.

Copy link
Member

khash commented Mar 14, 2018

I think this one is now in conflict with the build arguments will merge once the conflict is resolved.

Tighter integration with Docker multi-stage builds
Point multiple steps to the same Dockerfile, but with different stage by specifying `target` in `build.yaml`:

```yaml
build:
    version: 2016-03-14
    steps:
      base:
        name: base
        dockerfile: Dockerfile
        target: base
      compilation:
        name: compilation
        dockerfile: Dockerfile
        target: compile
        depends_on:
          - base
      runtime:
        name: runtime
        dockerfile: Dockerfile
        target: runtime
        depends_on:
          - compilation
```

whereas `Dockerfile` is:

```
FROM alpine:3.5 As base
RUN echo 'the base'

FROM base aS  compile
RUN echo 'compiling'

FROM base  AS runtime
RUN echo 'runtime'
```

so that:

- You can leverage Docker's native multi-stage build feature for theoretically faster assets propagation due to no `docker cp` between the nodes habitus and docker-daemon are running respectively.
- An easiger migration path for usual Docker users used to multi-stage builds. They benefit from habitus' advanced features like build secrets without splitting Dockerfile

The implementation of splitting Dockerfile for a targeted build is inspired by the implementation of [Docker's](https://github.com/moby/moby/blob/3a633a712c8bbb863fe7e57ec132dd87a9c4eff7/builder/dockerfile/builder.go#L232-L239).
But this one is slightly more naive than the original, due to `fsouza/go-dockerclient` doesn't support targeted build or injecting Dockerfile content rather than the filepath to it.
This implementation, although it is native, seemed better than greatly hacking fsouzo/go-dockerclient or incorporating the official docker-client just for this use-case.

This is verified manually by running habitus with the new example at `exampls/multistage/build.yml`:

```console
2018/03/14 18:50:47 ▶ Using '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/build.yml' as build file
2018/03/14 18:50:47 ▶ Collecting artifact information
2018/03/14 18:50:47 ▶ Building 3 steps
2018/03/14 18:50:47 ▶ Step 1 - base, image-name = 'base'
2018/03/14 18:50:47 ▶ Step 2 - compilation, image-name = 'compilation'
2018/03/14 18:50:47 ▶ Step 3 - runtime, image-name = 'runtime'
2018/03/14 18:50:47 ▶ Step 1 - Build for base
2018/03/14 18:50:47 ▶ Step 1 - Building base from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 1 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 1 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 1 - Building the base image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/2 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/2 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Successfully built 960b4976bfa7
Successfully tagged base:latest
2018/03/14 18:50:47 ▶ Step 2 - Build for compilation
2018/03/14 18:50:47 ▶ Step 2 - Building compilation from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 2 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 2 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 2 - Building the compilation image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/4 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/4 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Step 3/4 : FROM base aS compile
 ---> 960b4976bfa7
Step 4/4 : RUN echo 'compiling'
 ---> Running in a6a55d38773c
compiling
 ---> 598ac9b1ef0b
Removing intermediate container a6a55d38773c
Successfully built 598ac9b1ef0b
Successfully tagged compilation:latest
2018/03/14 18:50:47 ▶ Step 3 - Build for runtime
2018/03/14 18:50:47 ▶ Step 3 - Building runtime from context '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage'
2018/03/14 18:50:47 ▶ Step 3 - Parsing and converting 'Dockerfile'
2018/03/14 18:50:47 ▶ Step 3 - Writing the new Dockerfile into '/Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated'
2018/03/14 18:50:47 ▶ Step 3 - Building the runtime image from /Users/kuoka-yusuke/go/src/github.com/cloud66/habitus/examples/multistage/Dockerfile.generated
Step 1/6 : FROM alpine:3.5 As base
 ---> 6c6084ed97e5
Step 2/6 : RUN echo 'the base'
 ---> Using cache
 ---> 960b4976bfa7
Step 3/6 : FROM base aS compile
 ---> 960b4976bfa7
Step 4/6 : RUN echo 'compiling'
 ---> Using cache
 ---> 598ac9b1ef0b
Step 5/6 : FROM base AS runtime
 ---> 960b4976bfa7
Step 6/6 : RUN echo 'runtime'
 ---> Using cache
 ---> 7c6308850176
Successfully built 7c6308850176
Successfully tagged runtime:latest
2018/03/14 18:50:48 ▶ Step 1 - Removing unwanted image base
2018/03/14 18:50:48 ▶ Step 2 - Removing unwanted image compilation
```

Resolves #89

@mumoshu mumoshu force-pushed the mumoshu:enhance-docker-native-multi-stage-build branch from 7bffda5 to 42f4ec5 Mar 15, 2018

@mumoshu

This comment has been minimized.

Copy link
Collaborator Author

mumoshu commented Mar 15, 2018

@khash Thanks for your review! Just finished rebasing.

mumoshu added a commit to mumoshu/habitus that referenced this pull request Mar 15, 2018

Documentation about the docker multi-stage builds integration
This is an addition to documentation as per cloud66-oss#90

mumoshu added a commit to mumoshu/habitus that referenced this pull request Mar 15, 2018

@mumoshu

This comment has been minimized.

Copy link
Collaborator Author

mumoshu commented Mar 15, 2018

Updated the doc in #92

@khash khash merged commit 0e83cbb into cloud66-oss:master Mar 15, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.