Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build --secret with buildkit #1288

Merged
merged 2 commits into from Aug 17, 2018

Conversation

@tiborvass
Copy link
Contributor

tiborvass commented Aug 14, 2018

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

This comment has been minimized.

Copy link
Contributor Author

tiborvass commented Aug 14, 2018

@vdemeester @cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeester
Copy link
Member

vdemeester left a comment

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@tiborvass tiborvass force-pushed the tiborvass:build-secrets branch from e42eb79 to 77827e4 Aug 14, 2018

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Aug 14, 2018

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@            Coverage Diff             @@
##           master    #1288      +/-   ##
==========================================
+ Coverage   54.03%   54.05%   +0.01%     
==========================================
  Files         272      272              
  Lines       18072    18114      +42     
==========================================
+ Hits         9766     9792      +26     
- Misses       7690     7706      +16     
  Partials      616      616
@tiborvass

This comment has been minimized.

Copy link
Contributor Author

tiborvass commented Aug 14, 2018

@vdemeester fixed

@cpuguy83
Copy link
Collaborator

cpuguy83 left a comment

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

This comment has been minimized.

@cpuguy83

cpuguy83 Aug 14, 2018

Collaborator

Maybe we can add some test for these two new functions?

This comment has been minimized.

@tiborvass

tiborvass Aug 17, 2018

Author Contributor

Done!

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Aug 15, 2018

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf "hello secret" > ./mysecret.txt

export DOCKER_BUILDKIT=1

docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - . <<EOF
# syntax = tonistiigi/dockerfile:runmount20180618
FROM busybox
RUN echo "hello world"
RUN --mount=type=secret,id=mysecret echo "anything here"
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
EOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

This comment has been minimized.

Copy link
Collaborator

cpuguy83 commented Aug 15, 2018

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

This comment has been minimized.

Copy link
Member

AkihiroSuda commented Aug 16, 2018

Is # syntax = still needed?

@tiborvass tiborvass force-pushed the tiborvass:build-secrets branch from 77827e4 to 56cf398 Aug 17, 2018

@tiborvass

This comment has been minimized.

Copy link
Contributor Author

tiborvass commented Aug 17, 2018

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

build: implement build secrets with buildkit
This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass <tibor@docker.com>

@tiborvass tiborvass force-pushed the tiborvass:build-secrets branch from 56cf398 to c4c4825 Aug 17, 2018

vendor: Bump default API version to 1.39
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d

Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Aug 17, 2018

Tried this again, and looks good :)

printf "hello secret" > ./mysecret.txt

export DOCKER_BUILDKIT=1

docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - . <<EOF
# syntax = tonistiigi/dockerfile:secrets20180808
FROM busybox
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
EOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8       digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8         name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8      started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8    completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8     duration: 559.8019ms
#8 0.303 hello secret

#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9       digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9         name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9      started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9    completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9     duration: 600.4993ms
#9 0.343 hello secret
@thaJeztah
Copy link
Member

thaJeztah left a comment

LGTM

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Aug 17, 2018

/cc @albers for bash completion 😅

@thaJeztah thaJeztah merged commit cb142fa into docker:master Aug 17, 2018

9 checks passed

ci/circleci: cross Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: shellcheck Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
ci/circleci: validate Your tests passed on CircleCI!
Details
codecov/patch 60.46% of diff hit (target 50%)
Details
codecov/project 54.05% (+0.01%) compared to 9641739
Details
continuous-integration/jenkins/pr-head This commit looks good
Details
dco-signed All commits are signed

@GordonTheTurtle GordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018

@tiborvass

This comment has been minimized.

Copy link
Contributor Author

tiborvass commented Aug 17, 2018

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018

Improve build_args description
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.

As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.

docker/cli#1288
@haizaar

This comment has been minimized.

Copy link

haizaar commented Nov 4, 2018

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.