New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "secret" as allowed build args #36443

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
6 participants
@manabusakai
Copy link

manabusakai commented Feb 28, 2018

Signed-off-by: Manabu Sakai rea.527@gmail.com

- What I did

Added build-args to pass secrets variable when docker build.

I want to improve this.

Warning: It is not recommended to use build-time variables for passing secrets like github keys, user credentials etc. Build-time variable values are visible to any user of the image with the docker history command.

ref: https://docs.docker.com/engine/reference/builder/#arg

- How I did it

Added SECRET and secret to builtinAllowedBuildArgs.

- How to verify it

Dockerfile:

FROM ubuntu
RUN echo ${SECRET}

Then,

$ docker build -t build-args-test . --build-arg SECRET=secret-string
$ docker history build-args-test

- Description for the changelog

The SECRET and secret of --build-arg are no longer displayed in docker image history.

- A picture of a cute animal (not mandatory but encouraged)

dsc_0755

ref: http://blog.pokkeboy.com/about

Add "secret" as allowed build args
Signed-off-by: Manabu Sakai <rea.527@gmail.com>
@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 2, 2018

Hi @tonistiigi - Thank you very much for maintaining moby/docker!

I do know that you're working on builtkit and going to solve the secret management problem in more elegant ways in moby/buildkit#261 and moby/buildkit#262.

However, I still find this change to be useful enough to get merged.


Even though this isn't very intuitive at first glance, I'd like to emphasize that this is a GREAT step forward for the secret management 😃

I'm currently using something like docker build --build-arg FTP_PROXY=$GITHUB_USER:$GITHUB_TOKEN . to securely pass creds enable e.g. private GitHub repository access from the build context.

Obviously, this isn't a perfect solution to the secret management problem as:

  1. It still exposes the creds via process list, and
  2. The size of creds is capped at ARG_MAX
  3. Exploiting FTP_PROXY or HTTP(S)_PROXY for securely passing secrets to build context is counterintuitive

However, that's much better than unexpectedly exposing it via docker image metadata/layers. I believe this method should be used more widely wherever possible.

This PR just solves the 3rd issue with a minimum change, without any unwanted side-effects(I suppose so?).

Full disclaimer: I'm a colleague of @manabusakai

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 2, 2018

For anyone interested, this is a bit more context about how *_PROXY build args have been excluded from the history: #31584

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 2, 2018

An another justification of this change is #30637 from @ehazlett.

I wished #30637 to get merged in the past, but AFAICS it didn't make it due to the design issue.

If we can't have something like that in docker before builtkit goes GA, and not wishing to switch to alternative builders like support for secrets:

  • builtkit(doesn't support secrets, but looks promising)
  • img(wrapper around buildkit, no support for secrets, but loos promising)
  • imagebuilder(squash by default, hence lacks layer caching)
  • s2i(another syntax other than Dockerfile)
  • buildah(client is linux-only, another syntax)
  • habitat(great work, but completely different model to build docker images, no docker layer caching. cached at habit-level. awesome, but overkill for docker-only users)

this change is understandable to me.

Edit: I'm not saying any of those projects are bad. Not at all, they all are awesome. I'm just saying they all have different sweet-spots, not drop-in replacements of docker! And we're already docker users.


It is finally time for us to save people from accidentally leaking build creds, or giving up build caching, or switching to another tools only for secret management!

@AkihiroSuda
Copy link
Member

AkihiroSuda left a comment

Programs that are totally unrelated to Docker/Moby may use SECRET/secret, as this is very common English word.

So, at least, the environment variable name must be changed to an unique one. e.g. org.mobyproject.secret.

However, I don't think using environment variables for secret management is correct.
For example, a script executed in RUN may leak the environment variables to build log, e.g. for debugging purpose.

Also, potentially (although unlikely), future version of Docker (or other Dockerfile-compatible tools) MAY capture the environment variable set on completion on the each of the RUN steps. moby/buildkit#74

So I suggest revisiting @ehazlett 's proposal instead (as moby/buildkit#261).

For a while, you may want to use docker build --network=foo with a container in the network foo for hosting secret files.

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 4, 2018

Programs that are totally unrelated to Docker/Moby may use SECRET/secret, as this is very common English word.

So, at least, the environment variable name must be changed to an unique one. e.g. org.mobyproject.secret.

Good point! I agree with your suggested change. @manabusakai WDYT ?

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 4, 2018

@AkihiroSuda Thanks for the review!

However, I don't think using environment variables for secret management is correct.
For example, a script executed in RUN may leak the environment variables to build log, e.g. for debugging purpose.

I'm not really arguing here, but envvars won't be leaked without explicit actions. If you RUN echo $SECRET or RUN export it you leak it of course. But the same thing can be said for any secret passing mechanism.
What if you've mounted secrets and cat it?
Or printing the secret after receiving it from an another container in the same ns? 🙂

Also, potentially (although unlikely), future version of Docker (or other Dockerfile-compatible tools) MAY capture the environment variable set on completion on the each of the RUN steps. moby/buildkit#74

Sounds awesome towards more reproducible builds! Keep up the great work!

You may already know but the envvars and build-args are two different things.
Please don't capture build-args like *_PROXY and org.mobyproject.secret which may contain sensitive info.

So I suggest revisiting @ehazlett 's proposal instead (as moby/buildkit#261).

Would this be your suggested change?
In other words, are you willing to accept it if I followed @ehazlett's approach?
That's absolutely ok to me if you're willing to do so. I'm happy to reimplement it.

However, I'd rather suggest you to hold on until moby/buildkit#261 and moby/buildkit#262 and future integration of builtkit to moby, before making a considerable change like that.

The approach taken in this PR is much simpler and just works without any NEW drawbacks and huge engineering efforts.

For a while, you may want to use docker build --network=foo with a container in the network foo for hosting secret files.

Thx! It is always nice to get to know about more workarounds.

You may already know but we have bunch of them:

  • https://github.com/mdsol/docker-ssh-exec
  • Use host's permission to utilize hashicorp Vault or KMS from your cloudprovider(AWS KMS w/ IAM role associated to EC2 instance)
  • docker run -e SECRET=... or docker run -v mysecret:/mysecret and docker commit

Again, my point is that the approach of this change is much better than all those existing workarounds due to its simplicity.

@AkihiroSuda WDYT?

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 4, 2018

@AkihiroSuda Please consider my above comment as-is. I have no mean to force you maintainers to merge this, AT ALL of course. That's maintainers' responsibility or freedom to do so.

I just wanted to sync up with you, as you seemed to have missed my points(to me sry)!
I you think you haven't missed any point, I really appreciate it if you could just let me know.
I'll just rethink what I should do then.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 5, 2018

I'm not comfortable with this change, and I don't think this will address many use-cases. First of all, build-args were not designed for handling secret data; this was the main reason for explicitly mentioning in the docs that they should not be used for this.

While it's possible to use build-args for secrets (e.g. in a multi-stage build, as long as you take care not to have your secrets end up in the final image), they were not designed for it. The code path doesn't treat them secure, so it's generally not recommended.

Excluding a single name (SECRET) will likely not benefit many use-cases (often, I'd be needing multiple secrets, or my secrets have to be named different).

Based on the above, I'm -1 for this change, but agree that a long-term solution is still needed

@dnephin

This comment has been minimized.

Copy link
Member

dnephin commented Mar 5, 2018

I agree with @thaJeztah and @AkihiroSuda

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 5, 2018

@thaJeztah @dnephin Hi, thanks a lot for your comments! I'm more than welcome if this is gets relected reasonably.

Just curious but how do you pass your build-time secrets today, especially when you do multi-stage builds?

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 6, 2018

First of all, I'd try to keep my secrets out of my Dockerfile if possible. So instead of:

FROM foo
RUN git clone <my-private-repo>
RUN do-some-things

(which could also be non-deterministic)

I have my Dockerfile as part of the repository, and do:

git glone <my-private-repo>

docker build ...

If secrets are not avoidable, you can still do:

FROM foo AS stage1
RUN echo $SECRET
RUN mkdir /artefacts

# Compile your code, etc etc, and store the artefacts somewhere
RUN echo bla > /artefacts/something


# Start with a clean image
FROM alpine

# Copy the artefacts from the first stage
COPY --from=stage1 /artefacts/something /bin/something

The image produced by the first build-stage will only be present on the server where you built the Dockerfile; that image will have your secret (at least in its docker history):

docker history dfd834940c56 --no-trunc
IMAGE                                                                     CREATED                  CREATED BY                                                                                          SIZE                COMMENT
sha256:dfd834940c56300567882be6bb3e25254c8bacfce7fcbe0b660f27c46ef371a1   Less than a second ago   |1 SECRET=password /bin/sh -c echo bla > /artefacts/something                                       4B                  
sha256:e65ffa8ecca984e9079231a26b9c68bb4b4185ac0021ad2f6864c1ebe82172d6   Less than a second ago   |1 SECRET=password /bin/sh -c mkdir /artefacts                                                      0B                  
sha256:6922d952c6204e44e34c2eb2216bf8fa3f0377ddd2e95fb27d059add3ff7dc79   Less than a second ago   |1 SECRET=password /bin/sh -c echo                                                                  0B                  
sha256:d02eea28ed6fa01f072cef06f6bd9719fed43d21621df4c458881bfe8d74b212   Less than a second ago   /bin/sh -c #(nop)  ARG SECRET                                                                       0B                  
sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353   7 weeks ago              /bin/sh -c #(nop)  CMD ["/bin/sh"]                                                                  0B                  
<missing>                                                                 7 weeks ago              /bin/sh -c #(nop) ADD file:093f0723fa46f6cdbd6f7bd146448bb70ecce54254c35701feeceb956414622f in /    4.15MB        

As long as you don't push that intermediate image (and it's not a problem that the host on which you build has that image containing your secret), the secret is not distributed.

The final image (produced by the FROM alpine step), will not contain the secret, nor will it be visible in its "history" (docker history ...);

IMAGE                                                                     CREATED                  CREATED BY                                                                                                        SIZE                COMMENT
sha256:966d28bc618f3a73e6c520884f8d7f47c3e09e611a3fdef0da6d7968ab2f2dbf   Less than a second ago   /bin/sh -c #(nop) COPY file:3524e65f6d03e3cebb9745369624ceaf520707ed6f2a0ef962ee8819a2136594 in /bin/something    4B                  
sha256:3fd9065eaf02feaf94d68376da52541925650b81698c53c6824d92ff63f98353   7 weeks ago              /bin/sh -c #(nop)  CMD ["/bin/sh"]                                                                                0B                  
<missing>                                                                 7 weeks ago              /bin/sh -c #(nop) ADD file:093f0723fa46f6cdbd6f7bd146448bb70ecce54254c35701feeceb956414622f in /                  4.15MB              
@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 6, 2018

@thaJeztah Thank you very much for the the detailed explanation!

Seems like you and I are following the same path.

I began to avoid your first pattern due to that I have to resolve library deps for my nodejs/ruby/etcd project outside of docker build like:

git glone <my-private-repo>

# Part of library deps are downloaded from private Git repositories
bundle install

docker build ...

What I wanted achieve with Dockerfile in the first place is build reproducibility.

In that regard, running e.g. bundle install on host didn't sound "ideal" to me. What if you want to cross-build e.g. bundle install native ruby extensions for linux from macOS host.

I do chain docker-run and docker-build like docker run -v $(pwd)/build-artifacts:/build-articats bundle install --path /build-artifcacts and docker build -t myfinalimage . for this use-case.

However, hopefully, Docker multi-stage builds + ideal secret management make this chaining unnecessary.

Also note that this pattern works well with golang projects for example. That's because you tend to vendor your private library deps in SCM, rather than resolving it at build-time.


And in your second pattern - I believe you already know though - you're basically giving up layer caching for the first stage1 image, which of course slows down your consecutive docker builds.
docker pushing the first image results in a leakage of the secret so it wasn't acceptable(to me).
So I have started avoiding this pattern too.


I'm really curious. What would you do today if you want a maximum build reproducibility + faster docker builds without leaking secrets? 🤔

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 6, 2018

And in your second pattern - I believe you already know though - you're basically giving up layer caching for the first stage1 image, which of course slows down your consecutive docker builds.

Layer caching should be the same: multi-stage builds will still use caching, assuming you build again on the same host.

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 6, 2018

@thaJeztah Thanks for clarifying!

assuming you build again on the same host.

Certainly!

In most of my use-cases though, docker hosts are ephemeral. So I tend to docker pull the images serve as cache before I run a multi-stage build. And I also prefer that also for security perspective - I don't need to fear about my long-running docker host with bunch of sensitive images gets snooped somehow/by anyone 😉

Perhaps in your case this wan't a problem because you are happy with the long running docker host or discarding layer cache occasionally, right? (Just curious. Everyone seems to have too different use-cases and contexts when talking about this secret management + extra use-case)

@mumoshu

This comment has been minimized.

Copy link

mumoshu commented Mar 7, 2018

@thaJeztah @AkihiroSuda @dnephin I got to revisit your awesome write-up #13490, which covers many of the above concerns.

My best bet after that is to close this as not "change requested" but rather rejected, and consolidate this discussion.

Thanks anyway for taking your time, and maintaining moby/docker!

@manabusakai Thanks for this PR!

Unfortunately, we'd better close it for now and continue discussion in #13490 if necessary, so that we won't trouble maintainers(oh, sry but I was the person who troubled them most 😅).

@manabusakai manabusakai closed this Mar 7, 2018

@manabusakai manabusakai deleted the manabusakai:change-allowed-build-args branch Mar 7, 2018

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 7, 2018

@mumoshu oops, wanted to reply to your comment

In most of my use-cases though, docker hosts are ephemeral. So I tend to docker pull the images serve as cache before I run a multi-stage build. And I also prefer that also for security perspective - I don't need to fear about my long-running docker host with bunch of sensitive images gets snooped somehow/by anyone

Yes, that use-case is definitely more difficult to address, and I agree with all of the above.

I added my examples to show possible ways to work around the current limitations, and those examples may (or may not) fit your use case (I've seen many users complain about missing features (in general, not just this case) that could be addressed by a small change in their process)

TL;DR; while there are ways to address the problem for some use-cases, the build-time secrets problem is not yet resolved. Work is in progress on BuildKit (https://github.com/moby/buildkit), which provides a lot of low-level features that could be used to power docker build (I think integration with ssh-agent is possible with buildkit #6396, and would address some use-cases)

Yes; #13490 lists concerns/missing things for build-time secrets; feel free to participate there 👍

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Mar 10, 2018

Also; linking #33343, which is about build-secrets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment