Join GitHub today
Build secrets #33343
In a maintainers meeting last Thursday, there was a discussion how to move forward with Build secrets.
The previous PR #30637 is closed atm but we need to make sure that the issue is still tracked. The issue was closed because of design issues(listed below) and possible changes/features coming with #32507. This issue is mainly for keeping the secrets discussion from blocking #32507.
In #30637 secrets are sent from the client with the context tar. There were concerns if these should be loaded from swarm secrets instead. The use cases seem quite different but it does feel weird to have 2 secrets implementations. Also, the current "build-secrets" use cases are not as secure as swarm ones.
In #30637 user specifies target path for the secrets in cli command with
Most examples that show build secrets use it for SSH keys. There are other ways for exposing this specific feature. #32677 allows ssh forwarding(poc tonistiigi@a175773). By exposing git sources as build stages we could use any auth(ssh, oauth) for cloning git repos.
Build secrets can be exposed from
Instead of injecting the data with the context they would be exposed by the client session #32677 when builder actually asks for them. This is very similar how exposing source files outside of working dir should work for the builder in the future. The only difference is that tmpfs would be used for secret data and it would not be cached.
In Dockerfile a
referenced this issue
May 22, 2017
referenced this issue
Jun 15, 2017
referenced this issue
Jul 13, 2017
Not a huge fan of overloading such a flag.
I think we should focus on what the APIs are for this before trying to work out the exact CLI details.
So basically, how does an API client inject secrets?
I can't easily speak to your first two questions, but:
Simplicity is best, IMO. I would like to be able to say, for example:
Doing so would imply the semantics that no trace of anything out of that directory should be left behind in the image (what a lot of people do manually with an
I'd recommend the session added for
Daemon asks them from the client, client validates against cli parameters. (This is for local secrets, to access swarm secrets, daemon would need to query swarm).
In #33343 (comment) this is done with
On request from @cpuguy83 I will add my two cents:
Important things are:
From user point of view, I like "MOUNT" option from rocker, or the "RUN --mount", as it allows to solve two of the issues: and build secrets AND caching for package managers. It requires from both provider and consumer only support for files, which I think is the most versatile. I would expect it to be a total circumvention of Docker build containment, having no effect on image hash and not being uploaded to daemon as part context. It is a secret needed for build - either build passes with it, or fails because the secret is wrong.
Problem is some people will try to use it for example to cache gcc object files for builds inside docker, which I think will lead to problems. Solution for that might be a good documentation why shouldn't they do that. Build mounts are somewhat similar to run-time volumes as ways to circumvent the containment and ephemeral nature of a Docker container, so maybe if users already have to grasp that concept they will get the intended use for build mounts.
If you want "only data that must be persisted goes into volume" approach, make the mounts read-only, and promote Dockerfile shenanigans such as copying dependency specifications earlier into build, and installing them onto an lower fs layer to solve (some of) caching issues. I'm currently experimenting with that, and it ain't pretty, but it works (or would if it had an SSH key) with no external cache to break down.
Secrets in /run
Loading secrets from files and putting them in /run/ or environment variables is also workable, but puts more limitations on supporting file formats, structure of the secrets, etc. I think it will either be too restrictive (e.g. no support for variable number of secrets, or hierarchy/structure) or it will boil down to more complicated mount -ro option, users will struggle with for no practical gain.
Advanced secret management
Another approach would be doing something akin to k8s config maps and secrets, with yaml files declaring secrets, and instructions if they are to be available as env values or files. This leaves the least room for user doing stupid things, allows future "out of the box" support for configuration providers like etcd, or different vaults. But, the builds also happen on local dev machines, that probably won't use those. Also build secrets are different from running secrets, to be discarded as soon as build stage is finished. Frankly I think running secrets should not be handled by Docker at all, but on different layer by orchestrators.
I disagree. There are a number of ways to accomplish this right now without any specific build-secrets feature. Note that these aren't all necessarily good ideas, or easy ideas, but it is possible to do securely, when done carefully:
(these examples are neither exhaustive nor complete, and I leave implementation details as an exercise to the reader)
All this to say, it would be useful to have an easier-to-use secure secret introduction mechanism for build-time. But there are secure ways to do this, given a few different factors.
Also, at the end of the day, consider what the build-time secret is protecting. Generally a secret isn't worth protecting in and of itself, it's whatever that secret gives access to. If you're stuffing all manner of valuable IP into a container image, then you need to be careful to protect that image anyway.
In order to copy keys into image, (without networking) they must be first copied, or reside in context, which should never happen.
I need to use developers' ssh keys to access code repositories. That key is way more sensitive than the code. I've created additional set of keys that have read-only access to repositories, and I will be distributing that key with code that depends on those repositories. Because I refuse to copy someone's id_rsa into Docker context, even though I'm using multi-stage build. But this is absolutely not a solution, with some authorization keys going to all the places they should not be going, and being used by many different users and processes.
Now, I can use networking, mount volumes with NFS or other network storage (so, why is it not supported out of the box, with
simply means - no it's not possible. That's not how secure systems work.
It's quite easy to do this with build args. Just read the secret into a build arg when running the build command. In the Dockerfile, write the arg to a file, use it and remove it, all inside the same
The following example uses an SSH key when bundling a Ruby app.
FROM ruby WORKDIR /usr/src/app COPY Gemfile Gemfile.lock ./ ARG SSH_KEY RUN mkdir /root/.ssh \ && chown 0700 /root/.ssh \ && echo "$SSH_KEY" >/root/.ssh/id_rsa \ && chmod 0600 /root/.ssh/id_rsa \ && bundle install \ && rm /root/.ssh -fR
Build with docker
$ docker build --build-arg SSH_KEY="$(cat /path/to/ssh-key)" -t my-app .
Build with Docker Compose
version: "3.3" services: app: build: context: . args: SSH_KEY: "$SSH_KEY"
SSH_KEY="$(cat /path/to/ssh-key)" docker-compose build
I admit that this is not an ideal solution, but it works quite perfectly.
I think that is the main problem. A lot people don't realise that almost everything leaves their secrets somewhere in the image or something else (either metadata, image layer, shell history). There are some possibilities, but they are all quiet a bit of work, and you have to be very careful. But to quote @OJezu :
I think the mount solution would be best, as this won't embed the secret in the history of the shell, the mount will also make sure anything can read that file, or it can be symlinked (in the worst case the symlink will still exist in the resulting image, but that won't contain the actual secret.
I think the main reason people don't comment much on these topics anymore is because the past 3 years has shown that every single good proposal, or ready to merge solution, is getting shutdown. The main reasons I have seen: fear of being misused which will break the reproducable build. In my opinion this is not a good reason, for 2 reasons:
I think the whole reason while a lot of people just get really frustrated, and deterred from even attempt to contribute to this issue, is the long history without a solution. I would be willing to contribute, if it would actually fix this problem. But I don't see this happening unless the main maintainers decide this is a real and (after 3 years) a very urgent issue.
#13490 : more then 2 years old, it also refers back to this topic as preferred comment thread.
@oppianmatt Using multi-stage builds for this requires you to either:
Option 1 is difficult because it requires in-depth knowledge of what is being built. For example, if you need to install third-party packages (e.g. with
Option 2 breaks the image layer hierarchy, as every file in the image is changed in the new layer (it is really only useful for the final image to be
And neither of those allows for a persistent ephemeral disk cache of packages, forcing people to download the same packages every time, and do complicated gymnastics to avoid storing them in the image.
Using a volume mount (or a bind, in certain cases) would be fantastic for dealing w/ apt caches, and ruby, python etc. Maven too.
People already plan for the optional presence of volumes for all kinds of things today in finished images / containers. ex: mysql defaults to using an internal directory, but it perfectly happy to deal w/ a volume instead.
Multi-stage builds can be done in multiple steps, easily with (undocumented?, I don't know I found it by experimenting) building from earlier images.
FROM debian AS image-common # runtime deps FROM image-common AS image-build # build deps and build itself FROM image-common AS final-build COPY --from=image-build ./build-result ./
Lack of caching can be solved, or at least mitigated by downloading dependencies in an earlier step then copying application source. Benefit over mounting cache volume is not risking breaking of docker internal cache in any way. For PHP an example goes:
COPY composer.json composer.lock ./ # copy dependency spec first RUN composer install --no-scripts --no-plugins --no-autoloader --no-suggest --no-interaction COPY . ./ # copy all of the sources
I cannot find an acceptable solution for copying secrets though, and this is what this issue is about. Please, let us not dilute it with other problems. Maintainers must decide which solution should be pursued, and either do it or accept PR from community if somebody else does it.
Multi-stage builds are wonderful and well documented, but if build args + multi-stage builds are the recommended solution for build time secrets then this should be spelled out somewhere in the docs.
#32507 seems to be in the spirit of Docker/Moby...clean, flexible, and powerful. It solves this secrets problem and a host of other problems as well (e.g. avoiding copying files into the image which only need to be present for a single step). That gets my vote.
It looks like you're an ops guy trying to setup development environment on developer's machine. Is that really what you're trying to do? Isn't that developer's job in DevOps culture?
It looks like you call yourself a DevOps. But DevOps is not a synonym for Ops (i.e. "system administrator" or just "admin"). DevOps is a culture where Operations and Developers work together. "I'm DevOps" sounds similar to "I'm Agile" or "I'm Scrum". It doesn't make any sense.
And no, Docker is not meant to be used by Ops exclusively.
Consider your request is satisfied and now you're able to mount any file at build time. The next thing your security compliance guys will tell you that ssh keys shouldn't just seat at the disk in plain text. They should be encrypted with user's local password. So the next thing you ask from Docker is an ability to provide user password or otherwise unlock the ssh key. So you will be frustrated again.
For ssh keys we need an ability to unix sockets at build time so that we can provide SSH_AUTH_SOCK.
But the problem here is that Macs don't support unix sockets: docker/for-mac#483
So we need to go deeper.
I don't see how the discussed solution solves developer's SSH access problem. Unless you propose to store ssh keys on disk unencrypted.
I can't speak to his use case, but I can speak to the one I was trying to solve uptopic, and it had nothing to do with building for development machines. It had to do with the fact that our closed source project depends on other closed-source repositories of our own as gems and/or npm packages. Secrets are necessary to pull these packages at build-time no matter what the environment is.
I wound up finding a different way to solve the problem, but the fact that this issue remains not only unaddressed but virtually blown off is an absurdity.
It looks like we either have to allocate a separate ssh key for docker daemon or embed ssh agent/daemon into docker client/daemon.
Alternatively we have to abandon idea of building images in isolated environment and use client-side build.
Transferring or otherwise mounting ssh key from client to server first decrypting it on the client doesn't sound like a robust and secure solution.
You don't necessarily need ssh agent on the client (the current commit does). The client could do the same by just implementing