Secrets: write-up best practices, do's and don'ts, roadmap

[thaJeztah]

Handling secrets (passwords, keys and related) in Docker is a recurring topic. Many pull-requests have been 'hijacked' by people wanting to (mis)use a specific feature for handling secrets.

So far, we only discourage people to use those features, because they're either provenly insecure, or not designed for handling secrets, hence "possibly" insecure. We don't offer them real alternatives, at least, not for all situations and if, then without a practical example.

I just think "secrets" is something that has been left lingering for too long. This results in users (mis)using features that are not designed for this (with the side effect that discussions get polluted with feature requests in this area) and making them jump through hoops just to be able to work with secrets.

Features / hacks that are (mis)used for secrets

This list is probably incomplete, but worth a mention

• Environment Variables. Probably the most used, because it's part of the "12 factor app". Environment variables are discouraged, because they are;
  • Accessible by any proces in the container, thus easily "leaked"
  • Preserved in intermediate layers of an image, and visible in docker inspect
  • Shared with any container linked to the container
• Build-time environment variables (#9176, #15182). The build-time environment variables were not designed to handle secrets. By lack of other options, people are planning to use them for this. To prevent giving the impression that they are suitable for secrets, it's been decided to deliberately not encrypt those variables in the process.
• Mark .. Squash / Flatten layers. (#332, #12198, #4232, #9591). Squashing layers will remove the intermediate layers from the final image, however, secrets used in those intermediate layers will still end up in the build cache.
• Volumes. IIRC some people were able to use the fact that volumes are re-created for each build-step, allowing them to store secrets. I'm not sure this actually works, and can't find the reference to how that's done.
• Manually building containers. Skip using a Dockerfile and manually build a container, commiting the results to an image
• Custom Hacks. For example, hosting secrets on a server, curl-ing the secrets and remove them afterwards, all in a single layer. (also see https://github.com/dockito/vault)

So, what's needed?

• Add documentation on "do's" and "don'ts" when dealing with secrets; @diogomonica made some excellent points in #9176 (comment)
• Describe the officially "endorsed" / approved way to handle secrets, if possible, using the current features
• Provide roadmap / design for officially handling secrets, we may want to make this pluggable, so that we don't have to re-invent the wheel and use existing offerings in this area, for example, Vault, Keywiz, Sneaker

The above should be written / designed with both build-time and run-time secrets in mind

@calavera created a quick-and-dirty proof-of-concept on how the new Volume-Drivers (#13161) could be used for this; https://github.com/calavera/docker-volume-keywhiz-fs

Note: Environment variables are used as the de-facto standard to pass configuration/settings, including secrets to containers. This includes official images on Docker Hub (e.g. MySQL, WordPress, PostgreSQL). These images should adopt the new 'best practices' when written/implemented.

In good tradition, here are some older proposals for handling secrets;

• "Add private files support" #5836
• "Add secret store" #6075
• "Continuation of the docker secret storage feature" #6697
• "Proposal: The Docker Vault" #10310

[thaJeztah]

ping @ewindisch @diogomonica @NathanMcCauley This is just a quick write-up. Feel free to modify/update the description if you think that's nescessary :)

[dreamcat4]

This is useful infos:

hashicorp/vault#165

As is this:

hashicorp/vault#164

[thaJeztah]

@dreamcat4 there are some plans to implement a generic "secrets API", which would allow you to use either Vault, or Keywiz or you-name-it with Docker, but all in the same way. It's just an early thought, so it will require additional research.

[dreamcat4]

@thaJeztah Yep Sorry I don't want to detract from those efforts / discussion in any way. I am more thinking maybe it's a useful exercise also (as part of that longer process and while we are waiting) to see how far we can get right now. Then it shows up more clearly to others the limits and deficiencies in current process. What underlying is missing and needed the most to be added to improve the secrets.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

And perhaps also (for docker) we may also be worth to consider limitations (pros/cons) between solutions that provide a mechanism to handle the secrets "in-memory". As opposed to a more heavily file-based secrets methods or network based ones e.g. local secrets server. Which are the current hacks on the table (until proper secrets API). This can help us to understand some of the unique value (for example of stronger security) added by a docker secrets API which could not otherwise be achieved by using hacks on top of the current docker feature set. However I am not a security expert. So I cannot really comment on those things with such a great certainty.

[thaJeztah]

@dreamcat4 yes, you're right; for the short term, those links are indeed useful.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

Thanks! I think I had that in my original description, must have gotten lost in the process. I will add a bullet

However I am not a security expert.

Neither am I, that's why I "pinged" the security maintainers; IMO, this should be something written by them 😇

[diogomonica]

@thaJeztah great summary. I'll try to poke at this whenever I find some time.

[thaJeztah]

@diogomonica although not directly related, there a long open feature request for forwarding SSH key agent during build; #6396 given the number of comments, it would be good to give that some thinking too. (If even to take a decision on it whether or not it can/should be implemented)

[ebuchman]

Assuming you could mount volumes as user other than root (I know it's impossible, but humour me), would that be a favourable approach to getting secrets into containers?

If so, I'd advocate for an alternative to -v host_dir:image_dir that expects the use of a data-only container and might look like -vc host_dir:image_dir (ie. volume-copy) wherein the contents of host_dir are copied into the image_dir volume on the data-only container.

We could then emphasize a secure-data-only containers paradigm and allow those volumes to be encrypted

[kepkin]

I've recently read a good article about that from @jrslv where he propose to build a special docker image with secrets just to build your app, and than build another image for distribution using results from running build image.

So you have two Dockerfiles:

• Dockerfile.build (here you simply copy all your secrets)
• Dockerfile.dist (this one you will push to registry)

Now we can build our distribution like that:

# !/bin/sh
docker build -t hello-world-build -f Dockerfile.build .
docker run hello-world-build >build.tar.gz 
docker build -t hello-world -f Dockerfile.dist ^

Your secrets are safe, as you never push hello-world-build image.

I recommend to read @jrslv article for more details http://resources.codeship.com/ebooks/continuous-integration-continuous-delivery-with-docker

[lamroger]

Thanks for sharing @kepkin !
Just finished reading the article. Really concise!

I like the idea of exporting the files and loading them in through a separate Dockerfile. It feels like squashing without the "intermediate layers being in the build cache" issue.

However, I'm nervous that it'll complicate development and might require a third Dockerfile for simplicity.

[TomasTomecek]

@kepkin no offense but that doesn't make any sense. Secrets are definitely not safe, since they are in the tarball and the tarball is being ADDed to production image -- even if you remove the tarball, without squashing, it will leak in some layer.

[thaJeztah]

@TomasTomecek if I understand the example correctly, the tarball is not the image-layers, but just the binary that was built inside the build container. See for example; https://github.com/docker-library/hello-world/blob/master/update.sh (no secrets involved here, but just a simple example of a build container)

[kepkin]

@TomasTomecek I'm talking about secrets for building Docker image. For instance, you need to pass ssh key to checkout source code from your private GitHub repository. And the tarball contains only build artifacts but doesn't contain GitHub key.

[TomasTomecek]

@kepkin right, now I read your post again and can see it. Sorry about that. Unfortunately it doesn't solve the issue when you need secrets during deployment/building the distribution image (e.g. fetching artifacts and authenticating with artifact service). But it's definitely a good solution for separation between build process and release process.

[kepkin]

@TomasTomecek that's exactly how I fetch artifacts actually.

In Docker.build image I download some binary dependencies from Amazon S3 image which require AWS key & secret. After retrieving and building, I create a tarball with everything I need.

[sameer-kumar]

@darmbrust Yes, that did the trick.
However, don't you think its smelly? Any better recommendations?
Thanks!

[binarytemple]

It's probably not as smelly as exposing your ssh-agent credentials over tcp via socat for any local process to steal, but indeed, as with anything related to secrets, 'docker build' is pretty smelly indeed. Actually, I'd forgotten that Docker for Mac can't expose Unix domain sockets on the osx host to the containers, so that opens up even more of a can of worms. My current solution, run a Centos VM, GitHub machine user account credentials go into it, build using the "Rocker" (deprecated) tool.

…

On Thu, 26 Jul 2018, 21:49 Sameer Kumar, ***@***.***> wrote: @darmbrust <https://github.com/darmbrust> Yes, that did the trick. However, don't you think its smelly? Any better recommendations? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13490 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZk5iz1kvCpZ0s65ng4TwL7LmHa9zZvks5uKitDgaJpZM4Eq021> .

[darmbrust]

This entire bug is smelly. I haven't found a better way... there are several other approaches above, but I think all of the other secure ones require standing up a little http server to feed the information into the image. maybe less smelly, but more complexity, more tools, more moving parts.

Not sure that anyone has found a "good" solution... we are all stuck waiting on the docker people to do something about it... don't hold your breath, since this bug was written in 2015, and they haven't even proposed a roadmap yet, much less a solution.

[binarytemple]

It's so simple and obvious, allow the mounting of volumes, (files or directories) into the container during the build. This is not a technical limitation, it's a decision to not allow secrets in order to preserve the behaviour of - check out, run build, same inputs, same output, change a build arg if you want to invalidate the cache... The problem is, the abstraction has becoming increasingly leaky with people using all sorts of kludgy, insecure hacks to get a "secret" into a container. Newsflash, exposing your SSH keyring via TCP, even on localhost is not secure, neither is passing credentials via environment variables (hint, run ps, or peek in the /proc filesystem), command arguments, and environmental variables are all there, naked, for the world to see. For developers of golang code this traditionally hasn't been an issue as they copy-and-paste their dependencies into their projects rather than using a dependency management tool, golang developers call this practice, 'vendoring'. For anyone working in other ecosystems where the build system fetches dependencies from Git, or repositories that require authentication, it's a big problem. I'm pretty sure there is some startup rule somewhere along the lines of, "don't presume you know, how, or why, your users use the product".

…

On Thu, 26 Jul 2018, 22:00 Dan Armbrust, ***@***.***> wrote: This entire bug is smelly. I haven't found a better way... there are several other approaches above, but I think all of the other secure ones require standing up a little http server to feed the information into the image. maybe less smelly, but more complexity, more tools, more moving parts. Not sure that anyone has found a "good" solution... we are all stuck waiting on the docker people to do something about it... don't hold your breath, since this bug was written in 2015, and they haven't even proposed a roadmap yet, much less a solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13490 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZk5nvbBTj4BAv5TtELNIHhJN8mU0Ctks5uKi38gaJpZM4Eq021> .

[cpuguy83]

@binarytemple Everyone who has ever worked on Docker/moby (as in the engineers behind it) know exactly what the problem is and have even run up against it.

Volumes is a solution that is itself incredibly leaky.
There is a proposal, mentioned up the comment stream a bit, that attempts to solve this in a reasonable manner (#33343)

The main thing here is providing the "right" abstraction rather than "any abstraction that happens to work"... we of course know this is painful for many in more than just this case.

Lots of work has been done on the builder lately which isn't necessarily visible yet, but the fruits of this effort will begin to show up in coming months.
To begin with, Docker 18.06 ships with an alternative builder implementation backed by https://github.com/moby/buildkit.
You may think "how does this help me?". Buildkit provides a lot of low-level primitives that enables us to be much more flexible in the Docker builder. Even so much as to be able to provide your own build parser (which can be anything from an enhanced Dockerfile parser to something completely different). Parsers are specified at the top of the "Dockerfile" and are just any image you want to use to parse the file.

If you really want to see something right now, you can take buildkit itself and run with it today, it sits on top of containerd, you can build a custom integration pretty quickly.

[tonistiigi]

Secret mounts support was added to buildkit in moby/buildkit#522 . They appear strictly on tmpfs, are excluded from build cache and can use a configurable data source. No PR yet that exposes it in a dockerfile syntax but should be a simple addition.

[BenoitNorrin]

There are 2 solutions to build images with secrets.

Multi-stage build :

FROM ubuntu as intermediate
ARG USERNAME
ARG PASSWORD
RUN git clone https://${USERNAME}:${PASSWORD}@github.com/username/repository.git

FROM ubuntu
# copy the repository form the previous image
COPY --from=intermediate /your-repo /srv/your-repo

Then : docker build --build-arg USERNAME=username --build-arg PASSWORD=password my-image .

Using a image builder : docker-build-with-secrets

[binarytemple]

@BenoitNorrin sorry, but you've exposed that password to every process on the host system. Unix security 101 - don't put secrets as command arguments.

[BenoitNorrin]

Yes but there are some usages where security matters a little less:

• you want to build on your own computer
• you build on your entreprise CI server (like jenkins). Most of the time it's about having access to a private repository (nexus, git, npm, etc), so your CI may have her own credentials for that.
• you can use a VM created from docker-machine and remove it after.

[Yajo]

If that's the only problem, @binarytemple, then simply adding the flag docker image build --args-file ./my-secret-file should be a pretty easy fix for this whole problem, isn't it? 🤔

[binarytemple]

@Yajo could be, yes it's at least a workaround until buildkit ships with secrets mount. Good suggestion. Thanks. B

[pvanderlinden]

Unfortunately most of the workarounds mentioned in these and the many other tickets still expose the secrets to the resulting image, or only works with specific languages where you only need dependencies during compile time and not during installation.

@binarytemple that will never happen, the docker maintainers have already killed at least one PR fully documented and fully implemented of a safe secret feature. Given the rest of history (this 3 year old ticket isn't the oldest and definitely not the only ticket/PR on this topic) I think it's safe to say the docker maintainers don't understand the need for security, which is a big problem.

[caub]

The biggest pain point is secret rotations for me

you've to maintain a graph of secret to services dependencies, and update twice each service (to get back to original secret name)

listing secrets from services doesn't seem to be easy (I gave up after some attempts around docker service inspect --format='{{.Spec.TaskTemplate.ContainerSpec.Secrets}}' <some_service>), listing services dependencies from docker secret inspect <secret_name> would be useful too. So I just maintain that (approximate) graph manually for now.

You also have to specify the secret destination, when it's not the default /run/secrets/<secret_name> in the docker service update command

I just hope for a simpler way to rotate secrets

[BretFisher]

@caub here's some CLI help:

Docker docs for formatting help come up with the rest of your inspect format:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName}}{{end}}'

That'll list all secret names in a service. If you wanted both name and ID, you could:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName .SecretID}}{{end}}' nginx

I always have my CI/CD (service update commands) or stack files hardcode the path so you don't have that issue on rotation.

With labels you can have CI/CD automation identify the right secret if you're not using stack files (without needing the secret name, which would be different each time).

[AkihiroSuda]

docker build --secret is finally available in Docker 18.09 https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

@thaJeztah Are we ready to close this issue?

[andriy-f]

For older versions of docker, using multistage build with copying secrets before build command is viable option, right?

FROM debian as build
COPY ./secret.conf /path/on/image/
RUN build.sh
...

FROM debian
COPY --from=build ...

[thaJeztah]

@andriy-f yes, that works, as long as you;

• (obviously) don't copy the secret to the final stage 😉, or:
• use the build stage / stage in which a secret is present as a "parent" for the final image
• never push the build-stage to a registry
• trust the host on which your daemon runs; i.e. taking into account that your "build" stage is preserved as an image; someone with access to that image would be able to get access to your secret.

[thaJeztah]

build time secrets are now possible when using buildkit as builder; see the blog post here https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

and the documentation; https://docs.docker.com/develop/develop-images/build_enhancements/

the RUN --mount option used for secrets will graduate to the default (stable) Dockerfile syntax soon

[pdemro]

Thank you @thaJeztah I did just a little more digging and found that article shortly after posting (previous post is now deleted). Thanks again!

[Vanuan]

Cool. That closes the build time secrets question. Anything for runtime/devtime (ssh in OS X)?

[brybalicious]

What is the current best-practice for run-time secrets to pass e.g. to an entrypoint script, without the need to set up some external service? Right now, it seems the build-time secret solution is easier than run-time. I could just mount a local credentials.json from the build context and parse it for any required build-time secrets. Run-time secrets, however are very confused, and there's no single idea of what constitutes best-practice.

[DXist]

Best practice is to encrypt secrets with a public key and allow only your runtime controller to decrypt and mount them. Encrypted secrets can be stored in a Git repository.

For Kubernetes there is https://github.com/bitnami-labs/sealed-secrets

[brybalicious]

Best practice is to encrypt secrets with a public key and allow only your runtime controller to decrypt and mount them. Encrypted secrets can be stored in a Git repository.

For Kubernetes there is https://github.com/bitnami-labs/sealed-secrets

Nice. What about in a simple setup that is not a cluster? Just a single Dockerfile..?

[DXist]

There are basic tools to store secrets in a repository like https://git-secret.io/

Possible workflow:

1. Generate public/private gpg key pair during Docker image build.
2. Import gpg public key from the image, reencrypt secrets, using CI private key for decryption
3. mount encrypted keys, decrypt them in Docker entrypoint

[binarytemple]

I’ve seen this pattern many times and wondered to myself, “why would the software development team (who are busy installing npm and god knows what on their machines)“, be entrusted with runtime secrets such as the magic key that allows you to run up a $50k google bill overnight? Surely this is a problem for the operations/security teams? Do they really need the application software developers handling this stuff?

[jan-hudec]

Surely this is a problem for the operations/security teams?

Yes. They need the tools and documentation the most. When they deploy with docker, they need to inject a lot of secrets in them. And they often version the compose files, kubernetes manifests, helm charts, terraform manifests, ansible playbooks or whatever they use to deploy the containers in git, so storing encrypted bundles there is useful to them too.

Do they really need the application software developers handling this stuff?

Operations don't need them to, but developers themselves do. The build and test infrastructure also involves a bunch of secrets.