Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secrets: write-up best practices, do's and don'ts, roadmap #13490

Open
thaJeztah opened this issue May 26, 2015 · 221 comments
Open

Secrets: write-up best practices, do's and don'ts, roadmap #13490

thaJeztah opened this issue May 26, 2015 · 221 comments
Labels
area/security status/needs-attention Calls for a collective discussion during a review session

Comments

@thaJeztah
Copy link
Member

Handling secrets (passwords, keys and related) in Docker is a recurring topic. Many pull-requests have been 'hijacked' by people wanting to (mis)use a specific feature for handling secrets.

So far, we only discourage people to use those features, because they're either provenly insecure, or not designed for handling secrets, hence "possibly" insecure. We don't offer them real alternatives, at least, not for all situations and if, then without a practical example.

I just think "secrets" is something that has been left lingering for too long. This results in users (mis)using features that are not designed for this (with the side effect that discussions get polluted with feature requests in this area) and making them jump through hoops just to be able to work with secrets.

Features / hacks that are (mis)used for secrets

This list is probably incomplete, but worth a mention

So, what's needed?

The above should be written / designed with both build-time and run-time secrets in mind

@calavera created a quick-and-dirty proof-of-concept on how the new Volume-Drivers (#13161) could be used for this; https://github.com/calavera/docker-volume-keywhiz-fs

Note: Environment variables are used as the de-facto standard to pass configuration/settings, including secrets to containers. This includes official images on Docker Hub (e.g. MySQL, WordPress, PostgreSQL). These images should adopt the new 'best practices' when written/implemented.

In good tradition, here are some older proposals for handling secrets;

@thaJeztah
Copy link
Member Author

ping @ewindisch @diogomonica @NathanMcCauley This is just a quick write-up. Feel free to modify/update the description if you think that's nescessary :)

@dreamcat4
Copy link

This is useful infos:

hashicorp/vault#165

As is this:

hashicorp/vault#164

@thaJeztah
Copy link
Member Author

@dreamcat4 there are some plans to implement a generic "secrets API", which would allow you to use either Vault, or Keywiz or you-name-it with Docker, but all in the same way. It's just an early thought, so it will require additional research.

@dreamcat4
Copy link

@thaJeztah Yep Sorry I don't want to detract from those efforts / discussion in any way. I am more thinking maybe it's a useful exercise also (as part of that longer process and while we are waiting) to see how far we can get right now. Then it shows up more clearly to others the limits and deficiencies in current process. What underlying is missing and needed the most to be added to improve the secrets.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

And perhaps also (for docker) we may also be worth to consider limitations (pros/cons) between solutions that provide a mechanism to handle the secrets "in-memory". As opposed to a more heavily file-based secrets methods or network based ones e.g. local secrets server. Which are the current hacks on the table (until proper secrets API). This can help us to understand some of the unique value (for example of stronger security) added by a docker secrets API which could not otherwise be achieved by using hacks on top of the current docker feature set. However I am not a security expert. So I cannot really comment on those things with such a great certainty.

@thaJeztah
Copy link
Member Author

@dreamcat4 yes, you're right; for the short term, those links are indeed useful.

Also it's worth considering about the different situations of run-time secrets VS build-time secrets. For which there is also an area overlap area.

Thanks! I think I had that in my original description, must have gotten lost in the process. I will add a bullet

However I am not a security expert.

Neither am I, that's why I "pinged" the security maintainers; IMO, this should be something written by them 😇

@diogomonica
Copy link
Contributor

@thaJeztah great summary. I'll try to poke at this whenever I find some time.

@thaJeztah
Copy link
Member Author

@diogomonica although not directly related, there a long open feature request for forwarding SSH key agent during build; #6396 given the number of comments, it would be good to give that some thinking too. (If even to take a decision on it whether or not it can/should be implemented)

@ebuchman
Copy link

Assuming you could mount volumes as user other than root (I know it's impossible, but humour me), would that be a favourable approach to getting secrets into containers?

If so, I'd advocate for an alternative to -v host_dir:image_dir that expects the use of a data-only container and might look like -vc host_dir:image_dir (ie. volume-copy) wherein the contents of host_dir are copied into the image_dir volume on the data-only container.

We could then emphasize a secure-data-only containers paradigm and allow those volumes to be encrypted

@kepkin
Copy link

kepkin commented Nov 13, 2015

I've recently read a good article about that from @jrslv where he propose to build a special docker image with secrets just to build your app, and than build another image for distribution using results from running build image.

So you have two Dockerfiles:

  • Dockerfile.build (here you simply copy all your secrets)
  • Dockerfile.dist (this one you will push to registry)

Now we can build our distribution like that:

# !/bin/sh
docker build -t hello-world-build -f Dockerfile.build .
docker run hello-world-build >build.tar.gz 
docker build -t hello-world -f Dockerfile.dist ^

Your secrets are safe, as you never push hello-world-build image.

I recommend to read @jrslv article for more details http://resources.codeship.com/ebooks/continuous-integration-continuous-delivery-with-docker

@lamroger
Copy link

Thanks for sharing @kepkin !
Just finished reading the article. Really concise!

I like the idea of exporting the files and loading them in through a separate Dockerfile. It feels like squashing without the "intermediate layers being in the build cache" issue.

However, I'm nervous that it'll complicate development and might require a third Dockerfile for simplicity.

@TomasTomecek
Copy link
Contributor

@kepkin no offense but that doesn't make any sense. Secrets are definitely not safe, since they are in the tarball and the tarball is being ADDed to production image -- even if you remove the tarball, without squashing, it will leak in some layer.

@thaJeztah
Copy link
Member Author

@TomasTomecek if I understand the example correctly, the tarball is not the image-layers, but just the binary that was built inside the build container. See for example; https://github.com/docker-library/hello-world/blob/master/update.sh (no secrets involved here, but just a simple example of a build container)

@kepkin
Copy link

kepkin commented Nov 25, 2015

@TomasTomecek I'm talking about secrets for building Docker image. For instance, you need to pass ssh key to checkout source code from your private GitHub repository. And the tarball contains only build artifacts but doesn't contain GitHub key.

@TomasTomecek
Copy link
Contributor

@kepkin right, now I read your post again and can see it. Sorry about that. Unfortunately it doesn't solve the issue when you need secrets during deployment/building the distribution image (e.g. fetching artifacts and authenticating with artifact service). But it's definitely a good solution for separation between build process and release process.

@kepkin
Copy link

kepkin commented Nov 25, 2015

@TomasTomecek that's exactly how I fetch artifacts actually.

In Docker.build image I download some binary dependencies from Amazon S3 image which require AWS key & secret. After retrieving and building, I create a tarball with everything I need.

@binarytemple
Copy link

@yajo could be, yes it's at least a workaround until buildkit ships with secrets mount. Good suggestion. Thanks. B

@pvanderlinden
Copy link

pvanderlinden commented Jul 27, 2018

Unfortunately most of the workarounds mentioned in these and the many other tickets still expose the secrets to the resulting image, or only works with specific languages where you only need dependencies during compile time and not during installation.

@binarytemple that will never happen, the docker maintainers have already killed at least one PR fully documented and fully implemented of a safe secret feature. Given the rest of history (this 3 year old ticket isn't the oldest and definitely not the only ticket/PR on this topic) I think it's safe to say the docker maintainers don't understand the need for security, which is a big problem.

@caub
Copy link

caub commented Aug 14, 2018

The biggest pain point is secret rotations for me

you've to maintain a graph of secret to services dependencies, and update twice each service (to get back to original secret name)

listing secrets from services doesn't seem to be easy (I gave up after some attempts around docker service inspect --format='{{.Spec.TaskTemplate.ContainerSpec.Secrets}}' <some_service>), listing services dependencies from docker secret inspect <secret_name> would be useful too. So I just maintain that (approximate) graph manually for now.

You also have to specify the secret destination, when it's not the default /run/secrets/<secret_name> in the docker service update command

I just hope for a simpler way to rotate secrets

@BretFisher
Copy link

@caub here's some CLI help:

Docker docs for formatting help come up with the rest of your inspect format:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName}}{{end}}'

That'll list all secret names in a service. If you wanted both name and ID, you could:

docker service inspect --format='{{range .Spec.TaskTemplate.ContainerSpec.Secrets}}{{println .SecretName .SecretID}}{{end}}' nginx

I always have my CI/CD (service update commands) or stack files hardcode the path so you don't have that issue on rotation.

With labels you can have CI/CD automation identify the right secret if you're not using stack files (without needing the secret name, which would be different each time).

@AkihiroSuda
Copy link
Member

docker build --secret is finally available in Docker 18.09 https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

@thaJeztah Are we ready to close this issue?

@andriy-f
Copy link

For older versions of docker, using multistage build with copying secrets before build command is viable option, right?

FROM debian as build
COPY ./secret.conf /path/on/image/
RUN build.sh
...

FROM debian
COPY --from=build ...

@thaJeztah
Copy link
Member Author

@andriy-f yes, that works, as long as you;

  • (obviously) don't copy the secret to the final stage 😉, or:
  • use the build stage / stage in which a secret is present as a "parent" for the final image
  • never push the build-stage to a registry
  • trust the host on which your daemon runs; i.e. taking into account that your "build" stage is preserved as an image; someone with access to that image would be able to get access to your secret.

@thaJeztah
Copy link
Member Author

build time secrets are now possible when using buildkit as builder; see the blog post here https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066

and the documentation; https://docs.docker.com/develop/develop-images/build_enhancements/

the RUN --mount option used for secrets will graduate to the default (stable) Dockerfile syntax soon

@pdemro
Copy link

pdemro commented Sep 17, 2020

Thank you @thaJeztah I did just a little more digging and found that article shortly after posting (previous post is now deleted). Thanks again!

@Vanuan
Copy link

Vanuan commented Sep 18, 2020

Cool. That closes the build time secrets question. Anything for runtime/devtime (ssh in OS X)?

@brybalicious
Copy link

brybalicious commented Mar 4, 2021

What is the current best-practice for run-time secrets to pass e.g. to an entrypoint script, without the need to set up some external service? Right now, it seems the build-time secret solution is easier than run-time. I could just mount a local credentials.json from the build context and parse it for any required build-time secrets. Run-time secrets, however are very confused, and there's no single idea of what constitutes best-practice.

@DXist
Copy link

DXist commented Mar 4, 2021

Best practice is to encrypt secrets with a public key and allow only your runtime controller to decrypt and mount them. Encrypted secrets can be stored in a Git repository.

For Kubernetes there is https://github.com/bitnami-labs/sealed-secrets

@brybalicious
Copy link

Best practice is to encrypt secrets with a public key and allow only your runtime controller to decrypt and mount them. Encrypted secrets can be stored in a Git repository.

For Kubernetes there is https://github.com/bitnami-labs/sealed-secrets

Nice. What about in a simple setup that is not a cluster? Just a single Dockerfile..?

@DXist
Copy link

DXist commented Mar 5, 2021

There are basic tools to store secrets in a repository like https://git-secret.io/

Possible workflow:

  1. Generate public/private gpg key pair during Docker image build.
  2. Import gpg public key from the image, reencrypt secrets, using CI private key for decryption
  3. mount encrypted keys, decrypt them in Docker entrypoint

@binarytemple
Copy link

binarytemple commented Mar 5, 2021 via email

@ypadlyak

This comment has been minimized.

@aren55555

This comment has been minimized.

@KyeRussell

This comment has been minimized.

@jan-hudec
Copy link

Surely this is a problem for the operations/security teams?

Yes. They need the tools and documentation the most. When they deploy with docker, they need to inject a lot of secrets in them. And they often version the compose files, kubernetes manifests, helm charts, terraform manifests, ansible playbooks or whatever they use to deploy the containers in git, so storing encrypted bundles there is useful to them too.

Do they really need the application software developers handling this stuff?

Operations don't need them to, but developers themselves do. The build and test infrastructure also involves a bunch of secrets.

@petef19
Copy link

petef19 commented Sep 19, 2021

In our docker-compose file, we've been using shared volumes in read-only mode to pass secrets to the container that are needed for service configuration.
Once the container has successfully started and been configured, we then remove the shared files/folders on the host side. to leave no secrets on the host or in the container.
The container (and service running in it) has no issue with that as it only uses these secret files at start up.

Would love to get feedback about this approach. Anything wildly wrong with that ?

Thanks.

@lirantal
Copy link
Contributor

@petef19 you might benefit from the bit of knowledge on mounting secrets using the new DOCKER_BUILDKIT support in Docker: https://snyk.io/blog/10-best-practices-to-containerize-nodejs-web-applications-with-docker/

@petef19
Copy link

petef19 commented Sep 20, 2021

@lirantal

thanks for the link, great read.

Regarding using secrets, whether the newer build-secrets or the docker-compose-v3-secrets, the downside with that approach is that you still leave sensitive data inside the container - and the secrets can be read and accessed just like plain text files. If an attacker manages to hijack a container, it's open season.

With our approach ("unplugged" shared volumes) - and please chime in with opinions - there is no sensitive data left on the host or in the container (after it was build and service started). But the drawback may be scalability, as we currently only run a couple of servers, so not stress tested in that regard.

@lirantal
Copy link
Contributor

you still leave sensitive data inside the container

Well yes, but the point is that you mount secrets for a build stage that you don't actually use in the resulting image, and it's a nicer way of mounting data into that layer only.

@petef19
Copy link

petef19 commented Sep 20, 2021

you still leave sensitive data inside the container

Well yes, but the point is that you mount secrets for a build stage that you don't actually use in the resulting image, and it's a nicer way of mounting data into that layer only.

right, but what if you need the sensitive data to configure the service when the final container starts ?

@thaJeztah
Copy link
Member Author

Correct, with the build-secrets, the secrets will not be stored in the image-layer. They will be mounted during build, and no longer there after the build-step completes (there may be an empty file afterwards, due to how Linux mount points work (they require the destination to exist), but the secret themselves are not there).

For git operations, you can also use the --ssh option when using BuildKit, which will forward an ssh-agent into the container (https://docs.docker.com/develop/develop-images/build_enhancements/#using-ssh-to-access-private-data-in-builds)

@lirantal
Copy link
Contributor

It looks like @petef19 is seeking a run-time solution for when containers are running, not when they are built. To achieve that, there are the usual suspects:

  1. Put secrets in source code
  2. Pass environment variables
  3. Mount a directory for the secrets
  4. Use run-time API to fetch secrets (like say hashicorp vault and such)

** The above are organized from the least secure way of managing secrets to the most (my opinion only) **

@petef19
Copy link

petef19 commented Sep 20, 2021

It looks like @petef19 is seeking a run-time solution for when containers are running, not when they are built. To achieve that, there are the usual suspects:

  1. Put secrets in source code
  2. Pass environment variables
  3. Mount a directory for the secrets
  4. Use run-time API to fetch secrets (like say hashicorp vault and such)

** The above are organized from the least secure way of managing secrets to the most (my opinion only) **

yes, runtime secrets to configure services as they start up. I can extend your list...

(5) use cmd args
(6) if the service allows to read secrest from a file, use that option
(7) use shell scripts to init the service

Out of these, (3), (6) and (7) can all be used with shared volumes that can be "unplugged" after the service has started successfully. leaving nothing behind on host or container.

Question is could an attacker that hijacks the container still get out through the "unplugged" shared volume, although the "tunnel" does not exist anymore.... ? Especially if all secret mounts are read-only and are direct file mounts (not directories), so even if the shared volume was not "unplugged", wouldn't the attacker need a shared dir in order to attempt to traverse up... ?

Re (4): this seems to be a secure approach and easily scalable, but since the API connection creds are in the container (which are a secret itself) couldn't a hijacker use those creds to query the API for secrets.... ?

@Ilhamskhyi

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/security status/needs-attention Calls for a collective discussion during a review session
Projects
None yet
Development

No branches or pull requests