New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: The Docker Vault #10310
Comments
I like the idea of this being separate from the volumes, and using the direct mount points into the container. Forcing all secrets to be files, and to have a well known /run/ path also fits in with several other proposals and simplifies the interaction with the processes. It does mean the person providing the box and the image have to be in agreement on directory structure - so you have to change your images to take advantage of boxes. Did you consider letting the container specify where the boxes are mounted? |
@smarterclayton IMO it's much cleaner that people make their own symlinks to |
/ping @alexlarsson (since you were the main proponent of |
I like the idea; not sure what the implications are of allowing to specify a Portability could be an issue (thinking of deploying images to swarm; I need to have all boxes in place before deploying, right?) Just some initial things that came up while reading. Will read again and give it more thought. |
@thaJeztah No, @smarterclayton Also, boxes are directories that can have complete directory trees inside them. They're not just single files. |
From a user perspective, this sounds quite attractive - no separate daemon to worry about, and access to secrets on the host is controlled using familiar file system permission mechanisms, and via volume access from inside containers. It also decouples transfer of secrets from the orchestration layer - if a container being spun up needs a particular secret, then the orchestration layer will need to make sure the secret is in place on the container host first, but the container itself doesn't need to care how the secret got there. |
Couple questions: how (and where) is vault stored on disk? Is the box directly mounted in the container? If so, what happens if I inject box Nitpick: having two commands, Anyway, I really like this idea! I hope this will make it to 1.6 (together with squashing #9591). |
@cyphar how would you manage attempted operations at build time on paths provided by the BOX? or they would all fail by design? |
How would you feel about being able to flag certain box files as containing a set of key=value environment variables, and having Docker add them to the runtime environment of the container when creating and running it? Also, we couldn't have these env vars persisted when committing the container - they'd need to be separate from the container config. |
@ncdc using environment variables is a reiteration of a bad pattern. They eventually go off-sync after a restart and pollute the environment for all processes |
@gdm85 that may be true, but I'm thinking about images whose executable processes expect to use environment variables and trying to minimize necessary image modifications. To play devil's advocate, it's probably trivial to create a script to be the image's Command that could read the files from the box and turn them into env vars before invoking the actual executable. |
@gdm85 If a BOX requirement isn't fulfilled, then an empty box is mounted (from the proposal). If you're talking about a build process modifying a box inside a container, I'm still unsure if we should mount boxes @ncdc That is a completely separate proposal to this. The Docker vault is only concerned with storage of ethereal data on a container filesystem. Also, I personally don't like using environment variables from a |
The storage of the vault is not part of this proposal (it's an implementation feature which doesn't actually affect the API -- you could conceivably even store the vault data in memory and serialise it disk and it still wouldn't change the API). However, my current implementation will create a vault at
When you inject a box, a copy is mounted into the container because that's the only way to effectively follow the "changing a box will not change container state which already have it mounted". And this answers your next question too: if you have a running container with a box mounted, no operations on the box will affect the container state.
Two things:
Given that you have to specify which boxes will fulfil the image requirements (or just which ones you'll add), I can probably add some information in
|
It think it would nice for the proposal to sum up previous discussions and detail why having a new type of object Esp. If #8484 land before this, one could imagine having additional command to do CRUD operation on volume content thru the remote API, similar to the nice REST API surface you are proposing. |
was just curious
Copying the content of the box makes total sense to me. Therefore I don't think it's worth discussing whether it should be mounted
it was just a nitpick (how about |
@proppy I don't think that the Docker vault should be a special case of Docker volumes, just because the Docker vault allows you to store the data you want inside the Docker daemon and redistribute it to containers without having to worry about copies or consistency or whatever. In my opinion, it should be kept separate to volumes just because of the different purposes of both (given that vault boxes are copy-on-mount while volumes are just bind-mounts of host directories). But that's just my $0.02. |
@cyphar yes, I think that could be nice to sum up that point in the proposal description, esp. since I think this was also raised on the previous
s/while vaults/while volume/ If such a feature existed (copy on mount and copy on write volumes), would vault rely on it? |
@proppy If there's was a helper in the execdrivers to allow you to copy-on-mount (which I'm fairly sure is not a legit term) a box by bindmounting a As an aside, I think that such options to volumes would be a pretty cool feature (and I'd love to write |
Brainstorming here (so it may not make sense); What if boxes were implemented as images, kept in a separate image-store?
Obviously, it should not be possible to commit the containers (box instances) and/or push the boxes to the registry. However it would open the possibility to have a dedicated (private) registry for storing boxes. That (private) registry could be used to automatically deploy boxes to the docker hosts, without having to (manually) copy the files. Again, just brainstorming here.. |
Had a short chat with @cyphar on IRC about my previous blurb. Images are probably too much moving parts, just for managing a few files. A registry to store boxes is something that could be considered in the future, (including ACLs?) but definitely not part of the initial implementation; keep it simple for now. Giving it some more thought, I wonder if COW is really important for boxes; I expect secrets to be just a few files; would COW give advantages? (Might be overseeing things, so interested to hear if there are specific use cases for COW) 🐮 |
Hello! Sorry, this proposal has confused me a bit. Were you guys aware the 'Vault' is actually the name of a standalone precompiled go binary? From hashicorp: https://www.vaultproject.io/docs/install/index.html It seems like this proposal duplicates some of that functionality. It seems (from their software model) that docker support aught to be implemented some kind of a 'backend' plugin written for the main vault program. To make it work in docker containers. Sorry I'm just a bit confused about whether or not this proposal was meant to integrate with that one. And if it is not, then why do we need to be duplicating a lot of the generic aspects of the secrets management functionality? |
@dreamcat4 This proposal far outdates vaultproject, which is brand new. |
@cpuguy83 OK! On that note, [EDIT] have created a new issue on hashicorp's vault about Docker: |
I wonder if it wouldn't be easier to solve this by simply allowing people to specify The reason I'm suggesting this is because I'm worried that by having a new entity called "vault" that has a lifecycle that can be quite long, it means that people might forget about it. Which could pose a security issue. Whereas, requiring the client to specify the exact files/dir they want to expose into each container means we can delete those secret files with the container. e.g. |
I'm very 👎 about this idea. Managing secrets should not be Docker's concern. External volumes can solve this problem in a nice and non intrusive way. Take for example Keywhiz, a system that gives you strong guarantees to manage secrets. You'll be able to mount a fuse volume inside your containers and see the secrets by using this external volume: https://github.com/calavera/docker-volume-keywhiz-fs This is still in experimental, but there are going to be more improvements that will make it land in a release soon. |
Agreed, this should not be docker's concern and volume plugins enables some specific functionality around secrets. |
I agree. At the time this proposal was written, there was no clear strategy for handling secrets in Docker. By lack of alternatives at the time, this was an attempt to get things moving. Having said that, we still don't have an official roadmap for handling secrets (apart from some rough ideas / concepts), so #13490 is still needed badly. |
I'm not that familiar with those but will non-admin users be able to perform these actions? ie. will it require certain capabilities that might be disabled in certain environments? |
Hi everyone! We've come up with a simple solution to this problem: A bash script that once executed through a single Since we do all of this in a single RUN ONVAULT npm install --unsafe-perm Our first implementation around this concept is available at https://github.com/dockito/vault. To develop images locally we use a custom development box that runs the Dockito Vault as a service. The only drawback is requiring the HTTP server running, so no Docker hub builds. Let me know what you think :) I had previously mentioned this project at #6396 |
This proposal was to get the ball rolling on the fact that (because of the layered nature of Docker images), there's no reasonable way for us to store data ethereally during the build process such that it isn't saved in the image layers. Docker simply must provide the ability to tag a directory as "ethereal". Sure, the whole secrets thing may have been a bit overboard. A much better proposal IMO would be for us to be able to specify in a Dockerfile that certain directories are ethereal. This instruction would be a directive to the builder, and would not be added as a layer in the image graph. I'll write up a better proposal in a few weeks (if someone doesn't beat me to it :P). IMHO, in the sober light of day this proposal doesn't strike me as being particularly practical. |
@cyphar I have some old code that I was playing with that would mount the build context into the target container as a r/o volume. This allowed the executables running (via RUN cmd) access to files that were not COPY/ADD'd into the container/image. Could this be used to solve this vault/secret issue? Assuming people put those files into the build context. |
Awesome to hear that. I'm going to close this issue so we can move a future conversation somewhere else. 🤘 |
The Docker Vault
(aka The Art of Injecting Ethereal Data into Containers)
This proposal is a docs-based follow up to #6075 and #6697, with some
clarifications and improvements.
Purpose
The purpose for the "Docker vault" is to allow containers (both created in an
intermediate build step or created explicitly) to access data that is considered
to be "secret" while not allowing said data to be stored in images.
Essentially this makes the general purpose of the Docker vault to allow users to
inject arbitrary files into a container context while being assured that said
files will not be leaked into image layers if that container is committed.
This allows for certain use-cases to be accomplished with ease, such as:
allowed to be publicly available), while not storing said keys (which are
not allowed to be publicly available) in the intermittent layers.
container (such as SSL private keys) when said keys are not allowed to be
publicly available.
Overview
Data injected into a container using the Docker value will NOT (under any
circumstances) be saved when an image is created from the container. The data
is (to all intents and purposes) ethereal: you can only see it in a container
that has access to it, it cannot be saved into an image.
Containers can be given access to this data, and will be able to access this
data throughout their lifetime (i.e. purging files from the Docker value will
not remove it from running containers).
The data in the vault is conceptually stored in "boxes", where a "box" can be
considered a collection of files that are related in some fashion. Each box is
given a name, and containers are given access to specific boxes. An image can
hint what boxes it requires (just as it can hint the volumes it requires) and
boxes can be "aliased" in a container (you can tell a container that a box named
a
is actually a box namedb
which the image expects).Box names can consist of any characters except "
/
" and ":
".It is important to note that vault operations will NEVER affect a running
container, because a running container must remain consistent throughout its
lifetime.
Files in the Docker vault persist across restarts of the Docker daemon.
Containers access files they have been given access to by accessing the files
in
/run/vault/<box>/...
. Boxes can store directories too. All files anddirectories in
/run/vault/
are owned by UID and GID0
.Modifications of any kind to a box inside a container will not be reflected
in the vault's stored boxes. To this end, boxes are mounted
ro
inside acontainer (and
remount
ing themrw
will still not allow you to modifythe vault's stored boxes inside a container).
Usage
There are several ways to access the Docker vault functionality:
docker vault ...
: add, query, and delete data from the Docker vault.docker create --inject
: inject files from the Docker vault into a container.docker run --inject
: inject files from the Docker vault into a container.docker build --inject
: inject files from the Docker vault into theintermediate containers during the build.
Dockerfile
syntax explained below.docker vault
This subcommand has three classes of subcommands, with 6 subcommands in all:
Management (creates and destroys boxes):
docker vault create [options] <box>...
docker vault destroy [options] <box>...
Modification (adds files to and removes files from a box):
docker vault add [options] <box> <file>...
docker vault remove [options] <box> <file>...
Querying (lists information or accesses data inside boxes or the boxes
themselves):
docker vault list [options] [<box>...]
docker vault read [options] <box>[:<file>]...
It also has the following aliases (for the purposes of Unix-like simplicity):
docker valut ls
=>docker vault list
docker vault rm
=>docker vault remove
docker vault cat
=>docker vault read
It is very important to note that NONE of the
docker vault
subcommandswill affect running containers (whether or not they are using a box affected by
the vault operation). Containers use copies of vault data, not references to the
vault data itself -- in order to maintain consistency of a single container's
lifetime.
docker vault create [options] <box>...
This subcommand creates a new box called "
<box>
" with no files inside it. Ifa box with the given name already exists, this command will emit an error and do
nothing.
Options:
docker vault destroy [options] <box>...
This subcommand obliterates the box called "
<box>
" and any files storedwithin it.
Options:
-f
,--force
: ignore errors if the given box name does not exist.docker vault add [options] <box> <file>...
This subcommand adds the given list of files to the given box. The file names
are preserved when adding the files to the box.
If the box does not exist, the command will emit an error and do nothing.
If one of the paths in the given list does not exist, the command will emit an
error and continue execution.
If the given path points to a symlink, the symlink itself is copied verbatim.
If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).
If the path has some directory components, these will be reflected when the box
is injected into a container. In other words, boxes can store directories.
However, the path will be sanitised, so relative paths (
../a/b/c
) and absolutepaths
/a/b/c
will not be reflected. In both cases the paths would be preciselyidentical to
a/b/c
.If the given path points to a directory, the command will emit an error and
continue execution. If the
-r
flag is set, then all of the files anddirectories in that directory are also added to the box (as if their full paths
were also included in the command). If a directory with the given name already
exists, then the directories are merged.
Options:
-r
,--recursive
: recurses directories given, adding all of the contents ofthe directory to the box in addition to the directory itself.
docker vault remove [options] <box> <file>...
This subcommand removes the specified files from the given box.
If the path doesn't exist inside the box, then the command will emit an error
and continue execution.
If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).
If the path is a directory, then the command will emit an error and continue
execution unless the
-r
flag is set. If the-r
flag is set, then thedirectory and its contents are recursively removed.
Options:
-r
,--recursive
: recurses directories given, removing all of the contentsof the directories from the box in addition to the directory itself.
docker vault list [options] [<box>...]
This subcommand lists all files and directories stored inside the given boxes.
If no boxes are given,
docker vault
will list the files in every box stored inthe Docker vault.
If a given box does not exist, the command will emit an error and continue
execution.
Options:
-b
,--boxes
: only print the name of each box, not their contents.-f FORMAT
,--format=FORMAT
: formats each line with the given formatstring.
-r PATTERN
,--pattern=PATTERN
: only print entries where the filepaths match the given regular expression.
docker vault read [options] <box>:<file>...
This subcommand reads the contents of each file specified in the command line
and prints them to
stdout
. No information is printed about which box or filethe data came from.
If a path of box doesn't exist, the command emits an error and continues
execution.
If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).
If a path is a directory, the command emits an error and continue execution
unless the
-r
flag is set. If the-r
flag is set, then the directory isrecursed and all of its contents are
Options:
docker create [--inject <box>:[<alias>]]...
This option to
docker create
allows you to inject boxes into a container onits creation.
If an alias is specified, then the box is injected into
/run/vault/<alias>
.Otherwise, the box is injected into
/run/vault/<box>
.docker run [--inject <box>:[<alias>]]...
This option to
docker run
allows you to inject boxes into a container on itscreation.
If an alias is specified, then the box is injected into
/run/vault/<alias>
.Otherwise, the box is injected into
/run/vault/<box>
.docker build [--inject <box>:[<alias>]]...
This option to
docker build
allows you to inject boxes into each of theintermediate build containers during image creation. These boxes will (of
course) not be stored in the resultant image.
If an alias is specified, then the box is injected into
/run/vault/<alias>
.Otherwise, the box is injected into
/run/vault/<box>
.Dockerfile
Images can hint what boxes they expect in order to run (much like how volume
hinting works). If a hint is not fulfiled, then an empty box is mounted
instead.
Essentially the syntax has two forms (to mirror the
VOLUME
instruction):BOX <box> BOX ["<box>"...]
The first format is a legacy format, only allowing for one box name to be
specified. The second is the newer format, and it accepts a JSON array of
boxes.
Internals
The following documents the following internals:
RESTful API Changes
Several new endpoints will be added as a result of this functionality:
PUT /vault/<box>/
DELETE /vault/<box>/
PUT /vault/<box>/<path>
DELETE /vault/<box>/<path>
GET /vault/
GET /vault/<box>/
GET /vault/<box>/<path>
And several modified by this functionality:
POST /containers/<name>/create
POST /build
All of which are readibly apparent if you look at the docs for the command-line.
Container and Image Changes
Basically, both the container and image structures need to be updated to store:
Both of which are readily apparent if you look at the docs for the command-line.
/cc @shykes (this is a long one)
The text was updated successfully, but these errors were encountered: