Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Custom Secret Targets #32571

Merged
merged 4 commits into from May 10, 2017

Conversation

@ehazlett
Copy link
Contributor

commented Apr 12, 2017

This adds support for custom secret target paths lifting the restriction of limiting secrets to /var/run. Here is an example:

$> docker service create --name test --secret source=app,target=/etc/app -t alpine ash
# brace yourself for this one-liner...  yolo
$> docker exec -ti $(docker inspect --format '{{.Status.ContainerStatus.ContainerID}}' $(docker service ps --format '{{.ID}}' test)) mount | grep tmpfs | grep app
tmpfs on /etc/app type tmpfs (ro,relatime)
$> docker exec -ti $(docker inspect --format '{{.Status.ContainerStatus.ContainerID}}' $(docker service ps --format '{{.ID}}' test)) cat /etc/app
TESTING

Refs #32336 (comment)
Depends on docker/swarmkit#2118

@ehazlett

This comment has been minimized.

Copy link
Contributor Author

commented Apr 12, 2017

/cc @tianon @thaJeztah @diogomonica

vendor.conf Outdated
@@ -105,7 +105,8 @@ github.com/docker/containerd 9048e5e50717ea4497b757314bad98ea3763c145
github.com/tonistiigi/fifo 1405643975692217d6720f8b54aeee1bf2cd5cf4

# cluster
github.com/docker/swarmkit d5232280c510d70755ab11305d46a5704735371a
github.com/docker/swarmkit d58bab69b2305019be600726bca5f82392f4f116 https://github.com/ehazlett/swarmkit
github.com/golang/mock bd3c8e81be01eef76d4b503f5e687d2d1354d2d9

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann Apr 12, 2017

Contributor

Don't re-add mock. It was removed here: #32511

var refs []swarm.SecretReference
c.Assert(json.Unmarshal([]byte(out), &refs), checker.IsNil)
c.Assert(refs, checker.HasLen, 1)
func (s *DockerSwarmSuite) TestServiceCreateWithSecretSourceTargetPaths(c *check.C) {

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann Apr 12, 2017

Contributor

Slightly out of scope, but I realized recently that we don't seem to have tests that actually exec into the container to check that the secrets exist in the right place and have the right content. It's been on my list to add this. If you have the bandwidth to do it, I think it would be very valuable. Inspecting the container only goes so far - it doesn't confirm the secret mounts are working as expected.

This comment has been minimized.

Copy link
@ehazlett

ehazlett Apr 12, 2017

Author Contributor

Same here. I'll get some added.

@ehazlett ehazlett force-pushed the ehazlett:secret-targets branch from 804738a to c18d51f Apr 12, 2017

srcPath := filepath.Join(container.SecretMountPath(), filepath.Base(r.File.Name))
mounts = append(mounts, Mount{
Source: srcPath,
Destination: r.File.Name,

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann Apr 12, 2017

Contributor

Doesn't this need to be prefixed with containerSecretMountPath if the target is not an absolute path?

for _, r := range container.SecretReferences {
// secrets are created in the SecretMountPath at a single level
// i.e. /var/run/secrets/foo
srcPath := filepath.Join(container.SecretMountPath(), filepath.Base(r.File.Name))

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann Apr 12, 2017

Contributor

What happens if two secret targets have the same base name?

if os.IsNotExist(err) {
return nil
for _, r := range container.SecretReferences {
srcPath := filepath.Join(container.SecretMountPath(), r.File.Name)

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann Apr 12, 2017

Contributor

In SecretMounts this uses the base name. The code here doesn't seem consistent.

Either way, we should break this out into a shared function.

This comment has been minimized.

Copy link
@ehazlett

ehazlett Apr 12, 2017

Author Contributor

Good point 👍

c.Assert(refs[0].File.Name, checker.Equals, testTarget)
testPaths := map[string]string{
"app": "/etc/secret",
"test_secret": "test_secret",

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 Apr 12, 2017

Contributor

Can we test directory traversal?

@ehazlett ehazlett force-pushed the ehazlett:secret-targets branch 2 times, most recently from fba5af8 to 1c247ae Apr 13, 2017

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Apr 14, 2017

Design looks good to me 👍 Moving this to code review, but let me know if there's objections 😄

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented Apr 14, 2017

It still looks like this will break when two secrets have the same target base name.

@thaJeztah thaJeztah added this to the 17.06 milestone Apr 25, 2017

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented Apr 27, 2017

@ehazlett: I'd suggest this borrow the code from configs in #32336. That stores the config under the full target path, so there isn't a risk of collisions from targets that share the same basename. Also, it would be good for the config and secret implementations to stay consistent.

Let me know if it would be helpful for me to carry this PR.

@ehazlett

This comment has been minimized.

Copy link
Contributor Author

commented Apr 27, 2017

Ya if you want to carry that's fine with me.

@thaJeztah thaJeztah added this to backlog in maintainers-session Apr 27, 2017

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from 4678b9d to c7154ba Apr 28, 2017

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented Apr 28, 2017

Force-pushed to rebase and change the "local" storage paths to use the full target path instead of just the basename (which could collide).

Tested manually:

root@4d4d089ce2bf:/go/src/github.com/docker/docker# docker secret create simple -
simple
wvgodvdsfnu1r9jj06kqiijmy
root@4d4d089ce2bf:/go/src/github.com/docker/docker# docker secret create secret2 -
secret2
xjdubg4uatd025z9bctmejwxo
root@4d4d089ce2bf:/go/src/github.com/docker/docker# docker service create --name top --secret simple --secret source=secret2,target=/path/to/secret busybox top
xoiziqqmqq2rlskffqkiccr5x
Since --detach=false was not specified, tasks will be created in the background.
In a future release, --detach=false will become the default.
root@4d4d089ce2bf:/go/src/github.com/docker/docker# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
bc7e8dbbcb7a        busybox:latest      "top"               4 seconds ago       Up 2 seconds                            top.1.i224djxoo17c7twruoaasigl9
root@4d4d089ce2bf:/go/src/github.com/docker/docker# docker exec -ti bc7e8 sh
/ # ls /run/secrets
simple
/ # cat /run/secrets/simple
simple
/ # cat /path/to/secret
secret2
/ #

When I have a chance (either later today or early next week), I'll expand the automated tests to check that the files actually exist inside the container in the right places, with the right content, using both relative and absolute paths. I'll also improve the tests to cover invalid targets and referencing the same secret twice under different targets.

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from bd69787 to 1fc752c Apr 29, 2017

@diogomonica

This comment has been minimized.

Copy link
Contributor

commented May 5, 2017

@cyli @aaronlehmann since we don't see any security issues and this would just be a hygienic decision (no transversal), it would probably just be best effort.

I also agree that we shouldn't allow relative paths at all. Unless someone can come up with a good use case.

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from b2f4942 to 181f66e May 8, 2017

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented May 8, 2017

Rebased

aaronlehmann added a commit to aaronlehmann/cli that referenced this pull request May 9, 2017
[WIP] Support Custom Secret Targets
CLI counterpart to moby/moby#32571. Just
involves vendoring github.com/docker/docker/opts.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from 181f66e to d6ec5fe May 9, 2017

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented May 9, 2017

Corresponding CLI PR in docker/cli#50 (really just vendoring opts, which is a bit odd...)

@johnstep
Copy link
Member

left a comment

LGTM

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented May 9, 2017

Handler for DELETE /v1.30/containers/9e returned error: Unable to remove filesystem for 9e6c79a949b080473b9d4778f6eb9b3c9fb06a142260912457d57306a3e6d476: remove /var/lib/docker/containers/9e6c79a949b080473b9d4778f6eb9b3c9fb06a142260912457d57306a3e6d476/secrets: device or resource busy

This seems to happen with any secret target right now.

And I end up with leaked mounts:

tmpfs on /var/lib/docker/containers/67c604df32844744fea5d2613dc850d1c85227fc0a9406f4008938b4975358e4/secrets type tmpfs (rw,nosuid,nodev,noexec,relatime)
}
return err
}

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 9, 2017

Contributor

I think after this is done we need to detachMounted(container.SecretMountPath())

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann May 9, 2017

Contributor

You are right. I'm not sure why this got removed. I think we should defer the original contents of this function - do you agree?

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 9, 2017

Contributor

Yep, makes sense.

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann May 9, 2017

Contributor

@cpuguy83: I believe I've fixed this. I'm not seeing any errors or leaked mounts when I delete a container that uses secrets. I'd appreciate it if you could confirm on your side.

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from d6ec5fe to 694e738 May 9, 2017

if _, err := os.Stat(container.SecretMountPath()); err != nil {
if os.IsNotExist(err) {
return nil
func (container *Container) UnmountSecrets() (err error) {

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

Somewhere in here we are now getting a spurious EINVAL.
Maybe we can just check if err == unix.EINVAL { if mount.Mounted(path) { return err }}

Although I'm really curious as to where it's happening.

This comment has been minimized.

Copy link
@ehazlett

ehazlett May 10, 2017

Author Contributor

For a little more context:

DEBU[3206] containerd: process exited                    id=2759b664f3a609398c03bcbc6af6ee3e9996097c2363d63a7c5a0100cf79037d pid=init status=0 systemPid=1631
DEBU[3206] libcontainerd: received containerd event: &types.Event{Type:"exit", Id:"2759b664f3a609398c03bcbc6af6ee3e9996097c2363d63a7c5a0100cf79037d", Status:0x0, Pid:"init", Timestamp:(*timestamp.Timestamp)(0xc421526db0)} 
DEBU[3206] Revoking external connectivity on endpoint test.1.3g1qqhu9spa9apbz7t28ixkvv (a8d45de538bfa41fe4b92dc866eda1426a653ebf79e68480ee768905c96f764d) 
DEBU[3206] DeleteConntrackEntries purged ipv4:0, ipv6:0 
DEBU[3206] Releasing addresses for endpoint test.1.3g1qqhu9spa9apbz7t28ixkvv's interface on network bridge 
DEBU[3206] ReleaseAddress(LocalDefault/172.18.0.0/16, 172.18.0.2) 
WARN[3206] 2759b664f3a609398c03bcbc6af6ee3e9996097c2363d63a7c5a0100cf79037d cleanup: failed to unmount secrets: invalid argument 
srcPath := container.SecretFilePath(*r)
if _, err := os.Stat(srcPath); err != nil {
if os.IsNotExist(err) {
return nil

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

continue

if _, err := os.Stat(container.SecretMountPath()); err != nil {
if os.IsNotExist(err) {
return nil
func (container *Container) UnmountSecrets() (err error) {

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

I don't think we should return errors anywhere in this function since it stops all processing but cleanup should still happen for any remaining items.
The parent is only logging this error anyway.

Maybe if we do want to continue to return errors we can just collect all the errors into a single error.

ehazlett and others added 4 commits Apr 11, 2017
support custom paths for secrets
This adds support to specify custom container paths for secrets.

Signed-off-by: Evan Hazlett <ejhazlett@gmail.com>
Use "local" secret paths based on the secretID
This prevents targets with the same basename from colliding.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Extend test coverage of secrets
Actually look inside the container to see if the secret data is present
and correct. Test absolute paths, relative paths, and just a basename.
Test the scenario where a service references the same secret under
different targets.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Use forked version of CLI for tests
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

@aaronlehmann aaronlehmann force-pushed the ehazlett:secret-targets branch from 694e738 to 0da7bd0 May 10, 2017

@aaronlehmann

This comment has been minimized.

Copy link
Contributor

commented May 10, 2017

I've reverted all changes to UnmountSecrets. I think these were incorrect to begin with. The individual secret mounts are passed on to runc and should be cleaned up by runc. All that's necessary is unmounting the tmpfs on the host.

PTAL

DOCKERCLI_REPO=https://github.com/docker/cli
DOCKERCLI_COMMIT=c3648a9c9400d45524cc71b8fca4085b192c626f
DOCKERCLI_REPO=https://github.com/aaronlehmann/cli
DOCKERCLI_COMMIT=fd5a5910409fe5ed862b29de45a66f2bf8056894

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

Need to change these back before merge.

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann May 10, 2017

Contributor

They can't be changed back before the merge. This would cause CI to fail - unless you think the CLI PR should be merged before this one (but that seems backwards).

These circular dependencies are very hard to deal with.

This comment has been minimized.

Copy link
@johnstep

johnstep May 10, 2017

Member

Don't we need to keep this until docker/cli#50 is merged?

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

We can move the tests to a separate PR.

Opts package should really not be in this repo.
We're only using a couple of functions from it for cmd/dockerd.

This comment has been minimized.

Copy link
@aaronlehmann

aaronlehmann May 10, 2017

Contributor

The plan I discussed with @tiborvass and @thaJeztah was to merge this PR with the DOCKERCLI_REPO change intact, then change this back to the main repo in a followup.

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

Really even the opts change could be made on it's own since the existing code is not valid as is.

This comment has been minimized.

Copy link
@cpuguy83

cpuguy83 May 10, 2017

Contributor

@aaronlehmann That's fine.

@cpuguy83
Copy link
Contributor

left a comment

LGTM

@cpuguy83 cpuguy83 merged commit 08fdce7 into moby:master May 10, 2017

6 checks passed

dco-signed All commits are signed
experimental Jenkins build Docker-PRs-experimental 33935 has succeeded
Details
janky Jenkins build Docker-PRs 42532 has succeeded
Details
powerpc Jenkins build Docker-PRs-powerpc 2901 has succeeded
Details
windowsRS1 Jenkins build Docker-PRs-WoW-RS1 13773 has succeeded
Details
z Jenkins build Docker-PRs-s390x 2638 has succeeded
Details
@ehazlett

This comment has been minimized.

Copy link
Contributor Author

commented May 10, 2017

FWIW tested on my end again and LGTM

aaronlehmann added a commit to aaronlehmann/cli that referenced this pull request May 10, 2017
Support Custom Secret Targets
CLI counterpart to moby/moby#32571. Just
involves vendoring github.com/docker/docker/opts.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
andrewhsu pushed a commit to docker/docker-ce that referenced this pull request May 19, 2017
Support Custom Secret Targets
CLI counterpart to moby/moby#32571. Just
involves vendoring github.com/docker/docker/opts.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 978aa7ede06b347629f9ac5b422cf1b723580e59
Component: cli

@ehazlett ehazlett deleted the ehazlett:secret-targets branch Jun 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.