Assign secrets individually to each task

[sirlatrom]

Signed-off-by: Sune Keller sune.keller@gmail.com

- What I did
Made every task get its own copy of a secret such that it can be different for each task in case a secrets plugin driver provides different values per task. The use case is to install a secrets plugin driver that fetches a unique value for every task from HashiCorp Vault, e.g. a set of database credentials, or an opaque token. The preliminary plugin code is available here: https://gitlab.com/sirlatrom/docker-secretprovider-plugin-vault.

- How I did it
I made the following changes:

1. I added a new func CombineTwoIDs() in identity/combined_id.go that combines two existing IDs into a third by concatenating the IDs, separated by a ..

2. While assigning a task to a node and considering its secret references, it is now checked whether the secret with the referred ID has been added as a dependency for the task in question as opposed to whether it has been added as a dependency for the entire assignment. Previously, this was done once per secret ID in an entire assignment, which meant the value for any given secret ID would only ever be fetched once during an assignment.

3. Iff any secret driver is used, and the driver returns DoNotReuse: true in the SecretsProviderResponse, the secret is given a new ID, namely the concatenation of the secret ID and the task ID separated by a ., and marked as internal, resulting in a "secret update" change added to the assignment per task rather than per secret.

4. At the time when a secret is retrieved for a task, the secret ID and the task ID are combined and the secret with the resulting ID is looked up. If it does not exist, the original secret ID is looked up, as per default behavior.

An improvement would be to allow the driver/plugin to decide this behaviour rather than always doing it for all drivers, but currently, the Options field is never filled out for a secret driver object, as it is created when the secret spec is created, and I cannot find any mechanism in swarmkit for looking up information about a given secret driver plugin, which is installed in the Engine. Another approach would be to reserve a special secret label for the purpose of deciding this behaviour, e.g. 'com.docker.swarmkit.secret.driver.per-task=true`, but that would leave the burden on the user to know which types of secrets in the plugin would need that label.

- How to test it
Install the plugin, run a Vault dev server, verify that each task gets its own token when the plugin's vault_token mode is used on a secret:

plugin_name=sirlatrom/docker-secretprovider-plugin-vault
docker container run --detach --name vault --publish 8200:8200 vault server -dev -dev-root-token-id=1234
sleep 1
docker container exec -i --env VAULT_ADDR=http://127.0.0.1:8200 --env VAULT_TOKEN=1234 vault vault policy write snitch -<<EOF
path "secret/data/*" {
  capabilities = ["create","read","update"]
}
path "auth/token/create" {
  capabilities = ["create", "update"]
}
EOF
echo -n '1234' | docker secret create secret-zero -
docker service create --mode global --constraint 'node.role == manager' --name vault-helper-service --secret secret-zero --restart-condition on-failure busybox tail -f /dev/null
docker plugin install --grant-all-permissions ${plugin_name}
docker secret create --driver ${plugin_name} --label "dk.almbrand.docker.plugin.secretprovider.vault.type"="vault_token" generic_vault_token
docker node ls --filter role=worker -q | wc -l | grep -q 0 && snitch_role=manager || snitch_role=worker
docker service create --constraint 'node.role == '$snitch_role --detach --name snitch --replicas 2 --env VAULT_ADDR=http://172.17.0.1:8200 --secret generic_vault_token vault sh -c 'echo -n "generic_vault_token: "; cat /run/secrets/generic_vault_token; echo'
docker service logs -f snitch

The result should be that each task of the snitch service outputs a different value for the generic_vault_token secret, e.g.:

snitch.1.mrqrnv6cu7b9@redacted      | generic_vault_token: 17275db7-c139-a10d-e40c-847a9074345b
snitch.2.ok18z0rqw01c@redacted      | generic_vault_token: e1b24367-81cf-a86e-279c-f4187be6c820
snitch.2.usunxsf926kh@redacted      | generic_vault_token: 00b42ad2-d75c-24f3-38ca-f7ad8177201a
snitch.1.vc5hq4yoqqiv@redacted      | generic_vault_token: 4458b0e2-f1fd-bde7-36aa-da3d15e8d7e0
snitch.2.qygl3f1jrvqz@redacted      | generic_vault_token: 646bcac2-1e88-afe3-fdc4-ceadfc22e33e
snitch.1.nnwqfl0pa0x4@redacted      | generic_vault_token: 838ad7f2-793c-4727-73c6-4abd38ec5757

- Description for the changelog

Assign individual secret values to each task in a service when a secrets driver plugin indicates the secret is not to be reused.

- Cute animal

[sirlatrom]

Ping @dperny

[codecov]

Codecov Report

Merging #2735 into master will decrease coverage by 0.12%.
The diff coverage is 93.1%.

@@            Coverage Diff             @@
##           master    #2735      +/-   ##
==========================================
- Coverage   61.85%   61.72%   -0.13%     
==========================================
  Files         134      136       +2     
  Lines       21876    21907      +31     
==========================================
- Hits        13532    13523       -9     
- Misses       6882     6918      +36     
- Partials     1462     1466       +4

[sirlatrom]

I've updated the description with the approach I settled on, which works with passing CI.

@dperny PTAL

[dperny]

To address my comment (and also yours about options being only in the spec) we may need to add fields to the protos to accommodate the desired behavior.

[sirlatrom]

@dperny I've made changes as per your feedback by adding a SecretMappings field to the Task proto. I updated the PR description to match this.

[sirlatrom]

@justincormack Any chance you could weigh in here?

[sirlatrom]

@dperny The changes made should satisfy the requested changes. Is there any path forward for this PR? CC: @justincormack

[dperny]

Sorry, @sirlatrom. I have too many things going on concurrently right now and things are slipping, including a review of this. I'm reviewing right now.

[dperny]

is the information in secretMappings not a superset of the information in secretIDs? or does secretIDs also include the ephemeral task-specific secrets?

[sirlatrom]

The information in secretMappings only overlaps with secretIDs, as in all the keys of secretMappings are present in secretIDs, but not all the keys of secretIDs are necessarily present in secretMappings. Besides, the values in secretMappings are only present in that map, as secretIDs is a map[string]struct{}.

[dperny]

I feel like this has been mentioned before, but I cannot recall what the answer is... is there a reason we can't remap secrets in a way that is transparent for the task? For example, for secrets that are task-specific, using a key of secret ID and task ID together to pick the real secret information?

[sirlatrom]

I feel like this has been mentioned before, but I cannot recall what the answer is... is there a reason we can't remap secrets in a way that is transparent for the task? For example, for secrets that are task-specific, using a key of secret ID and task ID together to pick the real secret information?

I'm very much in favor of such an ID, as long as it does not invalidate any constraints there may be on ID lengths or anything like that. I presume having an ID of <secretID>.<taskID> would make troubleshooting the local store contents easier as compared to the new random ID. I'll make the change shortly.

[dperny]

I have this gut feeling that adding a secrets_mapping to the Task object is a bad idea, but I'm unsure how to articulate it.

However, after discussing out-of-band with @sirlatrom, we have the kernel of an idea that might work in lieu of that.

Basically, the rough idea is that you would leave the task object totally alone, and push all the changes off onto the Secret object and the things that manage its lifecycle.

When a Task using a secret that is task-specific (still unclear how we would indicate this), instead of assigning the secret directly, the Dispatcher get the secret value from the Driver, and then mints a new api.Secret object. The ID of this object would be the Task ID somehow combined (xor'd, maybe? concatenating is also a possibility, but I like it less) with the secret ID. This new secret created solely service this one task is ephemeral; it never gets written out to disk, and is basically totally gone from the manager as soon as it's been sent down as an assignment. It is, consequentially, opaque to the user. The only place the user can ever see the actual value of the secret is in the task itself.

When the executor goes to retrieve the Secret for a Task, it goes through a SecretGetter interface, which is implemented in agent/secrets/secrets.go as a taskRestrictedSecretsProvider. We'd alter the functionality of taskRestrictedSecretsProvider to do something like this:

// taskRestrictedSecretsProvider restricts the ids to the task.
type taskRestrictedSecretsProvider struct {
	secrets   exec.SecretGetter
	secretIDs map[string]struct{} // allow list of secret ids
        taskID    string
}

func (sp *taskRestrictedSecretsProvider) Get(secretID string) (*api.Secret, error) {
        if _, ok := sp.secretIDs[secretID]; !ok {
                return nil, fmt.Errorf("task not authorized to access secret %s", secretID)
        }

        // we'll first try getting a secret that is the result of combining the secret and task ID
        if secret, err := sp.secrets.Get(SomeIDXorFunc(secretID, sp.taskID)); err != nil {
                // if that fails, the secret must not be task-specific, so we get just the secret ID
                return sp.secrets.Get(secretID)
        }
        // then, we alter the secret ID, re-write it so the task never knows it's dealing 
        // with a "special" secret. In practice, this might involve copying the object so
        // we don't alter its value in the local secret store
        secret.ID = secretID
        return secret, err
}

Another option may be to set a flag on the secret spec of the "real" secret that says it's a task-specific secret, and then try getting by raw secret ID first, returning an error if the secret is one that is task specific, and trying again for the task-specific version in the case that we get that error.

The over-writing of the secret ID should be fine; once the secret leaves the local secret store, it's no longer really any of swarmkit's concern. It should only be used for bootstrapping the container, so if there are multiple secret objects floating around with the same ID but different data, they should never actually come into conflict with each other.

This approach gives us one really big benefit, in my opinion: it removes all visibility of the implementation of task-specific secrets from the user's view. This means that if this architecture turns out to be inflexible, insecure, or otherwise bad, we can change it substantially without worrying about breaking users.

I'm not sure what the correct answer is. This is certainly a more complicated approach to things. Additionally, I'm unsure about the feasibility of combining IDs like this, both from a usability and security perspective.

I'd appreciate a weigh-in from someone who has more experience with this kind of architecture. I'd be happy to write up a more detailed description of the secrets lifecycle, if needed.

[sirlatrom]

Its probably ok to squash everything into a single commit @sirlatrom

OK, I've squashed it into a single commit . Awaiting flaky CI or, if I'm lucky, a review :) and rebased on master.

[anshulpundir]

Looks good, minor comments.

[justincormack]

This xoring IDs is really gross. You should not xor strings, thats a type failure. If the ids are random IDs they should be stored as 256 bits in fixed length byte arrays, and then maybe xor is ok (although there are perhaps better options).

[sirlatrom]

This xoring IDs is really gross. You should not xor strings, thats a type failure. If the ids are random IDs they should be stored as 256 bits in fixed length byte arrays, and then maybe xor is ok (although there are perhaps better options).

My initial take was to append the task ID to the secret ID (separated by a dot) in order to generate a task-specific ID.

My second take was to generate new IDs entirely and add a secrets mapping field to the Secret proto to hold a map from the original IDs present in the spec to the new IDs present in the assignment.

@justincormack Are any of those takes more preferable, or is there a fourth one that would be better?

[justincormack]

I think appending with a dot is most readable unless there is any reason not to do so (I can't think of one).

[sirlatrom]

I think appending with a dot is most readable unless there is any reason not to do so (I can't think of one).

@justincormack I just added 7b98e00, which implements that. The only thing I can think of, which speaks against appending the task ID to the secret ID, is that the length of the ID string will double. But does that matter?

@dperny @anshulpundir Any pros/cons to add?

[sirlatrom]

I would argue the CI failure for 554ddbd is not related to this PR.

[wk8]

Clever hack... Just a minor nitpick :)

[sirlatrom]

I've updated the PR description to fit with the recent changes.

[sirlatrom]

Looks like flaky CI (2fcd8ad), I just wanted to add a comment explaining why I changed the check for task dependencies...