Expose secrets to containers in environment variables

[pmorie]

Now that #4126 and #4514 have landed, it should be possible to expose secrets to containers as environment variables. While this is arguably not the optimal way to consume secrets, it will be important to users taking their first steps into a cloud environment, and we should offer this feature to enable a good experience for users who need it. For what it's worth, we consider this to be an important use-case for users of OpenShift.

Let's start with getting all the requirements and challenges on the table:

1. How should this be modeled in the API?
2. What limits are reasonable to place on secrets exposed to containers this way? For example, there are different upper limits on aspects of environment variables on different platforms.
3. Secrets are not static but environment variables are. Should the consuming containers be restarted when a secret value changes?
4. Should environment variables containing secrets be exposed via docker inspect*?
5. Environment variables that contain secrets should not be captured by docker commit*

*Applies to future equivalencies in other supported container runtimes

On all aspects of the problem mentioned above, we should take into account possible cross-cuts with:

1. Pass pod info to container (Pass pod labels to containers and to lifecycle hooks #560)
2. Dynamic configuration (Support dynamic configuration distribution #1553)
3. Sundry other things collected under the downward/upward API aggregator (Container downward/upward API umbrella issue #386)

[pmorie]

@erictune @bgrant0607 @thockin @smarterclayton @csrwng

[smarterclayton]

One option is an environment file option to the container that reads/sources a file from the volumes prior to container start. Or, alternatively, enable this from a pre start hook. I think we want to avoid ways of scraping data from disk that aren't file based (no "extract this var from this file"). It's also worth asking how prestart hooks could use secret volume data to generate config files.

I would recommend framing this in terms of the existing use cases described in the security proposals to find one really compelling scenario. I agree this needs to take into account the other issues.

[erictune]

I'm not yet seeing the use case for environment variables. They have some terrible properties, which @pmorie does list above.

First, not all users have stringent security requirements. For some users, having access to the namespace in which the pod is created means that it is probably okay for you to see the secret. So, for those users, they don't need to use the /secrets type. They can just put it in the image.

Second, it seems like in many cases, there is a chance to wrap the program that needs the environment variable in some shell script which can read a file from a secrets volume and stick the contents into an env var.

[pmorie]

@erictune @smarterclayton I've been thinking about the read-from-file wrapper. I'm not quite sure what the API should be, but it might be something we could do in conjunction with volumes. As a random thought, maybe if you want to consume secrets as env vars, you could have an 'env' attribute on a SecretVolumeSource that made the volume plugin generate a single env file.

[pmorie]

@erictune A number of images expect to use environment variables; a random sample:

1. https://github.com/tutumcloud/tutum-docker-mysql
2. https://registry.hub.docker.com/_/postgres/
3. Images that use conjur to manage secrets

I think these fall under the use-case defined in the security docs for using images from the community and having them work.

[pmorie]

I was thinking about this today and had some more specific ideas I will probably start prototyping. Today it would be possible to:

1. Create a secret with a key, env, which contained a base64-encoded env file.
2. Consume that secret in a volume and mount it into a container at /etc/secret-volume
3. Source /etc/secret-volume/env in the container's command

I don't actually think that the above is a bad way to do it, but the experience can be a little better. Taking it a step further, what about:

1. Make it possible to configure a SecretVolumeSource to generate an environment file from a Secret
2. Add a finalizer that rewrites the command to source the env files from secret volumes

[pmorie]

Will add example shortly

[pmorie]

@csrwng Any thoughts on the above?

[pmorie]

Example of current way it can be done today:

Secret:

{
  "apiVersion": "v1beta2",
  "kind": "Secret",
  "id": "test-secret",
  "data": {
    "environment-file": "TVlTUUxfVVNFUk5BTUU9dXNlcg0KTVlTUUxfUEFTU1dPUkQ9bm90YXNlY3JldA0KTVlTUUxfREJfTkFNRT1hcHBkYg==",
  }
}

The environment-file secret contains the following data:

MYSQL_USERNAME=user
MYSQL_PASSWORD=notasecret
MYSQL_DB_NAME=appdb

Pod:

{
  "id": "secret-test-pod",
  "kind": "Pod",
  "apiVersion":"v1beta2",
  "labels": {
    "name": "hello-openshift"
  },
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "secret-test-pod",
      "containers": [{
        "name": "app-container",
        "image": "someapp",
        "command": [ "bash", "-c", "source /etc/secret-volume/environment-file && app start" ],
        "volumeMounts": [{
          "name": "secret-volume",
          "mountPath": "/etc/secret-volume",
          "readOnly": true
        }]
      }],
      "volumes": [{
        "name": "secret-volume",
        "source": {
          "secret": {
            "target": {
              "kind": "Secret",
              "namespace": "test",
              "name": "test-secret"
            }
          }
        }
      }]
    }
  }
}

I think a better experience would be:

{
  "apiVersion": "v1beta2",
  "kind": "Secret",
  "id": "test-secret",
  "data": {
    "mysql-username": "dXNlcg==",
    "mysql-password": "bm90YXNlY3JldA====",
    "mysql-db-name": "YXBwZGI=",
  }
}

{
  "id": "secret-test-pod",
  "kind": "Pod",
  "apiVersion":"v1beta2",
  "labels": {
    "name": "hello-openshift"
  },
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "secret-test-pod",
      "containers": [{
        "name": "app-container",
        "image": "someapp",
        "command": [ "bash", "-c", "app start" ],
        "volumeMounts": [{
          "name": "secret-volume",
          "mountPath": "/etc/secret-volume",
          "readOnly": true
        }]
      }],
      "volumes": [{
        "name": "secret-volume",
        "source": {
          "secret": {
            "target": {
              "kind": "Secret",
              "namespace": "test",
              "name": "test-secret"
            },
            "env": {
              "filename": "environment-file",
            },
          }
        }
      }]
    }
  }
}

Which would result in a file equivalent to the one in the example above being generated at
/etc/secret-volume/environment-file. The volume plugin would be extended to handle generation of
this file. The finalizer would mutate the command to source the generated environment file.

Another use-case @liggitt brought up is that the SecretVolumeSource could allow users to specify
adaptations of secret data keys into environment variable names.

[csrwng]

@pmorie - how would the finalizer determine that a pod is not "finalized" and that it needs to update the run command, or for that matter the scheduler know that the pod can't be scheduled without being finalized?

[smarterclayton]

This is a separate discussion that is still unresolved - Brian and I sketched up a few solutions but I don't think there's a clear answer yet.

On Feb 26, 2015, at 6:16 AM, Cesar Wong notifications@github.com wrote:

@pmorie - how would the finalizer determine that a pod is not "finalized" and that it needs to update the run command, or for that matter the scheduler know that the pod can't be scheduled without being finalized?

—
Reply to this email directly or view it on GitHub.

[pmorie]

@csrwng Good question. Since there are a lot of questions around finalizers still I think for now we can generate the env file in the manner I described above and the user will need to include the bits about sourcing in their command. We can add finalization in the future.

[csrwng]

SGTM - I like the idea of the SecretVolumeSource, it's more consumable.

[pmorie]

Thought about this some more today and I'm going to prototype the following:

// SecretVolumeSource adapts a Secret into a VolumeSource.
//
// The contents of the target Secret's Data field will be presented in a volume
// as files using the keys in the Data field as the file names.
type SecretVolumeSource struct {
    // Reference to a Secret
    Target ObjectReference `json:"target"`

    // EnvAdaptations adapts the secret into an environment file
    EnvAdaptations *SecretEnv `json:"env"`
    // FUTURE: fine-grained control of files
    // FileAdaptations *SecretFiles `json:"files"`
}

// SecretEnv adapts secret data into an environment file.  The keys in the
// secret data are converted from DNS_SUBDOMAIN into environment variable
// names by upper casing and subsituting the '_' character for '-' and '.'.
type SecretEnv struct {
    // Path is the path of the environment file to generate.
    Path string `json:"path"`

    // Adaptations is a list of adaptations to apply to the
    // names of environment variables superceding the normal
    // algorithm.
    Adaptations []StringAdaptation `json:"adaptations,omitempty"`
}

// StringAdaptation specifies a mapping from a value to a new value.
type StringAdaptation struct {
    From string `json:"key"`
    To   string `json:"to"`
}

In the first iteration the user will need to source the generated file manually in the command.

[pmorie]

@erictune @smarterclayton @csrwng

[ironcladlou]

Command wrapping seems to come with its own concerns. The plugin finalizer would be making assumptions about at least:

1. The specific shell in use by the container entrypoint (e.g. bash)
   1. The use of (or desire to use) a shell at all by the container
2. The availability of a shell on PATH in the container image, or
3. The availability of a shell at an absolute path within the container image

[pmorie]

@ironcladlou That would be the finalizer, not the plugin, but yeah, I agree that those are concerns to think through

[liggitt]

@pmorie another use of envvars I just thought of is the filename of a secret being in an envvar, e.g. KUBECONFIG or CURL_CA_BUNDLE. Just another case to keep in mind.

[pmorie]

@LiGgit do you want it automatically when secrets are exposed via the
volume plugin?
On Fri, Feb 27, 2015 at 10:03 PM Jordan Liggitt notifications@github.com
wrote:

@pmorie https://github.com/pmorie another use of envvars I just thought
of is the filename of a secret being in an envvar, e.g. KUBECONFIG or
CURL_CA_BUNDLE

—
Reply to this email directly or view it on GitHub
#4710 (comment)
.

[pmorie]

@liggitt
On Fri, Feb 27, 2015 at 10:06 PM Paul Morie pmorie@gmail.com wrote:

@LiGgit do you want it automatically when secrets are exposed via the
volume plugin?
On Fri, Feb 27, 2015 at 10:03 PM Jordan Liggitt notifications@github.com
wrote:

@pmorie https://github.com/pmorie another use of envvars I just
thought of is the filename of a secret being in an envvar, e.g.
KUBECONFIG or CURL_CA_BUNDLE

—
Reply to this email directly or view it on GitHub
#4710 (comment)
.

[liggitt]

dunno... if the goal is to set an envvar the container is already looking for, then I assume some adaptation would be needed in the pod spec.

Ways I see a pod spec consuming secrets:

• Put secret A into file /path/to/secretfile ([with file mode 0600, ...])
• Put secret B into a file somewhere and put that filename in envvar $SECRET_FILE
• Put secret C into envvar $SECRET_VALUE

[csrwng]

@liggitt - for the case where you want the env var to point to a secret file... couldn't you just set that in advance?

[pmorie]

I had thought the same thing @csrwng, I'm still trying to decide if there's merit to generating it.

[liggitt]

mmm.. suppose so. that would be a combination of a regular env var and the "Put secret A into file /path/to/secretfile" case. that's probably fine

[thockin]

I didn't follow this whole thread in enough detail, but I want to throw out
a few things.

1. I am against most or all forms of command wrapping. There's precious
   little you can assume about user containers - may not have a shell, may not
   be a shell you like, etc.

2. Every tmpfs mount has a cost to the kernel ( a few 10s of KB if I
   recall) and real impact on other things - for example kubelet scans
   /proc/mounts. If/when we get to per-pod tmpfs mounts, we should try to
   think about how not to abuse that. Consider making the secrets volume
   handle more than one secret at a time?

3. I'm naturally predisposed against env vars, this conversation has not
   helped :)

On Mon, Mar 2, 2015 at 7:49 AM, Jordan Liggitt notifications@github.com
wrote:

mmm.. suppose so. that would be a combination of a regular env var and the
"Put secret A into file /path/to/secretfile" case. that's probably fine

—
Reply to this email directly or view it on GitHub
#4710 (comment)
.

[pmorie]

@thockin responses below:

1. Re: command wrapping, good point re: shell.
2. Re: per-pod tmpfs mounts, I would say that plugins should share a single per-pod mount amongst all volumes
3. But is it nature or nurture?

[tomkukral]

We would also really appreciate env files since we usually need to share same credentials in many containers in a pod (db, app, bacula-backup).

Describing security concerns in doc is imho sufficient to mitigate security issues.

[liggitt]

what would prevent such a user from looking into the secret pod to grab the info he needs?

Passwords, key values, etc, showing up in log files is less than ideal... not everything dealing with the log may treat it as confidential, and the data could persist long after the pod and tmpfs volume are safely removed.

Describing security concerns in doc is imho sufficient to mitigate security issues.

Perhaps, though allowing secret data to be used by the platform in that way strongly implies it has been done in a secure way. "It was documented to be insecure" is small comfort if the user didn't notice that documentation.

[smarterclayton]

Since a user can work around this in a non optimal way anyway, protecting
users from themselves can probably be taken too far. It's better for users
to use secrets to inject them into env (or files) than it is for them to
just use env from the start. So I'm slightly in favor of allowing secrets
-> env via downward API to at least ensure the secrets aren't in the API
objects.

On Nov 16, 2015, at 10:52 PM, Jordan Liggitt notifications@github.com
wrote:

what would prevent such a user from looking into the secret pod to grab the
info he needs?

Passwords, key values, etc, showing up in log files is less than ideal...
not everything dealing with the log may treat it as confidential, and the
data could persist long after the pod and tmpfs volume are safely removed.

Describing security concerns in doc is imho sufficient to mitigate security
issues.

Perhaps, though allowing secret data to be used in that way strongly
implies it has been done in a secure way. "It was documented to be
insecure" is small comfort if the user didn't notice that documentation.

—
Reply to this email directly or view it on GitHub
#4710 (comment)
.

[thockin]

+1. It would be great to set environment variables from a file in a Pod. Docker already support this feature with property --env-file. This feature mixed with secrets makes a powerful combination.

Unless I am gravely mistaken, Docker's env-file is evaluated OUTSIDE the pod. Having it read from inside the pod is actually a LOT more interesting, but as far as I can tell that is NOT what is happening.

[thockin]

That said, I see no reason not to support secrets -> env, with the
documentum that secrets -> volume is strictly better in every way.

On Tue, Nov 17, 2015 at 10:35 PM, Tim Hockin thockin@google.com wrote:

+1. It would be great to set environment variables from a file in a Pod. Docker already support this feature with property --env-file. This feature mixed with secrets makes a powerful combination.

Unless I am gravely mistaken, Docker's env-file is evaluated OUTSIDE
the pod. Having it read from inside the pod is actually a LOT more
interesting, but as far as I can tell that is NOT what is happening.

[mhausenblas]

+1 to what @thockin said:

no reason not to support secrets -> env, with the documentum that secrets -> volume is strictly better in every way.

In my experience, even when suggesting or providing good practices it's still a challenge for certain folks to follow along, oftentimes leading to bad outcomes such as people baking secrets into images because they struggle to use a Kubernetes Secret.

[resouer]

So it is preStart hook being expected in this place?

[thockin]

I think this is as simple as adding a new field in EnvVarSource

[smarterclayton]

As described in #4710 (comment)

[e72575f3]

Is there currently a way to set ENV variables or do I have to hardcode secrets into my source code?

[pmorie]

@e72575f3 Are you looking for simple env var control, or populating them with kubernetes secrets, which is the topic of this issue? You can avoid hard-coding secrets in your source with the existing secret volume functionality.

[e72575f3]

@pmorie Thanks! With the current secrets functionality I have to read the secrets in from files, correct? My legacy code expects them as ENV vars, is there another way to set ENV vars for containers? Then it would be less overhead to change the code, if not then so be it :)

[thockin]

Wrap your binary in a shell script that does export X=$(cat /env/x); export Y=$(cat /env/Y); my_binary --flag - you can always solve problems
with a level of indirection. :)

On Fri, Dec 4, 2015 at 8:52 PM, e72575f3 notifications@github.com wrote:

@pmorie https://github.com/pmorie Thanks! With the current secrets
functionality I have to read the secrets in from files, correct? My legacy
code expects them as ENV vars, is there another way to set ENV vars for
containers? Then it would be less overhead to change the code, if not then
so be it :)

—
Reply to this email directly or view it on GitHub
#4710 (comment)
.

[thockin]

sorry, we don't cherry pick features like this

On Thu, Mar 17, 2016 at 12:26 AM, Gorka Lerchundi Osa <
notifications@github.com> wrote:

any chance to cherrypick this into 1.1 branch?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#4710 (comment)

[clarelewis73]

I was wondering...

If I created a Java microservice and wanted to use secrets to store passwords. From the operations side of DevOps, how would I change this password without changing the base code? I.e. how would I expose this from k8s to my Java app? Is there a program in fabric8? or is there anything else?

Any help would be great thanks!

[wernight]

Secrets are exposed as files by default. You can also expose them as environment variables (just check that you can access them through your Java stack). However if you change them, you'll need to restart the Pod for the changes to get picked up.