Document how Secrets are stored and how to secure them

[jonmoter]

For purposes of security and compliance, it's important to my organization to understand how sensitive information is stored, and what the attack vectors are for someone to compromise the secrets.

In the current documentation, it doesn't make any mention of where Secrets are stored at rest. Are they in etcd? Are they stored in plaintext, or are they encrypted? If they're encrypted, where does the key live?

In other words, if I was an attacker trying to gain access to the Secrets stored in kubernetes, how could I read them and/or change them? Also, what best practices should an admin put in place to block an attacker?

[derekwaynecarr]

@pmorie

[erictune]

I'll take a quick stab at it, but @pmorie can say better when he has a chance.

Secrets are stored in etcd in plaintext.
Admins should limit read and write access to etcd to only admin users.

They will hit the disk that etcd uses. Admins may want to shred/wipe disks that ever were used by etcd once not used anymore.

Secrets are stored in plaintext on a tmpfs file on the nodes. They don't hit the disk on a node.

Secrets are encrypted with the master's private key when the secrets go across the wire between the master and a node. The master's private key is stored on the master's disk.

Currently, any user who can access the K8S API can see all the secrets.

By default, a process in a pod can also see all the secrets by accessing the k8s API via its default service account credentials.

A process in a pod cannot see secrets via the filesystem, if it was not configured to use those secrets.

[erictune]

The story will get better once we have authorization.

But, even when that is in we might want to think about doing the following things:

• should secrets be encrypted in etcd using the master's private key so that if the user misconfigures etcd to be more widely readable, that the secrets are not exposed?
• default policy for default service account to not be able to read any secrets (@liggitt @deads2k how do you treat this on openshift?)

Once the default policy for service accounts is sorted out, then our advice to admins should also say that admins should limit which users have direct access to nodes (to root namespace, rather than a container) regardless of whether as root or non-root.

[dalanlan]

/sub

[deads2k]

default policy for default service account to not be able to read any secrets (@liggitt @deads2k how do you treat this on openshift?)

For authorization purposes in openshift, service accounts are treated as a normal user. This means that a service account has no privileges by default. Openshift authorization rules can be scoped to namespaces and we suggest granting some privileges to certain service accounts, but we do not force them. In particular,

1. GET access to image streams (openshift concept that allows you to name images in the local docker registry) in your namespace for all service accounts in the namespace
2. UPDATE access to image streams in your namespace for your builder service account
3. GET/UPDATE for RCs and Pods in your namespace for your deployer service account

We provide project templates that provide our recommended role bindings.

[pmorie]

@jonmoter The info @erictune posted is accurate.

I will add:

One thing that the secrets proposal hints about is modularizing the data store for secrets so that we can transparently (to clients of the API server) differentiate the storage used by the API server for secret data to facilitate different users' requirements for how secrets are stored. One example of this that has been asked about is using Hashicorp Vault as the store for secrets (#10439), but if we define an abstraction for a secrets store, we could also use that to implement variations like encrypting secrets in etcd as @erictune mentioned.

[derekwaynecarr]

@pmorie - is there a reason why we would not want to encrypt in etcd when
at rest always?

On Thursday, July 30, 2015, Paul Morie notifications@github.com wrote:

@jonmoter https://github.com/jonmoter The info @erictune
https://github.com/erictune posted is accurate.

I will add:

One thing that the secrets proposal hints about is modularizing the data
store for secrets so that we can transparently (to clients of the API
server) differentiate the storage used by the API server for secret data to
facilitate different users' requirements for how secrets are stored. One
example of this that has been asked about is using Hashicorp Vault as the
store for secrets (#10439
#10439), but if
we define an abstraction for a secrets store, we could also use that to
implement variations like encrypting secrets in etcd as @erictune
https://github.com/erictune mentioned.

—
Reply to this email directly or view it on GitHub
#11937 (comment)
.

[pmorie]

@derekwaynecarr Probably not -- I mention it because it seems possible because it seems like the type of thing we would want to enable with a flag or config item since it would be a significant change to how the data is stored. One thing I thought of after posting earlier: what happens when you rotate the master's key? The data in etcd will have to be reencrypted with the new key, right?

[liggitt]

Normally with key rotation you have a single encrypting key and an ordered list of decrypting keys, so re-encrypting can be done live, while old data can still be read.

[pmorie]

@jonmoter Can I ask: do you feel that the secrets docs and proposal did not adequately document the info you requested?

[pmorie]

@jonmoter Disregard my Q, I had misread your original comment opening the issue.

[jonmoter]

The responses to this issue have been really useful, and have given me the information I'm looking for. What @erictune said in his first response to me covers most of what I was interested in, and it's also helpful to see what y'all are considering.

I think if you take the information in this thread (heck, even copy/paste the comment that @erictune made), and put that in the Security Properties section of the Secrets page, I'd be satisfied with that.

Thanks for the info!

[pmorie]

Closing since the secrets UG was updated with the requested info.