Avoid putting secrets to ETCD

[max-lobur]

Current implementation creates an etcd object with base64 encoded secret, which may potentially leak later.

Allow to put secrets to a volume, e.g. add a mutation webhook which adds an initContainer, which fetches & writes secrets to a volume shared with a container.

Pros:

• no secrets in etcd
• iam role can be granularly assigned to each initContainer (based on namespace annotation for example) when using https://github.com/uswitch/kiam , no need to give access to all secrets to the controller

[silasbw]

Hey @max-lobur, how much does using encryption at rest mitigate your concerns around etcd storage?

[max-lobur]

• It's just about etcd hacked/dumped eventually.
• Depending of https://coreos.com/etcd/docs/latest/op-guide/configuration.html#--auto-compaction-retention etcd even can have values of deleted secrets.
• Etcd (and k8s) backups must be treated carefully because now they contain secrets.

[silasbw]

Thanks, that's helpful for understanding the potential priority of an addition like this.

[silasbw]

I did some planning for this and some initial implementation work:

• POC Init Container example
• Issue to discuss / plan what the "API" is
• Implementation work to get the admission webhook server up
• Tie it all together end-to-end

Feel free to jump in a give feedback (especially on #78) or help out with coding 😛

[gazal-k]

• It's just about etcd hacked/dumped eventually.
• Depending of https://coreos.com/etcd/docs/latest/op-guide/configuration.html#--auto-compaction-retention etcd even can have values of deleted secrets.
• Etcd (and k8s) backups must be treated carefully because now they contain secrets.

If etcd is encrypted does all of this still matter? My understanding is that EKS does encrypt etcd; aws/containers-roadmap#263 (comment). I haven't verified, but I assume EKS uses KMS to encrypt etcd.

[max-lobur]

Yes it matters because it's unencrypted in etcd api / k8s api.

Implementation example https://github.com/cruise-automation/daytona
(fetching to in-mem volume)

[gazal-k]

Can't K8s API access be controlled using rbac?

I understand that there are integrations for vault that don't store secrets in etcd. This one: https://github.com/banzaicloud/bank-vaults uses a mutating admission webhook.

I'm not suggesting that they're all getting it wrong. I'm just trying to fully understand it if I can.

[kuuji]

Yes it matters because it's unencrypted in etcd api / k8s api.

Implementation example https://github.com/cruise-automation/daytona
(fetching to in-mem volume)

You can encrypt the actual value in etcd on EKS this way -> https://aws.amazon.com/blogs/containers/using-eks-encryption-provider-support-for-defense-in-depth/

Note that this is not at rest, the kms encryption happens at the k8s api level, before it's stored in etcd.

I believe other k8s implementations should support envelope encryption with a CMK as well.

[IamViditAgarwal]

Hi,
Any update on this, when it will come live. Waiting to see it

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.