K8TRE - base infrastructure options: Secrets

[manics]

Kubernetes Secrets are an abstraction for storing multiple types of secrets including tokens, certificates, and arbitrary data. Can K8TRE function using only this standard secrets API, or does it have additional requirements?

Kubernetes provides default secrets storage but we can recommend alternative back-ends with increased security.

[manics]

AWS has several options:

• Encrypt by default using a KMS key, I think this is effectively the default Kubernetes secrets implementation, but using AWS KMS to encrypt secrets when they're stored in the cluster
• AWS Secrets manager requires the Kubernetes Secrets Store CSI Driver which also supports multiple backends from other vendors. The AWS Secrets backend supports using AWS IAM roles attached to pods to restrict access on top of the RBAC permissions supported by K8s
• Sealed Secrets allows secrets to be stored in git by encrypting them with a public key before they're added to git, and decrypting it with a private key inside K8S when the secret is requested.

[vvcb]

There are two related issues to solve:

1. How do we store secrets in k8s and make them available to applications?

We use Azure KeyVault and Kubernetes Secrets. Secrets stored in Azure KV are distributed as k8s secrets in the correct namespaces so that each application does not have to be configured to read directly from Azure KV. The pods then access these secrets either as environment variables or from a mount point.

However, by default, k8s secrets are stored as base64-encoded plain text. K8TRE should strongly discourage this. We must also consider encryption at rest for resources, including secrets, stored in etcd.

One option may be to use the secrets-store-csi-driver but with abstractions for different environments just as we are doing with storage.

However, the k8s-native KMS provider seems to solve this and will probably become our solution after testing in the MVP.

2. How do we generate secrets and get them into k8s in the first place?

At LSC-SDE, most secrets begin their life in Azure KV before making it into k8s through the secrets distributor. We are revisiting this.

Sealed Secrets appears to be a good solution but the idea of storing secrets (even encrypted ones) in git needs digesting.

We must avoid dependency on cloud-specific options for K8TRE. Abstracting storage, DNS, etc. is going to be difficult enough.

[manics]

I think storing encrypted secrets in a private Git repository is OK- ultimately it's not that different from how cloud password managers work.

For production TREs in organisations with significant IT infrastructure I think the optimal solution is to somehow hook in to whatever the organisation currently uses for secrets if possible- this supports centralised management of credentials across an org. I tihnk this means we'd have two recommendations, one for "standalone" TREs which require their own secrets management, and one for how organisations should integrate K8S secrets with their existing solutions.

[vvcb]

Related to #k8tre/k8tre#6