Secret Management in Helm

[johnw188]

Consider configuration of a Jenkins instance. In our values.yaml we might encounter

Master:
  AdminUsername: admin
  AdminPassword: passw0rd!

with a secret being defined as

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  {{ if .Values.Master.AdminPassword }}
  jenkins-admin-password:  {{ .Values.Master.AdminPassword | b64enc | quote }}
  {{ else }}
  jenkins-admin-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{ end }}
  jenkins-admin-user: {{ .Values.Master.AdminUsername | b64enc | quote }}
{{- end }}

The problem is that anyone with access to tiller can run a helm get values command and be sent back the password in plain text.

Requirements

• Chart authors should be able to provide, through helm inspect, a listing of secrets that can be set for the chart
• Chart consumers should be able to edit values for these secrets through a --set directive or by passing a yaml file, as they do with values.yaml currently.
  • It should be obvious that secrets are being treated differently by Helm
• Consumers should be able to update secret values through a helm upgrade command
• Secrets will be available to chart templates in the same fashion as non secret values
• Once tiller executes it's commands against the kubernetes API server, the secret should not be accessible
• No assumptions are made about how chart authors utilize secret values. While the standard use case will be to create kubernetes secrets, this will be entirely handled through the existing template system.
• No additional effort will be made to improve upon the existing kubernetes secrets support from kubernetes core. That work should be undertaken at the core kubernetes level and then utilized by helm once it makes a release.

Proposed implementation

1. Allow users to specify secrets. There are two possible solutions here that I can see.

• Add a special, root key to values.yaml of Secrets (helm:Secrets?). This would inform Tiller that the values under that key are to be treated as secret values. The downside to this approach is it feels a bit magick-y.
• Split the existing values.yaml approach into values.yaml and secrets.yaml. This approach is more clear, but the existing workflow of helm inspect stable/jenkins > myvalues.yaml -> helm install -f myvalues.yaml stable/jenkins is quite elegant and I can't see a good way to maintain that simplicity with two separate config files.

2. Allow chart authors to reference secrets in their templates. This implementation is straightforward, I would change {{ if .Values.Master.AdminPassword }} to {{ if .Secrets.Master.AdminPassword }}
3. Ensure tiller is configured to receive and handle the new secret values.
4. Tiller should substitute secret values with a one way hash of the secret value before writing to its storage backend. This allows tiller to compare a provided secret with the previous value of that secret without needing to store the secret in plaintext.
5. Add a --set-secret command to helm to allow for command line setting/upgrading of secrets

[ejether]

I'm in favor of this. What is the proposed method for storing the plaintext secret in the code repository?

[technosophos]

The method proposed above does not seem like it would work with helm rollback or simple helm upgrade commands. The secret must be stored in an accessible (decryptable, not hashed) way in order to be re-injected into the templates.

I would be more interested in seeing an actual secrets storage mechanism (Vault-like) that we could use to inject secrets into the template at template resolution time. I like the idea of having {{.Secrets.Get "someSecretName"}} as a way of accessing those in a template. In that model, the .Secrets object would actually retrieve the named secret from whatever backend secret storage was implemented. For example, if it were Vault, the .Secret.Get would contact Vault and ask for someSecretName. For keys that rotate, this would actually work quite well with rollback, upgrade, and other commands.

That said, I have no idea how we would proceed with supporting a secret storage backend. While developing the backend seems out of scope for Helm, bridging Tiller and the secret storage would be something I would absolutely salivate over.

[ejether]

I think it could be reasonable to use Kubernetes secrets to do what you're suggesting @technosophos instead of involving another service like Vault. Brainstorming a little:
if helm create secret was a wrapper for creating and uploading a secret ala kubectl create -f /some/secret/file.yaml, then referring to the secret in the Chart {{ .Secrets.Get "someSecretName" }} could intelligently retrieve and use the secret OR the developer can just utilize valueFrom: secretKeyRef in the deployment. The secret is store on kube, which helm and the user has access to. Updating a secret could force an upgrade. Whereas, doing a helm upgrade would not not necessarily require the secret to be in the values.yaml or an additional -f config.yaml file.

[johnw188]

You're right about helm rollback of course, I didn't consider that. One suggestion that came up after I wrote this up was to simply store helm data in kubernetes secrets (instead of configmaps) and wait on kubernetes support for more robust secret managements.

[johnw188]

As for supporting a secret backend, the hardest thing would probably be ensuring tiller remains authenticated. For vault the actual operations would simply be API calls to set/retrieve secrets.

I'd imagine a Vault implementation would be as follows:

• One Vault path per chart, prefixed with some helm prefix (helm-myrelease)
• Each Vault key can have multiple key/value pairs as data. Keys would be the keys referenced in the Chart, values would be the secrets themselves.
• Secrets would either be passed through a file (through a --secrets=secretsFile) or prompted for by helm install/helm upgrade.
• Secrets aren't versioned, they exist outside of the helm upgrade/helm rollback flow.

Example:

End user

As an end user, I would run

helm install --name myrelease server/mychart
Please set values for secrets requested by mychart
# The password for your admin user
adminPassword: 
# Your github API key
apiKey:

These secrets are defined in a secrets.yaml sister file to the values.yaml config file, and the comments describing their usage are pulled from there.

Rotating secrets could be achieved by adding a helm update-secrets [release]/helm update-secrets --key apiKey [release].

I like the approach of prompting people to enter their secrets because it will keep them from committing them to source along with the rest of their config values while maintaining the simplicity of the helm install chart process. Prompting also keeps secrets out of your bash history.

Tiller

I have a release called myrelease, referencing two secrets - adminPassword and apiKey. They get referenced in the go template as {{ .Secrets.adminPassword }} and {{ .Secrets.apiKey }}. From the tiller side, upon receiving the secret values it would make a post request to the vault API as follows

curl -X POST -H "X-Vault-Token:$VAULT_TOKEN" -d '{"adminPassword":"passw0rd!", "apiKey":"asdf1234asdf1234"}' https://vault.vaultnamespace:8200/v1/secret/helm-myrelease

Then, at the point where tiller is finalizing the set of values to be used with the release, it would make an API call to vault again

curl -X GET -H "X-Vault-Token:$VAULT_TOKEN" https://vault.vaultnamespace:8200/v1/secret/helm-myrelease

which would return

{
  "auth": null,
  "warnings": null,
  "wrap_info": null,
  "data": {
    "adminPassword": "passw0rd!",
    "apiKey": "asdf1234asdf1234"
  },
  "lease_duration": 2764800,
  "renewable": false,
  "lease_id": "",
  "request_id": "5e246671-ec05-6fc8-9f93-4fe4512f34ab"
}

These values would get merged in with the rest of the values and dropped into templates as normal. My thought would be to use a .Secrets.key format as opposed to adding special functions to retrieve them because then they would work exactly like all other helm values when it comes to authoring charts.

[technosophos]

So if we combined @ejether 's suggestion with @johnw188 's well-thought-out ideas about Vault. It might be possible for us to support secret management with a default path of storing in Kubernetes secrets, and a more robust path that would support Vault.

That would make it simple for people using things like Minikube for dev, but also provide enterprises with a solution that would meet regulatory compliance.

I agree with @johnw188 's bullet list above. And even if we add secret rotation later, the proposal above makes sense to me as to how we would present that to the user.

I think that the experimental Rudder work that @nebril is working paves the way for making a plugin-style backend work. So this might be the ideal candidate for a second kind of Rudder backend. (See #2079)

[johnw188]

@technosophos Reading through the release module proposal you wrote up here, it seems that tiller would still be responsible for generating a complete set of template outputs before passing them on to the release module for installation. How do you see this integrating with that architecture, as the secret values will need to be pulled in during the template evaluation step.

I like the idea of having multiple secret backends, starting with kube secrets. We could define a simple interface along the lines of

type secretBackend interface {
	getSecrets() map[string]string
	getSecret(key string) string
	setSecrets(map[string]string)
	setSecret(key string, value string)
}

do an initial implementation backed by kubernetes secrets, then implement a vault backend.

[thomastaylor312]

@johnw188 Can you take a look at #2695 and see what you think? It is an expanded proposal that builds on this and could encompass both the normal secrets and vault use case

[RemingtonReackhof]

I would like to expand this proposal by adding for supporting for secret versioning. (This is copied from a separate issue for organization)

We want to move storage of Helm manifests to Secrets from ConfigMaps. This is favorable because of the future updates coming to k8s secrets storage in the near future (1.7). K8s will support secrets encryption at rest, making the usage of Tiller much more secure for storing versions of secrets than the current situation. In addition, there is talk within the Kubernetes community to expand support for multiple backends to handle secrets, specifically Vault. By staying with k8s secrets, we will be able to take advantage of these features for free.

Inside of the Helm chart templates, we will specify a secret object in which we will populate the values of the secrets by reading in a file called secrets.yaml which will resemble the format and function of values.yaml.

Tiller

Tiller will have a new storage option called secret:

tiller --storage=secret

This will configure Tiller to store and look for deployment versions in secrets instead of ConfigMaps. This will obviously ruin old deployed versions if one wanted to switch from configmaps to secrets storage, but potentially a one-time script could be made to port those over pretty easily.

Use Cases

Application Install

When a user installs a chart, have Helm look for a secrets.yaml file and populate any secret values that it defines in the templates where it is referenced. This will behave identically to the use of values.yaml, the caveat being that secrets.yaml should never be committed to your repository.

A new argument will be added to helm install and will act similarly to --values, namely its ability to specify a file from which to get secrets from:

helm install stable/mysql --secrets=my-secrets.yaml

If no file is specified, then Helm will default to using secrets.yaml.

The manifest of the chart’s version will then be stored as a secret instead of a ConfigMap. This way we will be able to support secret versioning that corresponds to deployment versions.

We'll also add support for a new argument --set-secrets which would work identically as --set for secrets.

Upgrading application

With the secrets.yaml file in your chart directory, updating secrets should be just as easy as updating values. Thus, another argument from upgrade would need to be added:

helm upgrade opining-rooster --secrets=secrets.yaml

Rolling back

Helm rollback should take care of this as old charts will point at old secrets

[thomastaylor312]

Tagged this as a feature since discussion seemed to be concluded and @RemingtonReackhof has some code in the works

[dustinmm80]

@RemingtonReackhof's workflow looks great to me, and that pattern may work well with security tools like Summon.

That said, are Secrets or ConfigMaps the best place for storing potentially sensitive charts?

Ideally this solution would support both string (API keys, etc) and file (SSL certs, etc) secrets. Given that the current implementation of K8S Secrets has several outstanding issues - unencrypted (base64 doesn't count), unauthenticated, unaudited, not well scoped, etc - seems like a Memory volume is a better fit. That way secrets can be shared within a Pod without clear boundaries (Secrets and ConfigMaps are pretty much global right?). In the Pod, a pluggable sidecar can handle authn/authz, fetching secrets, and passing them to the app container. The same or another sidecar could provide rotation-handling logic.

[thomastaylor312]

@dustinmm80 I think we all agree that secrets aren't perfect, however, encrypted secrets have been enabled as alpha in k8s 1.7 and so it is a good starting point that can be expanded later into using something like Vault.

[dustinmm80]

That sounds great, thanks a bunch @thomastaylor312! Appreciate all the hard work put into this project :)

[willejs]

@RemingtonReackhof @thomastaylor312 any news on this?

[bacongobbler]

Waiting on a second review (which I can tackle today).

[RXminuS]

Are you guys aware of this implementation? https://github.com/futuresimple/helm-secrets It's the nicest one I've seen so-far and feels a lot like other systems like Ansible, but would be great if instead of using a wrapper around the helm binary we can just integrate something similar with helm.

[thomastaylor312]

I am going to pull this out of 2.7 since the PR only partially implements this

[thomastaylor312]

/remove-lifecycle rotten
/lifecycle frozen

[monotek]

+1

[balan2010]

+1

[rogaha]

+1

[dragonsmith]

Hello!
For all who's interested, I wrote a tool that converts Tiller's ConfigMaps to Secrets like it was proposed in the comment above.
Take a look at this repo: https://github.com/dragonsmith/tiller-releases-converter

Here is a short write-up about the problem: https://dev.to/evilmartians/painless-migration-of-existing-helms-tiller-setup-to-kubernetes-secrets-d1p

Hope it will help somebody!

[bacongobbler]

@dragonsmith not to dissuade you, but Tiller has had the option to store release information as secrets via the --storage flag, and has been available since 2.7.0 :) https://docs.helm.sh/using_helm/#storage-backends

However, I could definitely see this being useful as a migration tool!

[bacongobbler]

To answer your questions, @itaysk,

Is this currently in 2.7/2.8?

Yes, storing release information as Secrets has been available since 2.7 via #2721. The comment you mention about pulling it off 2.7 was just the fact that we were close to considering pulling it off the milestone due to inactivity, but we managed to squeeze it in on the day before the release was cut. As mentioned in my comment above, documentation on how to enable the backend is available in the docs.

Is is considered "partially implemented" only because of single storage (kube secrets)?

No. It's considered "partially implemented" because the OP indicated interest in also adding support for splitting values.yaml into two objects: values.yaml and secrets.yaml, which was not a part of #2721. More information can be found in the original comment.

If we'll support additional backends, will the secrets be mixed in the resulting deployment files as plain text, or first saved in Kubernetes as secrets, then referenced from the template?

We don't plan on supporting additional backends for Helm 2. I'm not sure I understand the second part of your question though, would you mind elaborating more?

Also worthy to note that in Helm 3, state management of releases will be tracked through a Release object and a Secret. See https://github.com/helm/community/blob/master/helm-v3/003-state.md for more information.

[dragonsmith]

@bacongobbler

but Tiller has had the option to store release information as secrets via the --storage flag, and has been available since 2.7.0 :)

Yeah, I'm exploiting exactly that, and I've mentioned it in the write-up (sorry, it seems dev.to doesn't provide direct links to headers.)

Still, it is the problem to migrate from ConfigMaps to Secrets for Tiller if you have a lot deployed via Helm.

This small app is about helping you do that using default options, I'm not claiming more. It can convert ConfigMaps to Secrets, configure Tiller to use --storage option and cleanup old ConfigMaps.

[schollii]

We use sops to keep only encrypted file in storage, and decrypt on the fly, not perfect but works:

helm install -f <( sops -d hide/secrets.sops.yaml ) -n release_name chart
helm upgrade -f <( sops -d hide/secrets.sops.yaml ) release_name chart

[ejether]

@schollii are you also using --storage=secret flag? Otherwise, all your secrets are stored in the release Configmap

[schollii]

@ejether Thanks for reminder, it's on list of things to do soon!

[aba182]

Have we seen this project? https://github.com/futuresimple/helm-secrets

Seems like it offers many of the desired features but does not support additional external sources. If you stacked this with the secret storage setting seems like you would have security all the way through so long as you keep your GPG key safe.

[bacongobbler]

@aba182 helm-secrets seems a lot like the solution @schollii came up with; encrypt value files client-side and decrypt on-the-fly them prior to calling helm install or helm upgrade.

Both solutions miss an important use case though: helm-secrets and sops both decrypt the values prior to being submitted to Tiller, which means that the secret is stored in plain text in etcd.

If I'm understanding the use case, most users here seem to be asking for an end-to-end encryption solution from source code all the way to the runtime.

Has anyone tested whether or not pairing sops with encryption of secret data at rest works? That should cover the last security concern most people seem to have here, which is the issue of values being stored unencrypted in etcd.

[schollii]

@bacongobbler if the values are stored encrypted in etcd, then the container needs to decrypt them somehow, say with sops and an AWS profile mounted from a config map. But if that info is present in the container, then anyone able to enter the container (which presumably would be anyone able to snoop into etcd), can also decrypt them. Is there a way to prevent that?

[astorath]

@bacongobbler @schollii

1. You can setup your kube cluster to store data encrypted in etcd: https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
2. Preventing unauthorized access to your data (stored in secrets) is available with RBAC

[monotek]

Imho even encryption in etcd will not help as long you can do: "helm get values supersecretapp"

[astorath]

This is a problem of Tiller architecture, not secrets management. If you have Tiller access, you can do anything: even if these secrets are encrypted they are still accessible by tiller, cause it must be able to deploy apps that use the secrets.

[bacongobbler]

Imho even encryption in etcd will not help as long you can do: "helm get values supersecretapp"

If you're using the secret storage backend and asking Kubernetes to encrypt the release objects Helm creates, those values will be encrypted in etcd. The last case to cover in that case would be how Helm could decrypt those values. This should be much simpler to handle in Helm 3 given the client-only architecture, so I'm curious to hear if anyone's experimented with that feature, and how we can accomodate this so Helm can encrypt/decrypt those values on the fly, keeping that data encrypted at rest.

[bacongobbler]

This ticket has been open for over 3 years, yet there hasn't been any movement on this proposal since it first came up. I'm going to close this as it looks like there aren't any community members interested enough to implement the proposal.

If someone is interested in moving the proposal further, please feel free to chime in here and we can re-open this ticket. Thanks!