Document solution to problem of random passwords change on upgrade (from helm/charts)

[scottrigby]

I was replying in helm/charts#3591 and realized this is a much bigger problem that deserves a charts issue to decide whether we should agree on and document a policy for this across charts.

Can we address this at one of the next charts meetings?

Related:

• [stable/mysql] Update mysql user passwords on secrets change charts#3591
• [stable/percona] Support the new {{ .Secrets.password }} management charts#4258
• feat(2196): secrets management  helm#2721
• Secret Management in Helm helm#2196
• [incubator/keycloak] DB and Admin user secrets overwritten when upgrading charts#5129
• [stable/redis] helm upgrade change password secret charts#4472
• Add doc about redis data only persistence charts#4965
• Redis updates secret to invalid password charts#2538
• [patroni] Support updating database credentials via chart upgrade charts#2636
• Problems with randAlphaNum and upgrades helm#3053
• https://github.com/kubernetes/helm/blob/master/docs/charts_tips_and_tricks.md#automatically-roll-deployments-when-configmaps-or-secrets-change
• mongodb chart not able to run charts#2488

[unguiculus]

When I became a maintainer I learned it was best practice to auto-generate passwords and have told folks to follow this pattern. I never really questioned this but am not really happy about it either. I appreciate you brought up the discussion.

Auto-generating passwords is kind of nice. It makes it easy to get started and is certainly better than a hard-coded password. But for a real-world installation, I wouldn't rely on this. So, we need at least better documentation.

Thinking about it again, wouldn't it be better not to provide a default password at all? We could use the required function forcing the user to explicitly set a password. For CI, a custom values file could provide a password.

[unguiculus]

cc @kubernetes/charts-maintainers

[alexvicegrab]

This is indeed a problem. Currently I'm overcoming this by specifying a value with --set flag to override sensitive information (e.g. passwords) during the upgrade process.

I tend to save charts on git repositories but without the password. Ideally I neither want to:

1. Cause bloat, by saving one copy of each chart values.yaml for each (e.g.) PostGres chart I install, nor:
2. Cause security issues by saving passwords in Git repositories
3. Cause high complexity (what I currently do) by having to retrieve the passwords from the relevant Helm deployment secrets and resupplying them in the update command.

A way to automate 3 in a K8S/Helm way would go a long way.

[alexvicegrab]

e.g. it could be useful to add an annotation that prevents updating of a secret like this:

"helm.sh/resource-policy": keep

Except this only works for preventing deletion.

[geekuillaume]

I had the same problem. You can create a secret only on install by using a pre-install hook. The secret will be left untouched on upgrades. The only problem is that it's not managed by Helm and so not deleted when you delete the release.

[michaelfig]

I think the issue is not feeding back secrets into a chart upgrade, nor a problem with generating passwords. I think the issue is that if a chart is generating a random password, it should not overwrite an existing secret.

This seems like it should use the following wrapper around the secret definition:

{{- if or .Values.somePassword .Release.IsInstall}}
apiVersion: v1
kind: Secret
...
data:
  {{- if .Values.somePassword }}
  some-password: {{ .Values.somePassword | b64enc | quote }}
  {{- else }}
  some-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}
{{- end}}

Thoughts?

[alexvicegrab]

@michaelfig I like this, either we explicitly specify the password, or we autogenerate it during an install.

Does this work? (i.e. Have you tried running this to check that it does not entirely delete the secret during an update when not finding the secret template?)

[alexvicegrab]

Tried it myself just now, alas Helm will simply wipe out the secret.

[achempion]

The service is writing password to the secrets. Maybe it will be better to check first for existence and use old if a secret has a value already.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[alexvicegrab]

Is there any progress on setting up a way for random secrets to not be reset implicitly during upgrades?

[bacongobbler]

From the dev call, one possible solution to try would be through an install hook that writes out a configmap.

[bacongobbler]

That or checking for the presence of .Release.IsUpgrade.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[yurrriq]

Any update?

[arianitu]

This is pretty annoying, especially with things like Jenkins and Grafana, you're constantly looking up the new password on upgrades.

[giacomoguiulfo]

@arianitu Since Grafana is usually a not mission-critical component, a workaround is to add an annotation to the deployment's spec.template that looks like this:
checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}
to upgrade the Grafana pods every time the secret changes (when the password is re-generated). In my experience, this makes the pod be in sync with the secret most of the time.

[pastukhov]

Hi all!
My secrets keep deleting when chart upgrade, even "helm.sh/resource-policy": keep is set. My yaml:

{{ if .Release.IsInstall }}
{{- $graylogPasswordSecret := randAlphaNum 128 }}
{{- $graylogAdminPassword := randAlphaNum 64 }}
---
apiVersion: v1
kind: Secret
metadata:
  annotations:
    "helm.sh/resource-policy": keep
  name: {{ template "graylog.fullname" . }}-secrets
  labels:
    configVersion: v1
    app: {{ template "graylog.name" . }}
    chart: {{ template "graylog.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
type: Opaque
data:
  GRAYLOG_PASSWORD_SECRET: {{ $graylogPasswordSecret | b64enc | quote }}
  GRAYLOG_ROOT_PASSWORD_SHA2: {{ $graylogAdminPassword | sha256sum | b64enc | quote }}
  GRAYLOG_ADMIN_PASSWORD: {{ $graylogAdminPassword | b64enc | quote }}
{{ end }}

Any advice pls!

[pastukhov]

Solve this quiz with pre-install hook that check if secret exist
https://gist.github.com/pastukhov/9e894a71e5793cc56a5096a3dc199f5e

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[dtshepherd]

/remove stale

[chrisaige]

Any progress?

[squillace]

I'm curious, @bacongobbler, whether any of the work in Helm 3 will resolve or at least provide a direct workaround to this. thots?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[manics]

This issue is still relevant

[desaintmartin]

There has been some discussion here : helm/charts#10400

[hedayat]

My opinion: The problem is NOT only for randomly generated passwords in helm charts. Even for not random passwords, you need to always pass proper --set params on helm upgrades for passwords, which is very inconvenient and you should always remember the passwords of a specific release. This is also a problem for automatic upgrade tools: they should store passwords somewhere so that they can reuse them for upgrades, while they are already stored by k8s. So, it created redundancy, and also they should secure the passwords they store.

[bacongobbler]

I'm curious, @bacongobbler, whether any of the work in Helm 3 will resolve or at least provide a direct workaround to this. thots?

Hey @squillace, this is just an issue that nobody has documented a working convention in chart development. Helm 3 won't solve this issue necessarily.

Has anyone tried the suggestions made here? If so, does that solve the issue?

[michaelfig]

@bacongobbler I have a working, backward-compatible convention that relies on Helm PR helm/charts#5290

As per the documentation in the PR, you can just do the following to an existing chart, and it will just work, even for people who upgrade an existing installation:

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "helm-random-secret.fullname" . }}
{{- if not .Values.somePassword }}
  annotations:
    "helm.sh/resource-policy": no-upgrade-existing
{{- end }}
  labels:
    app: {{ template "helm-random-secret.name" . }}
    chart: {{ template "helm-random-secret.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
data:
  {{- if .Values.somePassword }}
  some-password: {{ .Values.somePassword | b64enc | quote }}
  {{- else }}
  some-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}

The semantics are:

# Use a specific password.
# If omitted, will use existing password otherwise generate a new random one.
# somePassword: "my Pas$w0rd"

Thoughts?

[bacongobbler]

Dropped a comment in that PR. Let's continue the discussion on that particular feature over there.

[tahmmee]

This might be a useful workaround. With helm 3.1 there is a lookup function,
which means it's possible to skip upgrading of a secret/configmap if it already exists

{{- if not (lookup "v1" "Secret" .Release.Namespace "my-secret") }}
---
apiVersion: v1
kind: Secret

edit:
This appears to be working, except I had to reuse a specific attribute of the secret rather than entire resource because helm will delete it if the template is empty...

{{- define "cluster.password" -}}

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "my-secret") -}}
{{- if $secret -}}
{{/* 
   Reusing current password since secret exists
*/}}
{{-  $secret.data.password -}}
{{- else -}}
{{/* 
    Generate new password
*/}}
{{- (randAlpha 6) | b64enc | quote -}}
{{- end -}}

Then elsewhere in secret

---
apiVersion: v1
kind: Secret
data:
    password: {{ template "cluster.password"  .  }}

[herrberk]

Here is an alternative (requires Helm v3) to tahmmee's solution, in which the generated values are stored in template variables and if the resource already exists, the existing values are used:

{{- $mongoRoot := (randAlpha 16) | b64enc | quote }}
{{- $mongoReplicaKey := (randAlpha 64) | b64enc | quote }}

{{- $secret := (lookup "v1" "Secret" .Release.Namespace "mongo-keys") }}
{{- if $secret }}
{{- $mongoRoot = index $secret.data "mongodb-root-password" }}
{{- $mongoReplicaKey = index $secret.data "mongodb-replica-set-key" }}
{{- end -}}

apiVersion: v1
kind: Secret
metadata:
  name: mongo-keys
  namespace: {{ .Release.Namespace }}
type: Opaque
data:
  mongodb-root-password: {{ $mongoRoot}}
  mongodb-replica-set-key: {{ $mongoReplicaKey }}

[movergan]

Please let's have it fixed. It's ok to use workaround for in-house charts, not for 3rd party dependency though.

[mjgallag]

@herrberk Any ideas for rendering $mongoRoot in NOTES.txt on install? Thanks.

[aure-olli]

Hi

I've rewritten kubernetes replicator and added some annotations to deal with this kind of problems: https://github.com/olli-ai/k8s-replicator#use-random-password-generated-by-an-helm-chart

Now can generate a random password with helm, and replicate it only once to another secret thus it won't be change by helm in the future.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: admin-password-source
  annotations:
    k8s-replicator/replicate-to: "admin-password"
    k8s-replicator/replicate-once: "true"
stringData:
  password: {{ randAlphaNum 64 | quote }}

Hope it will help people.

[monotek]

From helm 3.1.0 you should be able to use the lookup function to read a previously created random.password from kubernetes: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/

[llech]

From helm 3.1.0 you should be able to use the lookup function to read a previously created random.password from kubernetes: https://helm.sh/docs/chart_template_guide/functions_and_pipelines/

How can it work, if the docs state, that:

Keep in mind that Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return an empty list (i.e. dict) in such a case.

so exaclty in the cases where that values would be used?

[jglick]

Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run

helm install --dry-run does contact the API server.

If you use this trick from helm template, the chart will not see an existing secret, so it will simply generate a new one each time.

[dudicoco]

@jglick this would still show a diff when using the helm-diff plugin, so still problematic :(

[rgembalik]

Wouldn't a new value for helm.sh/resource-policy be a decent solution? E.g. update to the annotation functionality so that:

1. helm.sh/resource-policy can accept multiple values (comma-separated, trimmed by helm upon reading so it can be a nice list)
2. helm.sh/resource-policy can accept no-update value which will stop updates to once created resource.

This will allow you to specify the following helm.sh/resource-policy values:

• empty - no special action is taken against the resource, it can be modified and changed freely
• keep - resource is not removed upon uninstall
• no-update - resource will not be updated upon changes (unless the change explicitly removes this resource policy, this is for nice to have)
• keep, no-update, no-update, keep - do not modify or remove the resource once it's created (again, unless update changes this annotation if we feel fancy)

[iTaybb]

As a solution, I've created the following template:

{{/*
Returns a secret if it already in Kubernetes, otherwise it creates
it randomly.
*/}}
{{- define "getOrGeneratePass" }}
{{- $len := (default 16 .Length) | int -}}
{{- $obj := (lookup "v1" .Kind .Namespace .Name).data -}}
{{- if $obj }}
{{- index $obj .Key -}}
{{- else if (eq (lower .Kind) "secret") -}}
{{- randAlphaNum $len | b64enc -}}
{{- else -}}
{{- randAlphaNum $len -}}
{{- end -}}
{{- end }}

Then you can simply configure secrets like:

---
apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  PASSWORD: "{{ include "getOrGeneratePass" (dict "Namespace" .Release.Namespace "Kind" "Secret" "Name" "my-secret" "Key" "PASSWORD") }}"

[mrtndwrd]

That works great @iTaybb ! I have a suggestion to improve it a bit so you can still change the contents of the secret in the future:

{{- define "getOrGeneratePass" }}
{{- $len := (default 16 .Length) | int -}}
{{- $obj := (lookup "v1" .Kind .Namespace .Name).data -}}
{{- $val := (index $obj .Key) -}}
{{- if $val }}
{{- $val -}}
{{- else if (eq (lower .Kind) "secret") -}}
{{- randAlphaNum $len | b64enc -}}
{{- else -}}
{{- randAlphaNum $len -}}
{{- end -}}
{{- end }}

By checking if $obj .Key has a true-ish value, you can add data to your secret and the new data will be generated while keeping the old data in tact.

I know this wouldn't work if you have empty data in your secrets, but in my case that's not a problem.

[saharhagbi]

this was my solution for this problem.
first, I override existingSecret parameter in postgres sub-chart and thus, instruct postgres to use my own secret and not the auto-generated subchart secret.

postgresql:
  existingSecret: "some name"

then, I created my own secret as the following:

apiVersion: v1
kind: Secret
metadata:
  name: "some-name"
type: Opaque
data:
{{- if .Release.IsUpgrade }}
  postgresql-password:  {{ index (lookup "v1" "Secret" .Release.Namespace .Values.postgresql.existingSecret).data "postgresql-password" }}
{{ else }} # install operation
  postgresql-password: {{ randAlphaNum 20 | b64enc }} 
{{ end }}

EDIT:

I had a requirement to give the existingSecret parameter a templating name - {{ .Release.Name }} + "-postgresql".
So, It was a little-bit difficult becasue values.yaml forbid using templating, but after some playing I managed to do that with the following assign:

postgresql:
  existingSecret: "{{ .Release.Name }} -postgresql"

And it works! It seems to me that the string value - "{{ .Release.Name }} -postgresql" - parsed as if we had used tpl function on it. Can it be ?

[sanzenwin]

@saharhagbi , thanks, it works.

[varungupta19]

Same issue here. I resolved it by rolling back to previous release using helm and getting the old password out. Then i upgraded my helm kong release again with this password. And this worked. I used kong helm chart from bitnami and here is the solution mentioned in there articles https://docs.bitnami.com/general/how-to/troubleshoot-helm-chart-issues/

[needleshaped]

Since 1.21 Kubernetes provides an option to set individual Secrets and ConfigMaps as immutable: https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable.

Can it be a solution?

[llech]

Since 1.21 Kubernetes provides an option to set individual Secrets and ConfigMaps as immutable: https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable.

Can it be a solution?

What will happen if Helm will try to delete or modify such immutable secret?

If the request will be gently ignored, it would be the solution.

[saharhagbi]

Since 1.21 Kubernetes provides an option to set individual Secrets and ConfigMaps as immutable: https://kubernetes.io/docs/concepts/configuration/secret/#secret-immutable.

Can it be a solution?

Did you check it locally?
If it works, PostgreSQL chart needs to be updated first

[eoneoff]

My workaround is to combine lookup and resource-policy/keep

{{- if not (lookup "v1" "Secret" .Release.Namespace "mysecret") }}
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
  annotations:
    "helm.sh/resource-policy": "keep"
type: Opaque
stringData:
  password: {{ randAlphaNum 24 }}
{{- end }}

Now on first deployment the secret is created. On upgrade the secret is not charted and is marked to be deleted, but the resource policy prevents it.

[jglick]

@eoneoff I think the idiom shown in #1259 (comment) is preferable in general, as it ensures that other changes to the Secret resource defined by the chart (say, non-Helm-related annotations) are honored when upgrading.

[antevens]

How about a pattern similar to this where a password definition in _helpers.tpl can be referenced multiple times but yields the same random password in an upgrade safe way?

`{{/*
Get current/future passwords in upgrade safe way
*/}}
{{- define "myChart.randomPassword" -}}
{{- $secret := lookup "v1" "Secret" .Release.Namespace "my-secret" }}
{{- if $secret -}}
{{- get $secret.data "password" | b64dec -}}
{{- else }}
{{- get ( $randAlphaNum16 := set . "randAlphaNum16" ( default ( randAlphaNum 16 ) ( get . "randAlphaNum16" ))) "randAlphaNum16" -}}
{{- end }}
{{- end }}

---

apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: my-secret
data:
password: {{ include "myChart.randomPassword" . | b64enc | quote }}

---

apiVersion: v1
kind: ConfigMap
data:
config: |-
serverpassword: {{ include "myChart.randomPassword" . }} `

[peterwwillis]

I have cobbled together a solution from a couple different solutions. My use case:

• Generate a random password for a Postgres install at helm install time
• Populate a K8s secret with the random password
• Reuse the same randomly-generated password in the same Helm run
• Reuse the K8s secret at helm upgrade time (bonus points: overwrite the secret value outside of Helm and Helm won't care)

First you create a function to generate static passwords. This has the neat property of creating a different random password per Release.Name, though of course you could get more granular with it. Props to https://stackoverflow.com/a/75723185/3760330 for the random password generator function:

{{- define "generate_static_password" -}}
{{- /* Create "tmp_vars" dict inside ".Release" to store various stuff. */ -}}
{{- if not (index .Release "tmp_vars") -}}
{{-   $_ := set .Release "tmp_vars" dict -}}
{{- end -}}
{{- /* Some random ID of this password, in case there will be other random values alongside this instance. */ -}}
{{- $key := printf "%s_%s" .Release.Name "password" -}}
{{- /* If $key does not yet exist in .Release.tmp_vars, then... */ -}}
{{- if not (index .Release.tmp_vars $key) -}}
{{- /* ... store random password under the $key */ -}}
{{-   $_ := set .Release.tmp_vars $key (randAlphaNum 20) -}}
{{- end -}}
{{- /* Retrieve previously generated value. */ -}}
{{- index .Release.tmp_vars $key -}}
{{- end -}}

Next you create a function that will try to lookup the value in the K8s secret on upgrade. If it's not an upgrade (assume it's install) it requires two values, which are the name of the secret to create, and the key to store the password as in the secret. Note that lookup() returns an empty state on template or dry-run, so this solution creates a default dict to avoid errors.

{{- define "random_pw_reusable" -}}
  {{- if .Release.IsUpgrade -}}
    {{- $data := default dict (lookup "v1" "Secret" .Release.Namespace .Values.random_pw_secret_name).data -}}
    {{- if $data -}}
      {{- index $data .Values.random_pw_secret_key | b64dec -}}
    {{- end -}}
  {{- else -}}
    {{- if and (required "You must pass .Values.random_pw_secret_name (the name of a secret to retrieve password from on upgrade)" .Values.random_pw_secret_name) (required "You must pass .Values.random_pw_secret_key (the name of the key in the secret to retrieve password from on upgrade)" .Values.random_pw_secret_key) -}}
      {{- (include "generate_static_password" .) -}}
    {{- end -}}
  {{- end -}}
{{- end -}}

Finally you create your secret:

---
apiVersion: v1
kind: Secret

metadata:
  name: "{{ .Values.random_pw_secret_name }}"
  namespace: "{{ .Release.Namespace }}"

type: Opaque

{{- if .Values.secretData -}}
data:
  {{- range $k, $v := .Values.secretData }}
    "{{ $k }}": "{{ tpl $v $ | b64enc }}"
  {{- end }}
{{- end -}}

values.yaml for the secret:

random_pw_secret_name: db-credentials
random_pw_secret_key:  DB_PASSWORD
secretData:
  DB_HOST: db
  DB_NAME: mydatabase
  DB_USER: postgres
  DB_PASSWORD: '{{- include "random_pw_reusable" . -}}'
  DB_CONNECTION_STRING: 'postgresql://postgres:{{- include "random_pw_reusable" . -}}@db:5432/mydatabase'

As you can see, a new secret will be created with a set of KEY=VALUE pairs. Both DB_CONNECTION_STRING and DB_PASSWORD call the random_pw_reusable function, and both of them will receive the same randomly-generated string at install time, or the value of the DB_PASSWORD at upgrade time.

In this way you can deploy Postgres at install time with a random password, upgrade it and retain the same password, and rotate the password at any time (outside of Helm) yet still retain that rotated password. Resources are managed as normal so a delete/uninstall will work fine. I suppose the only problem is if the secret gets deleted, this won't create a new random password...