PostgreSQL with existing secret broken after update 4.0.0

[archer-321]

Describe your Issue

Setting postgresql.global.postgresql.auth.existingSecret and the secret keys in postgresql.global.postgresql.auth.secretKeys doesn't work after update 4.0.0.

Trying to deploy Nextcloud with the configuration below returns PASSWORDS ERROR: The secret "postgres-creds" does not contain the key "password" instead.

Steps to reproduce:

1. Create values.yml and secret.yml with the below content.
2. Create a new namespace: kubectl create namespace nextcloud
3. Create the secret: kubectl apply -f secret.yml
4. Try to deploy any version >=4.0.0: helm install -n nextcloud -f values.yml albacore nextcloud/nextcloud --version=4.0.0

Version 3.5.22 works as expected.

Logs and Errors

> helm install -n nextcloud -f values.yml albacore nextcloud/nextcloud --version=4.0.0

Error: INSTALLATION FAILED: execution error at (nextcloud/charts/postgresql/templates/secrets.yaml:23:16):
PASSWORDS ERROR: The secret "postgres-creds" does not contain the key "password"

Describe your Environment

• Kubernetes distribution:
  Tested with a fresh multi-node minikube cluster and a production cluster maintained with kubeadm

• Helm Version (or App that manages helm): 3.12.0

• Helm Chart Version: 4.0.0 and above

• values.yml:

ingress:
  enabled: true
externalDatabase:
  enabled: true
  type: postgresql
  host: albacore-postgresql-0:5432
  user: nextcloud
  database: nextcloud
  existingSecret:
    # Both, `true` and `false` for this value don't work.
    enabled: false
    secretName: postgres-creds
    passwordKey: user
internalDatabase:
  enabled: false
postgresql:
  enabled: true
  global:
    postgresql:
      auth:
        existingSecret: postgres-creds
        secretKeys:
          adminPasswordKey: admin
          userPasswordKey: user
          replicationPasswordKey: replication

• secret.yml (for testing purposes):

apiVersion: v1
data:
  admin: Zm9v
  replication: YmF6
  user: YmF6
kind: Secret
metadata:
  name: postgres-creds
  namespace: nextcloud
type: Opaque

[jessebot]

Thanks for reporting this! Can you please let me know if you're also setting your externalDatabase settings? This is also required when using anything that isn't the internalDatabase. Example using some of the values you mentioned in your post:

##
## External database configuration
##
externalDatabase:
  enabled: true

  ## Supported database engines: mysql or postgresql
  type: postgresql

  ## Database host
  host: localhost:5432

  ## Database user - if it's not nextcloud, please change this to your preferred database user
  user: nextcloud

  ## Database name
  database: nextcloud

  ## Use a existing secret
  existingSecret:
    enabled: false
    secretName: postgres-creds
    # usernameKey: db-username
    passwordKey: user
    # hostKey: db-hostname-or-ip
    # databaseKey: db-name

[provokateurin]

I'm using Postgres with existing secret and it still works for me 🤔 Here is the relevant part of my values.yaml:

externalDatabase:
  enabled: true
  type: postgresql
  host: "acid-nextcloud-cluster:5432"
  existingSecret:
    enabled: true
    secretName: nextcloud.acid-nextcloud-cluster.credentials.postgresql.acid.zalan.do
    usernameKey: username
    passwordKey: password

[archer-321]

Can you please let me know if you're also setting your externalDatabase settings? This is also required when using anything that isn't the internalDatabase.

Thank you for the response!

I indeed should have included this section in my testing configuration, but it's filled out in my production configuration.

I included your section in my example, and I can still reproduce the problem on a fresh minikube cluster. Here's my new values.yml file (I'll also update the issue description):

ingress:
  enabled: true
externalDatabase:
  enabled: true
  type: postgresql
  host: albacore-postgresql-0:5432
  user: nextcloud
  database: nextcloud
  existingSecret:
    # Both, `true` and `false` for this value don't work.
    enabled: false
    secretName: postgres-creds
    passwordKey: user
internalDatabase:
  enabled: false
postgresql:
  enabled: true
  global:
    postgresql:
      auth:
        existingSecret: postgres-creds
        secretKeys:
          adminPasswordKey: admin
          userPasswordKey: user
          replicationPasswordKey: replication

[archer-321]

I'm using Postgres with existing secret and it still works for me 🤔 Here is the relevant part of my values.yaml:

Does this also work with keys not named password? From the error message I listed in the issue description, it seems like the key name password might be hardcoded somewhere 🤔

[provokateurin]

helm/charts/nextcloud/templates/_helpers.tpl

Line 108 in 0b6bd84

key: {{ .Values.externalDatabase.existingSecret.passwordKey | default "db-password" }}

looks like it is not hardcoded

[jessebot]

As Kate mentioned, it doesn't look like it's hardcoded, but I went and checked my own values and it looks like I don't use the word password for my key at all here:

# note this is only indented like this because it's deployed via argo, 
# but normally this would not be indented
            internalDatabase:
              enabled: false
            externalDatabase:
              enabled: true
              type: postgresql
              host: localhost:5432
              user: nextcloud
              database: nextcloud
              existingSecret:
                enabled: true
                secretName: nextcloud-pgsql-credentials
                usernameKey: nextcloudUsername
                passwordKey: nextcloudPassword
            postgresql:
              enabled: true
              # we can't use fullnameOverride till https://github.com/nextcloud/helm/pull/442 is merged
              # fullnameOverride: nextcloud-postgresql
              global:
                postgresql:
                  auth:
                    username: nextcloud
                    database: nextcloud
                    existingSecret: nextcloud-pgsql-credentials
                    secretKeys:
                      userPasswordKey: nextcloudPassword
                      adminPasswordKey: postgresPassword

Out of curiosity, since this seems to be an issue with the bitnami postgresql sub chart, can you try and use the bitnami helm chart on it's own and see if you can deploy it? I ask because the error you posted shows:

nextcloud/charts/postgresql/templates/secrets.yaml:23:16

And that points to this file:
https://github.com/bitnami/charts/blob/396cec8e6bd754cffd6b4114c33c2d5f5199b717/bitnami/postgresql/templates/secrets.yaml#L23C44-L23C44

Any parameter passed under the postgresql parameter in values.yaml gets passed to that subchart, which in this case is bitnami/postgresql.

[archer-321]

TL;DR - Working Nextcloud values.yaml

externalDatabase:
  enabled: true
  type: postgresql
  host: albacore-postgresql-0:5432
  user: nextcloud
  database: nextcloud
  existingSecret:
    enabled: false
    secretName: postgres-creds
    passwordKey: user
internalDatabase:
  enabled: false
postgresql:
  enabled: true
  # ========== This section is important ==========
  auth:
    existingSecret: postgres-creds
    secretKeys:
      adminPasswordKey: admin
      userPasswordKey: user
      replicationPasswordKey: replication
  # ========== End of important section ==========
  global:
    postgresql:
      auth:
        existingSecret: postgres-creds
        secretKeys:
          adminPasswordKey: admin
          userPasswordKey: user
          replicationPasswordKey: replication

---

can you try and use the bitnami helm chart on it's own and see if you can deploy it?

After copying the values set by Nextcloud (most notably, global.postgresql.auth.username), I could indeed reproduce this problem with the upstream Bitnami chart.

As far as I can tell, this problem is caused by this block setting the key to the value of auth.secretKeys.userPasswordKey for every deployment that doesn't use the postgres user.

With the chart's default values, this evaluates to password.

A simple fix is copying postgresql.global.postgresql.auth.secretKeys to postgresql.auth.secretKeys in values.yml for Nextcloud.

---

Bitnami's documentation states the following for global.postgresql.auth.secretKeys.userPasswordKey.

Name of key in existing secret to use for PostgreSQL credentials (overrides auth.secretKeys.userPasswordKey).

At this point, I'm unsure whether the documentation is wrong, the upstream Bitnami chart is implemented incorrectly, or whether Nextcloud's default values should recommend adding a postgresql.auth section if users decide to use custom keys for their credentials secret 😕

[jessebot]

First of all, great detective work! I know this was frustrating, so thanks for sticking with it :)

After copying the values set by Nextcloud (most notably, global.postgresql.auth.username), I could indeed reproduce this problem with the upstream Bitnami chart.

Then this should actually be resolved in the bitnami chart. Can you please open an issue there? In the meantime, anyone having issues with nextcloud's community helm chart, this chart, can search the issues here, but it should definitely be fixed upstream.

[archer-321]

Can you please open an issue there?

I agree that this is likely unexpected behaviour (and thus a bug) in Bitnami's chart. I created an issue after double-checking that this has yet to be reported. You can find the bug report here: bitnami/charts#19202.

[archer-321]

With bitnami/charts#19281 merged, this seems to be fixed upstream in bitnami/postgresql 12.11.2 🚀

[jessebot]

Awesome work, @archer-321 ! :) And thanks for keeping us updated 💙

[jessebot]

Oh, got too excited there 😅

We are happy to accept a PR to get PostgreSQL upgraded in the sub-chart here as well :) You just need to tick up both the subchart version and the nextcloud helm chart version, and then run helm dep update in the helm chart directory and push up all the files.