[bitnami/rabbitmq] invalid credentials issue when extraConfiguration used #4635

bmailhe · 2020-12-07T16:35:46Z

Description

When using extraSecrets to loadDefinition for extraConfiguration, the user and password defined (in another secret) is made invalid. This occurs when adding this configuration:

  extraSecrets:
    rabbit-policies:
      rabbit_policies.json: |
        {
          "vhosts": [
            {
              "name": "/"
            }
          ],
          "policies": [
            {
              "name": "ha-mirror-policy",
              "pattern": ".*",
              "vhost": "/",
              "definition": {
                "ha-mode": "all"
              }
            }
          ]
        }
  loadDefinition:
    enabled: true
    existingSecret: rabbit-policies
  extraConfiguration: |
    load_definitions = /app/rabbit_policies.json

The message in the log is

rabbit-0 rabbitmq 2020-12-07 10:07:17.593 [error] <0.2621.0> Error on AMQP connection <0.2621.0> (100.120.128.11:51874 -> 100.105.0.33:5672, state: starting):
rabbit-0 rabbitmq PLAIN login refused: user 'myuser' - invalid credentials
...
rabbit-0 rabbitmq HTTP access denied: user 'myuser' - invalid credentials

ℹ️ Deploying the chart without the extraSecrets to loadDefinition for extraConfiguration defined above work perfectly.

The extra conf is well loaded in the container :

$ kubectl exec -it rabbit-0 -- cat /app/rabbit_policies.json
{
  "vhosts": [
    {
      "name": "/"
    }
  ],
  "policies": [
    {
      "name": "ha-mirror-policy",
      "pattern": ".*",
      "vhost": "/",
      "definition": {
        "ha-mode": "all"
      }
    }
  ]
}

I can also see the env for

I have no name!@rabbit-0:/$ env |grep -e RABBITMQ_USERNAME -e RABBITMQ_PASS -e RABBITMQ_ERL_COOKIE
RABBITMQ_PASSWORD=mypass
RABBITMQ_USERNAME=myuser
RABBITMQ_ERL_COOKIE=JFHDDPMHz37tfNAkL4kEOmPqjkOEslc6

The secret defining the RABBITMQ_PASSWORD :

➜ kgsec rabbit -o yaml
apiVersion: v1
data:
  rabbitmq-erlang-cookie: SkZIRERQTUh6Mzd0Zk5Ba0w0a0VPbVBxamtPRXNsYzY=
  rabbitmq-password: bXlwYXNzCg==
kind: Secret
metadata:
  name: rabbit
  namespace: test1100
type: Opaque

the templating for the password is

  {{- if .Values.auth.password }}
  rabbitmq-password: {{ .Values.auth.password | b64enc | quote }}
  {{- else }}
  rabbitmq-password: {{ randAlphaNum 10 | b64enc | quote }}
  {{- end }}

Steps to reproduce the issue:

Deploy the chart without the extraSecrets, loadDefinition and extraConfiguration
Log on the interface with user and password work correctly ✅
Deploy the chart with the extraSecrets, loadDefinition and extraConfiguration (see above). I also tested with an another extra configuration defining only the vhostsand not the policies and it also fails.
Log on the interface with user and password does not work anymore ❌

Another test
Steps:

Deploy the chart without the extraSecrets, loadDefinition and extraConfiguration
Log on the interface with user and password work correctly ✅
Scale down sts to replicas=0
Add a secret with the extra conf, edit the sts to add the volumeMount and volume for this new secret and edit the configmap to enable the load_definitions = /app/rabbit_policies.json
Scale up sts to 1
Now it works we are able to log in and the policy is well created and loaded.

Versions used
Chart: rabbitmq bitnami v8.0.4
Image tag: 3.8.9-debian-10-r37

The text was updated successfully, but these errors were encountered:

juan131 · 2020-12-08T13:09:16Z

Hi @bmailhe

I think this could be the reason why the user/password are not configured:

https://github.com/bitnami/bitnami-docker-rabbitmq/blob/master/3.8/debian-10/rootfs/opt/bitnami/scripts/librabbitmq.sh#L530

As you can see, it skips invoking the rabbitmq_change_password function when RABBITMQ_LOAD_DEFINITIONS is "yes". It will also explain why it works when you scale to 0, since that block of code is skipped when persistent data is detected:

https://github.com/bitnami/bitnami-docker-rabbitmq/blob/master/3.8/debian-10/rootfs/opt/bitnami/scripts/librabbitmq.sh#L520

The thing is.. I'm unable to find where we're setting that env. var to "yes" anywhere in the chart. Could you please confirm that, when you access the rabbit-0 pod that env. var is not set?

bmailhe · 2020-12-08T17:11:26Z

Hi @juan131

Could you please confirm that, when you access the rabbit-0 pod that env. var is not set?

No, empty return for env|grep RABBITMQ_LOAD_DEFINITIONS

bmailhe · 2020-12-08T18:33:55Z

In the logs I have Persisted data detected. Restoring... (on a fresh install) which means that RABBITMQ_DATA_DIR (librabbitmq.sh#L520) is not empty when adding the loadDefinition and so the rabbitmq_change_password is never executed in this case.

rabbit-0 rabbitmq  15:08:10.81 Welcome to the Bitnami rabbitmq container
rabbit-0 rabbitmq  15:08:10.82 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbit-0 rabbitmq  15:08:10.83 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbit-0 rabbitmq  15:08:10.84
rabbit-0 rabbitmq  15:08:10.84 INFO  ==> ** Starting RabbitMQ setup **
rabbit-0 rabbitmq  15:08:10.88 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbit-0 rabbitmq  15:08:10.91 INFO  ==> Initializing RabbitMQ...
rabbit-0 rabbitmq  15:08:10.96 INFO  ==> Persisted data detected. Restoring...
rabbit-0 rabbitmq
rabbit-0 rabbitmq  15:08:10.97 INFO  ==> ** RabbitMQ setup finished! **
rabbit-0 rabbitmq  15:08:11.00 INFO  ==> ** Starting RabbitMQ **

ghost · 2020-12-09T09:19:25Z

To keep the directory clear I changed /app to /secret here

charts/bitnami/rabbitmq/templates/statefulset.yaml

Line 279 in 6257aa5

mountPath: /app

Which it now appears like it might have set the password the logs still don't say that it reset the user/pass ....

I have set auth.username and auth.password in my values.yaml

{edit) tried on a new namespace and it tried to reset the user/pass but failed

09:23:03.29 DEBUG ==> Changing password for user 'admin'...
Changing password for user "admin" ...
2020-12-09 09:23:07.043 [warning] <0.1394.0> Failed to change password for user 'admin': the user does not exist
Error:
User "admin" does not exist
09:23:07.07 ERROR ==> Couldn't change password for user 'admin'.
09:23:07.08 INFO ==> Stopping RabbitMQ...

found this in the logs further up
2020-12-09 09:22:56.362 [info] <0.302.0> Will not seed default virtual host and user: have definitions to load...

bmailhe · 2020-12-09T10:16:25Z

I managed it to work by adding the user configuration in the load_definition :

{
  "users": [
    {
      "name": "user",
      "password": "pass",
      "tags": "administrator"
    }
  ],
 "permissions": [
    {
      "user": "user",
      "vhost": "/",
      "configure": ".*",
      "write": ".*",
      "read": ".*"
    }
  ],
  "vhosts": [
    {
      "name": "/"
    }
  ],
  "policies": [
   {
     "name": "ha-mirror-policy",
     "pattern": ".*",
     "vhost": "/",
     "definition": {
       "ha-mode": "all"
     }
   }
  ]
}

and by removing the .Values.auth settings

ghost · 2020-12-09T10:31:00Z

that lines up with the message I saw about not seeding. So for me the question is how I would have to modify the helm chart I'm currently working on which has this chart as a dependency, where I'm pushing through values from that value file.

juan131 · 2020-12-09T11:30:49Z

Great @bmailhe !! I'm glad you were able to solve by adding the user configuration in the load_definition!

@christopher-russell regarding your issue, if you're using RabbitMQ as a subchart, you can still use the extraSecrets, loadDefinition and extraConfiguration parameters as @bmailhe explained. But, in this case, you'll have to use them in the form: rabbitmq.extraSecrets, rabbitmq.loadDefinition and rabbitmq.extraConfiguration.

ghost · 2020-12-09T11:50:42Z

Thanks @juan131 yes I've tested that but if you set auth.username and auth.password these are ignored and placing them as templated vulues doesn't work as the vulues aren't translated e.g.


{
  "users": [
    {
      "name": {{ .Values.auth.username | quote }},
      "password": {{ .Values.auth.password | quote }},
      "tags": "administrator"
    }
  ],

or 

{
  "users": [
    {
      "name": "{{ .Values.auth.username }}",
      "password": "{{ .Values.auth.password }}",
      "tags": "administrator"
    }
  ],

ghost · 2020-12-09T11:58:21Z

My configuration looks like this at the moment

  auth:
    username: admin
    password: password
  extraSecrets:
    rabbitmq-definitions:
      rabbitmq-definitions.json: |-
        {
          "users": [
            {
              "name": "{{ .Values.auth.username }}",
              "password": "{{ .Values.auth.password }}",
              "tags": "administrator"
            }
          ],
          "vhosts": [
            {
              "name": "/"
            }
          ],
          "policies": [
            {
              "vhost":"/",
              "name":"ha",
              "pattern":"",
              "definition": {
                "ha-mode":"all",
                "ha-sync-mode": "automatic",
                "ha-sync-batch-size":10
              }
            }
          ]
        }
  loadDefinition:
    enabled: true
    existingSecret: rabbitmq-definitions
  extraConfiguration: |
    load_definitions = /app/rabbitmq-definitions.json

juan131 · 2020-12-09T12:00:57Z

If you're using RabbitMQ as a subchart, try changing .Values.auth.XXX with .Values.rabbitmq.auth.XXX

ghost · 2020-12-09T12:23:54Z

unfortunately its not subbing the values in ..

2020-12-09 12:21:00.960 [info] <0.683.0> Applying definitions from file at '/app/rabbitmq-definitions.json'
2020-12-09 12:21:00.960 [info] <0.683.0> Asked to import definitions. Acting user: rmq-internal
2020-12-09 12:21:00.960 [info] <0.683.0> Importing concurrently 1 users...
2020-12-09 12:21:00.973 [info] <0.679.0> Created user '{{ .Values.rabbitmq.auth.username }}'
2020-12-09 12:21:00.980 [info] <0.679.0> Successfully set user tags for user '{{ .Values.rabbitmq.auth.username }}' to [administrator]
2020-12-09 12:21:00.981 [info] <0.683.0> Importing concurrently 1 vhosts...
2020-12-09 12:21:00.981 [info] <0.679.0> Adding vhost '/' without a description

miguelaeh · 2020-12-09T12:54:17Z

Hi @christopher-russell ,
I have tried your configuration parameters and it seems to be working fine in my case:

# Source: rabbitmq/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-definitions
  namespace: macabrera
  labels:
    app.kubernetes.io/name: rabbitmq
    helm.sh/chart: rabbitmq-8.5.0
    app.kubernetes.io/instance: RELEASE-NAME
    app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
  rabbitmq-definitions.json: |-
    {
      "users": [
        {
          "name": "admin",
          "password": "password",
          "tags": "administrator"
        }
      ],
      "vhosts": [
        {
          "name": "/"
        }
      ],
      "policies": [
        {
          "vhost":"/",
          "name":"ha",
          "pattern":"",
          "definition": {
            "ha-mode":"all",
            "ha-sync-mode": "automatic",
            "ha-sync-batch-size":10
          }
        }
      ]
    }

In my case, I am using the chart version 8.5.0. Could you give it a try updating the chart?

ghost · 2020-12-09T21:39:41Z

@miguelaeh di you try that when the rabbitmq chart was a subchart?

I'm starting to wonder if that issue is related to helm/helm#2492

Either way I think extra definitions these should be in an overwrite mode not an or as in defaults or definitions or at least a toggle to specify which you would prefer.

ghost · 2020-12-10T01:34:58Z

I've managed to get a workaround sorted, by creating the secret in the parent chart and then passing it as an existing secret to rabbit

I've attached the config for anyone in the future with the same issue as me .

rabbit-definition.conf goes in a conf directory at chart root level
rabbit-definition.conf.txt

secrets in your templates directory
secrets_rabbitmq.yaml.txt

config below in your values.yaml

ingress:
   enabled: true
   # Path must start with a slash
   path: "/"

 # Enable Queue Mirroring between nodes
 # See https://www.rabbitmq.com/ha.html
 loadDefinition:
   enabled: true
   existingSecret: "{{ .Release.Name }}-rabbitmq-definitions"
 extraConfiguration: |
   load_definitions = /app/rabbitmq-definitions.json

juan131 · 2020-12-10T07:57:18Z

Great @christopher-russell !! I'm glad you were able to workaround it!!!
thanks so much for sharing the solution, I'm pretty sure it'll be very helpful for other users under similar situations.

stale · 2020-12-25T10:11:45Z

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

stale · 2021-02-25T14:54:27Z

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

ghost mentioned this issue Dec 9, 2020

Migrate to Helm v3, Change dependencies from deprecated official helm charts to Bitnami charts StackStorm/stackstorm-k8s#163

Merged

stale bot added the stale 15 days without activity label Dec 25, 2020

stale bot closed this as completed Feb 25, 2021

elsopapa mentioned this issue Sep 6, 2021

Problem with nfs deploy on job job-st2-register-content StackStorm/stackstorm-k8s#238

Closed

Kapildev2018 mentioned this issue Sep 26, 2022

Reinstalling the new custom pack is not getting rendered on the Web UI StackStorm/stackstorm-k8s#315

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bitnami/rabbitmq] invalid credentials issue when extraConfiguration used #4635

[bitnami/rabbitmq] invalid credentials issue when extraConfiguration used #4635

bmailhe commented Dec 7, 2020

juan131 commented Dec 8, 2020 •

edited

bmailhe commented Dec 8, 2020

bmailhe commented Dec 8, 2020

ghost commented Dec 9, 2020 •

edited by ghost

bmailhe commented Dec 9, 2020

ghost commented Dec 9, 2020

juan131 commented Dec 9, 2020 •

edited

ghost commented Dec 9, 2020 •

edited by ghost

ghost commented Dec 9, 2020

juan131 commented Dec 9, 2020

ghost commented Dec 9, 2020

miguelaeh commented Dec 9, 2020

ghost commented Dec 9, 2020

ghost commented Dec 10, 2020 •

edited by ghost

juan131 commented Dec 10, 2020

stale bot commented Dec 25, 2020

stale bot commented Feb 25, 2021

[bitnami/rabbitmq] invalid credentials issue when extraConfiguration used #4635

[bitnami/rabbitmq] invalid credentials issue when extraConfiguration used #4635

Comments

bmailhe commented Dec 7, 2020

juan131 commented Dec 8, 2020 • edited

bmailhe commented Dec 8, 2020

bmailhe commented Dec 8, 2020

ghost commented Dec 9, 2020 • edited by ghost

bmailhe commented Dec 9, 2020

ghost commented Dec 9, 2020

juan131 commented Dec 9, 2020 • edited

ghost commented Dec 9, 2020 • edited by ghost

ghost commented Dec 9, 2020

juan131 commented Dec 9, 2020

ghost commented Dec 9, 2020

miguelaeh commented Dec 9, 2020

ghost commented Dec 9, 2020

ghost commented Dec 10, 2020 • edited by ghost

juan131 commented Dec 10, 2020

stale bot commented Dec 25, 2020

stale bot commented Feb 25, 2021

juan131 commented Dec 8, 2020 •

edited

ghost commented Dec 9, 2020 •

edited by ghost

juan131 commented Dec 9, 2020 •

edited

ghost commented Dec 9, 2020 •

edited by ghost

ghost commented Dec 10, 2020 •

edited by ghost