[self-hosted] Couldn't start RabbitMQ in background

[jgallucci32]

Bug description

When redeploying the messagebus pod, it fails to redeploy with the following error message (with BITNAMI_DEBUG=true)

rabbitmq 16:45:42.94 
rabbitmq 16:45:42.94 Welcome to the Bitnami rabbitmq container
rabbitmq 16:45:42.95 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-rabbitmq
rabbitmq 16:45:42.95 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-rabbitmq/issues
rabbitmq 16:45:42.95 
rabbitmq 16:45:42.95 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 16:45:42.97 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 16:45:42.99 INFO  ==> Initializing RabbitMQ...
rabbitmq 16:45:43.00 DEBUG ==> Creating environment file...
rabbitmq 16:45:43.01 DEBUG ==> Creating enabled_plugins file...
rabbitmq 16:45:43.02 DEBUG ==> Creating Erlang cookie...
rabbitmq 16:45:43.02 DEBUG ==> Ensuring expected directories/files exist...
rabbitmq 16:45:43.04 INFO  ==> Starting RabbitMQ in background...
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmq-server: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmq-server: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAME: not found
/opt/bitnami/rabbitmq/sbin/rabbitmqctl: 3: /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq-env.conf: MY_POD_NAMESPACE: not found
rabbitmq 16:47:23.10 ERROR ==> Couldn't start RabbitMQ in background.

Steps to reproduce

1. Deploy Gitpod 0.10.0 to a self-hosted Kubernetes cluster using Helm chart
2. Redeploy the messagebus pod (alternatively scale down to 0 then to 1)

Expected behavior

The messagebus pod restarts successfully

Example repository

n/a

Anything else?

This is a known issue with Bitnami documented in rancher/rancher#18312

Environment

Gitpod 0.10.0
Rancher Server 2.5.7
RKE 1.19.13
Docker CE 20.10.7
Red Hat Enterprise Linux 8.3

[jgallucci32]

Currently the best workaround I can find for now is to set the messagebus StatefulSet to 0 and then to run helm upgrade on the chart.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jgallucci32]

In the next week or so I should have a Rancher v2.5.9 cluster to test and verify this was an issue with Rancher and not a problem with the chart.

[sagor999]

@jgallucci32 we have deprecated helm chart since then and instead are using installer. Would you be able to try it with the new installer? And see if this is still an issue?

[corneliusludmann]

Closing this since it seems stale. Please let me know if this is still relevant.

[jakoberpf]

@corneliusludmann I just tried installing gitpods with the installer and seeing this issue. I am not so familiar woth kots, so I don't know how to add overlays to debug this further. Maybe you got a hint for me?