rejoining the cluster failing

[pasztorl]

I created a 3 node rabbitmq cluster with the operator.
For failover testing i deleted the PVC and the pod rabbitmq-server-1, so the pod restarted with a new, empty PV.
Then rabbitmq-server-1 can't join the cluster but created an another one. Here is the log:

2025-09-20 15:43:17.356782+00:00 [notice] <0.18879.0> Feature flags: nodes `rabbit@rabbitmq-server-1.rabbitmq-nodes.dev` and `rabbit@rabbitmq-server-0.rabbitmq-nodes.dev` are compatible
2025-09-20 15:43:17.357647+00:00 [error] <0.18879.0> Mnesia: node 'rabbit@rabbitmq-server-0.rabbitmq-nodes.dev' thinks it's clustered with node 'rabbit@rabbitmq-server-1.rabbitmq-nodes.dev', but 'rabbit@rabbitmq-server-1.rabbitmq-nodes.dev' disagrees
2025-09-20 15:43:17.357694+00:00 [warning] <0.18879.0> Peer discovery: could not auto-cluster with node 'rabbit@rabbitmq-server-0.rabbitmq-nodes.dev': {error,{inconsistent_cluster,"Mnesia: node 'rabbit@rabbitmq-server-0.rabbitmq-nodes.dev' thinks it's clustered with node 'rabbit@rabbitmq-server-1.rabbitmq-nodes.dev', but 'rabbit@rabbitmq-server-1.rabbitmq-nodes.dev' disagrees"}}
2025-09-20 15:43:17.357797+00:00 [debug] <0.18879.0> Peer discovery: lock released
2025-09-20 15:43:17.357824+00:00 [info] <0.18879.0> Peer discovery: will try to unlock with peer discovery backend rabbit_peer_discovery_k8s
2025-09-20 15:43:17.358007+00:00 [error] <0.18879.0> Peer discovery: could not discover and join another node; proceeding as a standalone node

Version and environment information

• RabbitMQ: 4.0.7
• RabbitMQ Cluster Operator: 2.10.0
• Kubernetes: 1.32.2

[pasztorl]

Update: if i run rabbitmqctl forget_cluster_node rabbit@rabbitmq-server-1.rabbitmq-nodes.dev on a healty node this node can rejoin.
This is normal or the operator manages the rejoin process?
I checked the operator logs, and the operator does not logged anything about the node loss/rejoin.

[pasztorl]

Update2: previously i hade quorum queues which using all 3 pod as members. After forgetting/reconnecting rabbitmq-server-1 quorum queues does not include rabbitmq-server-1 anymore so i had to run rabbitmq-queues add_member... on each quorum queue.
These queues are created by hand, not by the Queue CR. If i create these queues with the CR the operator will check and run add_member if needed?

[mkuratczyk]

RabbitMQ 4.1 should behave the way you expect: rabbitmq/rabbitmq-server#13643

[mkuratczyk]

Well, with one caveat: 4.1 also changed how k8s peer discover works
and currently if you lose server-0 (the pod with index 0), it won't rejoin
the cluster automatically. Other nodes would #1957

[pasztorl]

so the operator currently not handling the failover? using 4.1 + let the operator automatically adjust the peer discover by selecting a healthy pod. Would it be solve the index-0 problem?

[mkuratczyk]

I'm not sure what you mean by "handling failover" but the Operator doesn't do anything special for nodes that were wiped out.
There's currently no automated solution for the index-0 problem - if you lose the index 0 pod, you should do forget_cluster_node/join_cluster manually.

[pasztorl]

i mean trying to recover cluster state if "handleable" error occured like this. for example mariadb operator can (or at least try) recover a galera cluster if not healty.
so currently the operator can do the first deployment and modify config and statefulset parameters, but not managing cluster state, right?

[mkuratczyk]

Yes. Most of the functionality is around the deployment and setting up things correctly. However, the operator pattern allows us to do additional things. The index-0 issue is something that we may consider automatically resolving in the future for example.

[pasztorl]

sounds awesome! thank you very much!