[FEATURE REQUEST] RabbitMQ needs to be highly available

[toszo]

Is your feature request related to a problem? Please describe.
When RabbitMQ is deployed as pods inside the Kubernetes cluster teh application needs RMQ to be highly available and not lose configuration or data when pods are moved, or VMs restarted, or cluster restored for backup/restore or DR activities.

Describe the solution you'd like
RabbitMQ clustered and replicated.

Describe alternatives you've considered
None

Additional context
Teams request, EP-112
More context from duplicate issue #1485 in comments below.

---

DoD checklist

• Changelog
  • updated
  • not needed
• COMPONENTS.md
  • updated
  • not needed
• Schema
  • updated
  • not needed
• Backport tasks
  • created
  • not needed
• Documentation
  • added
  • updated
  • not needed
• Feature has automated tests
• Automated tests passed (QA pipelines)
  • apply
  • upgrade
  • backup/restore
• Idempotency tested
• All conversations in PR resolved
• Solution meets requirements and is done according to design doc
• Usage compliant with license

[toszo]

May be related and resolved similat to #1485

[mkyc]

@seriva as far as I understand this task it depends on whole #1455 epic. We need to have persistence layer in k8s first. Am I right?

[toszo]

Looks like. Added dependency.

[mkyc]

Added section "More context is provided also in duplicate task #1485" to issue description.

[Harshini097t]

Hello Team, I was looking into this issue and wondering is it still something that is to be addressed? In that case what is expected to be done for it.

[seriva]

#1455 is almost completed which would make this possible. If there is a need for it we can consider it.

[seriva]

Adding comment from #1485 so we can close that one:

Describe the bug
StatefulSet's "volumeMounts" should use "volumeClaimTemplates", like in this Kubernetes doc https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/ or in this example https://gist.github.com/pmint93/cb87c5f46502ce8047a084238cad03e4, to mount "/var/lib/rabbitmq/mnesia" to a PersistentVolume.
At the moment, all that's configured is a volume of type configMap, that injects RMQ configuration into the running container, and that's not enough to avoid losing RMQ configuration and data when the RMQ pod gets moved or the node hosting the RMQ pod gets shut down.

To Reproduce

Create Epiphany with RabbitMQ cluster on top of K8s
Add users, permissions, etc. to RMQ
Shut down Kubernetes nodes that host the RMQ pods, then restart them.
Result: any RMQ configuration done on top of the default one, and all the RMQ data are lost.
Expected behavior
Data on RabbitMQ should be persistent and work between restarts.

Config files
If applicable, add config files to help explain your problem.

OS (please complete the following information):

OS: [e.g. RHEL7]
Additional context
Teams request, EP-118

Suggestion from creator

So in the StatefulSet configuration, in addition to this:
volumeMounts:

mountPath: /etc/rabbitmq
name: config-volume
volumes:

configMap:
defaultMode: 420
items:

• key: rabbitmq.conf
  path: rabbitmq.conf
• key: enabled_plugins
  path: enabled_plugins
  name: rabbitmq-config
  name: config-volume

add something like this (dependent on storage provider):
volumeMounts:

name: rabbitmq-data
mountPath: /var/lib/rabbitmq/mnesia
volumeClaimTemplates:

metadata:
name: rabbitmq-data
annotations:
volume.beta.kubernetes.io/storage-class: gp2
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 4Gi

[seriva]

We need to check this with outstanding pull #3122. @Harshini097t could you check if this is what you are working on?

[seriva]

Not going todo anymore feature requests for Epiphany.