Allow setting sizeLimit when persistence disabled

[miguelmdh]

Is your feature request related to a problem? Please describe.

When no persistence is used with a RabbitmqCluster, the Operator sets an emptyDir volume for the persistence:

    - name: persistence
      emptyDir: {}

This causes that in the RabbitMQ UI, the available disk is the host's disk:

This introduces a potential problem in the host, since its disk may fill up and a RabbitMQ cluster can affect the rest workloads of the host.

Describe the solution you'd like

Kubernetes lets to set the sizeLimit for an empty dir: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir-configuration-example

The Operator could add a sizeLimit when no persistence is used. Questions we ask ourselves:

• What is the default value for sizeLimit?
• Should the user modify this value? If so, shoud the CRD be modified to add a new field?
• What about other related fields, such as the disk threshold alarm?

Describe alternatives you've considered

• Use a default value for sizeLimit (500Mb) that can be overridden by an annotation.
• Have a disk usage limit alarm. RabbitMQ now defines the disk_free_limit.absolute alarm, which allows to define an alarm when the disk has less that the specified amount of space. We propose the opposite: setting an alarm when disk usage exceeds a specified amount of space.

[MirahImage]

The option of using an emptyDir is only intended for short-lived development purposes, for example, when running locally on kind, and should never be used in other circumstances. If you are deploying a long-lived RabbitMQ cluster, then it should always have dedicated persistent volumes for storage.

[DanielDorado]

Hi, @miguelmdh, @jonathanCaamano, and I are in the same team.

We are collaborating humbly on this project because we are planning to use RabbitMQ widely across the company where we work. Our idea is to set up development environments without PVCs for testing, but we do not want to exhaust the nodes because the cluster is shared among multiple teams.

If the operator can use an empty dir, limiting it should be an option, as otherwise the Kubernetes node's disk can be exhausted.

I could understand no empty-dir at all, but allowing it without limiting it is not the best decision.

We are offering to do the Pull Request, and it is a minor change that does not harm anybody, and the review will take a little time.

[MirahImage]

If you're planning on using shared dev environments where disk exhaustion is a concern, why not just provision a PVC.

[DanielDorado]

Because using PVCs is more expensive for us.

For example, thanks to this PR #1899, we can put RabbitMQ clusters to zero when we do not plan using them (night, weekends...), but if we have the PVCs, the cost associated with the volumes is charged.

[MirahImage]

If you want to provide a PR for an optional sizeLimit for when no persistence is used, we'd consider it.

[DanielDorado]

Thanks a lot!