Added useful variables to override the default #125
Conversation
pull-request-size
bot
added
the
size/XS
values.yaml
Outdated
| @@ -440,6 +441,11 @@ rabbitmq-ha: | |||
| rabbitmqUsername: admin | |||
| # TODO: Use default random 24 character password, but need to fetch this string for use by downstream services | |||
| rabbitmqPassword: 9jS+w1u07NbHtZke1m+jW4Cj | |||
| # RabbitMQ available memory | |||
| rabbitmqMemoryHighWatermark: "0.8" | |||
There was a problem hiding this comment.
This is very custom value which I'd argue makes sense for everyone.
Can you clarify the practical use case and why it's important in K8s environment and is absolutely recommended for any environment?
There was a problem hiding this comment.
That value should not be fixed necessarily but the stackstorm-ha deployment starts working erroneously when the default memory watermark attached to the RabbitMQ Pods is full (to me, this happened after less than one day) RabbitMQ Issue #12730.
This is why I insist on adding these attributes to the values file of stackstorm-ha and maybe you can help me with a more suitable default value.
There was a problem hiding this comment.
@stefangusa Thanks for the explanation.
If no resources limits set for the RabbitMQ Deployment/Pods, how the relative memory watermark setting
rabbitmqMemoryHighWatermark: "0.8"
rabbitmqMemoryHighWatermarkType: relative
behaves in a real-world K8s deployment and how much memory RabbitMQ cluster acquires in that situation?
There was a problem hiding this comment.
From the RabbitMQ docs,
The default value of 0.4 stands for 40% of availalbe (detected) RAM or 40% of available virtual address space, whichever is smaller. E.g. on a 32-bit platform with 4 GiB of RAM installed, 40% of 4 GiB is 1.6 GiB, but 32-bit Windows normally limits processes to 2 GiB, so the threshold is actually to 40% of 2 GiB (which is 820 MiB).
That setting is something I have chosen heuristically based on my needs, I haven't made a deep analysis to potentially lower the value as I haven't had to so far.
There was a problem hiding this comment.
The default value in RabbitMQ Helm chart is
rabbitmqMemoryHighWatermark: 256MB
rabbitmqMemoryHighWatermarkType: absolute
I'm wondering it was set that way to an absolute value by a reason and what's happening with the relative settings if we recommend it to user, considering there is no Pod limit is set.
Will the RabbitMQ take entire possible cluster memory or what in that scenario?
Depending on how it behaves results in what we can suggest to the users.
There was a problem hiding this comment.
According to the docs:
Default:
vm_memory_high_watermark.relative = 0.4
I don't really figure out why that absolute value is set but I found an issue #17089 where that default value solved the issue. I can also see other issues regarding this problem in the GitHub Charts repo issues.
Regarding the last question, according to the docs:
rabbitmqctl set_vm_memory_high_watermark fraction
When using the absolute mode, it is possible to use one of the following memory units:
M, MiB for mebibytes (2^20 bytes)
MB for megabytes (10^6 bytes)
G, GiB for gibibytes (2^30 bytes)
GB for gigabytes (10^9 bytes)
it is revealed that both rabbitmqMemoryHighWatermark and rabbitmqMemoryHighWatermarkType parameters need to be set correctly (fraction for relative type and value + memory units for absolute type) in order that RabbitMQ can start.
There was a problem hiding this comment.
I'm referring to RabbitMQ chart defaults we're using which relies on absolute rabbitmqMemoryHighWatermarkType and hard memory limit by default. This probably happens by a reason as we're not in VM environment anymore where entire memory could be potentially dedicated to a single app like RabbitMQ.
It's important to make sure safety settings before we suggest our users to rely on relative memory in a highly distributed K8s environment. So can you please verify/experiment that behavior in K8s cluster?
What I mean, specifically:
If RabbitMQ-HA is deployed on K8s cluster that has for example 100GB of memory pool, with * 0.8 rabbitmqMemoryHighWatermark setting how much memory would RabbitMQ cluster may take before enforcing connection throttling, considering Pods have no memory limits set? This could be a huge and non-practical value that can affect entire K8s cluster.
What if K8s cluster consists from a nodes with different memory resources. And so for example 1 Pod of RabbitMQ HA cluster is deployed on a node with 5GB of memory, but another is deployed by a scheduler on a node with 128GB of memory, how * 0.8 memory watermark will behave in reality with this relative setting?
There was a problem hiding this comment.
Good, now I see the problem and watching it from this point of view I agree that there is an issue and thank you for explaining it!
The RabbitMQ with relative high watermark takes a percent of the memory available from the machine where it runs on. An if three replicas run on three machines with different memory resources and there is no bound, each replica is going to have another amount of memory available.
For this reason, I suggest we keep both of the fields but with other values: rabbitmqMemoryHighWatermarkType: absolute and rabbitmqMemoryHighWatermark: 512MB.
This way, along with the comment in code, people are made aware of these fields and are given a short description of what they mean and in addition to this, a "safer" value (according to the issues mentioned in the code comment and above in this discussion) for the memory available is set. An if this turns out not to be enough for somebody, he would know precisely what to overwrite.
There was a problem hiding this comment.
Makes sense, agree with that 👍
Just left a few code recommendations so we make these suggestions clearer in code comments.
Co-Authored-By: Eugen C. <armab@users.noreply.github.com>
Co-Authored-By: Eugen C. <armab@users.noreply.github.com>
pull-request-size
bot
added
size/S
Closes #124