Alternative memory used by Erlang VM calculation strategy

[gerhard]

We have seen the following scenario play out many times:

• Memory alarm is not triggered, publishers continue publishing
• RabbitMQ claims more memory via the Erlang VM
• Erlang VM claims more memory from the Kernel
• Kernel cannot allocate more memory since there isn't enough system memory available
• Kernel kills Erlang VM

This erlang-questions thread explains some of the discrepancies: the kernel and the runtime use different approaches to accounting and the kernel is blissfully unaware of whatever pre-allocation strategies a process can use.

We are proposing to use the memory as reported by the kernel in all places that erlang:memory/0 is called. When setting {use_system_memory_reporting,1}, we will try to use the process memory as reported by the system for vm_memory_high_watermark, most likely via the /proc/pid subsystem. If this is not possible, we will fallback to the current implementation.

We are thinking of introducing this as an experimental feature in 3.6 and learning whether this can be a viable default in 3.7

A few related issues that could benefit from this: #993 & #965.

UPD: the option name was changed to rabbit.vm_memory_calculation_strategy (which can be rss or erlang, the former is the new default, the latter is used to go to the earlier calculation strategy which is less accurate, often significantly so).

[michaelklishin]

Windows is particularly problematic here because gathering certain OS level metrics currently requires an external tool to be installed. The plan is to fall back to the less precise implementation if that's not available.

Also worth pointing out that we already obtain the total amount of memory in an OS-specific way (such as /proc/meminfo on Linux).

[michaelklishin]

A follow-up change for Windows: #1270.

[michaelklishin]

An update: due to what we've learned in #1343 and a few other places, as of 3.6.13 there are further adjustments to the strategy: as of rabbitmq/rabbitmq-common#225 it now uses runtime's allocators stats (which supposedly track every single malloc performed). This means no external tools are invoked. In addition, as of rabbitmq/rabbitmq-common#221 we avoid frequent calls to the function in question: it is now invoked once a second.

Because existing strategy names no longer make sense with these changes, we renamed them to allocated (née rss) and legacy (née erlang). rss and erlang are still supported for backwards compatibility, allocated/rss is still the default.