The memory usage is huge on MQTT with cleanSession=false and can not be limited by high water mark (3.12.13)

[absolute8511]

Describe the bug

As showed below, the mqtt reader seems using more than 10MB for each connection, and this memory can not be limited by high watermark setting @ansd

Memory
Total memory used: 11.711 gb
Calculation strategy: rss
Memory high watermark setting: 0.7 of available memory, computed to: 22.5486 gb
connection_other: 42.0437 gb (85.27 %)
allocated_unused: 5.4495 gb (11.05 %)
other_proc: 0.8869 gb (1.8 %)
mnesia: 0.2494 gb (0.51 %)
queue_procs: 0.2216 gb (0.45 %)
binary: 0.1292 gb (0.26 %)
plugins: 0.113 gb (0.23 %)
other_system: 0.0682 gb (0.14 %)
other_ets: 0.0536 gb (0.11 %)
metrics: 0.0514 gb (0.1 %)
code: 0.0367 gb (0.07 %)
atom: 0.0015 gb (0.0 %)
msg_index: 0.0004 gb (0.0 %)
quorum_ets: 0.0 gb (0.0 %)
mgmt_db: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)
File Descriptors
Total: 13508, limit: 1048479
Sockets: 13506, limit: 943629
Free Disk Space
Low free disk space watermark: 2.0 gb
Free disk space: 105.9739 gb
Totals
Connection count: 13506
Queue count: 76494
Virtual host count: 1

Reproduction steps

1. using the newest rabbitmq 3.12.13
2. using the server config with below, with 5 nodes

rabbitmq:
    additionalConfig: |
      cluster_partition_handling = pause_minority
      collect_statistics_interval = 160000
      log.file.rotation.count = 20
      log.file.rotation.date = $D0
      tcp_listen_options.backlog = 512
      tcp_listen_options.keepalive = true
      tcp_listen_options.linger.on = true
      tcp_listen_options.linger.timeout = 0
      tcp_listen_options.nodelay = true
      tcp_listen_options.recbuf = 32768
      tcp_listen_options.sndbuf = 32768
      mqtt.tcp_listen_options.sndbuf  = 4024
      mqtt.tcp_listen_options.recbuf  = 4024
      mqtt.tcp_listen_options.buffer = 4024
      mqtt.mailbox_soft_limit = 10
      vm_memory_high_watermark.relative = 0.7
      classic_queue.default_version = 2
      management_agent.disable_metrics_collector = true
      log.console = true
      log.console.level = warning
    additionalPlugins:
    - rabbitmq_mqtt
    - rabbitmq_web_mqtt
    advancedConfig: |-
      [
        {rabbit, [
            {connection_max, 900000}
        ]}
      ].
    envConfig: |
      RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+P 40000000 +zdbbl 1000000 -mnesia dump_log_write_threshold 50000 +S 20:20 +SDcpu 10:10 +SDio 15"
      ERL_MAX_PORTS=5000000

3. use bench mark below

./emqtt_bench sub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i
          --qos=1 -c 20000 -i 4 --host xxx
          --port 1883 --version 4 --username xxx --password xxx --clean false

./emqtt_bench pub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i
          --qos=1 --interval_of_msg=120000 --size=64 -c 20000 -i 10 --host xxx
          --port 1883 --version 4 --username xxx --password xxx --clean false;

wait for running for a while and everything looks ok, until we start a new bench again below, we can see the server memory usage will grow quickly

./emqtt_bench sub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i
          --qos=1 -c 20000 -i 4 --host xxx
          --port 1883 --version 4 --username xxx --password xxx --clean false

Expected behavior

Should not use so much memory and the memory used by mqtt should be limited

Additional context

No response

[michaelklishin]

This topic has been documented for years in Tuning for a Large Number of Concurrent Connections

[absolute8511]

tcp options has already been used in the config above, but it is not work as expected. From the output of rabbitmq-diagnostics observer and rabbitmq-diagnostics status, we can see the memory is not used by tcp buffer.
And by the way, in this test the connections is just 60K which is not so much

[michaelklishin]

This blog post uses very low TCP buffer sizes but for mostly idle connections this should be perfectly fine.

Also, the high VM memory watermark does not limit the number of connections. What it does is documented in the Alarms guide.

There is a separate mechanism (more than one, in fact) that allows you to limit the number of connections specifically to protect the node from applications that leak connections.

[ansd]

FWIW, I tried to repro.

I deployed a single node (not 5 node) cluster on a Linux box:

make run-broker PLUGINS="rabbitmq_web_mqtt rabbitmq_management rabbitmq_prometheus" RABBITMQ_CONFIG_FILE="rabbitmq.conf" RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+S 6 +P 40000000 +zdbbl 1000000 -mnesia dump_log_write_threshold 50000" FULL=1

with the following rabbitmq.conf:

cluster_partition_handling = pause_minority
collect_statistics_interval = 160000
tcp_listen_options.backlog = 512
tcp_listen_options.keepalive = true
tcp_listen_options.linger.on = true
tcp_listen_options.linger.timeout = 0
tcp_listen_options.nodelay = true
tcp_listen_options.recbuf = 32768
tcp_listen_options.sndbuf = 32768
mqtt.tcp_listen_options.sndbuf  = 4024
mqtt.tcp_listen_options.recbuf  = 4024
mqtt.tcp_listen_options.buffer = 4024
mqtt.mailbox_soft_limit = 10
vm_memory_high_watermark.relative = 0.7
classic_queue.default_version = 2
management_agent.disable_metrics_collector = true
connection_max = 900000

I started clients as follows:

./emqtt_bench sub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i --qos=1 -c 20000 -i 4 --port 1883 --version 4 --username guest --password guest --clean false

./emqtt_bench pub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i --qos=1 --interval_of_msg=120000 --size=64 -c 20000 -i 10 --port 1883 --version 4 --username guest --password guest --clean false

after 2 mins:

./emqtt_bench sub --keepalive=600 --num-retry-connect=5 --topic=pod-test2/%i --qos=1 -c 20000 -i 4 --port 1883 --version 4 --username guest --password guest --clean false

Messages flow through the system and memory usage of the broker is kept at only 3.2 GB.
Memory usage per MQTT connection is kept below 66 KB.

[ansd]

If yes, try again without such a policy, and please report back whether this helps.

[absolute8511]

Since the total memory usage is more than 25GB, maybe the biggest 183MB is not an issue? After all connections finished, the total memory usage is fall below to only 3GB. For each connection taking 12MB, with 2000/s creation it will cost 24GB at the peak.

[absolute8511]

I found the queue_master_locator=min-masters is set by default in rabbitmq cluster-operator, so I test with queue_master_locator=random again, it seems much better. Under 2000/s connection creation it used less than 4GB memory.

[ansd]

So, we found the bottleneck:
queue_master_locator = min-masters (applicable to classic queues) reads all queue records from the database, which leads to unnecessary high resource usage.
In contrast, queue_leader_locator = balanced (applicable to quorum queues and streams) provides a better implementation since nodes will be selected randomly when there are more than 1000 queues.

queue_master_locator = min-masters is indeed the default in the cluster-operator.

I suggest you use queue_master_locator = client-local so that your MQTT clients can consume locally from the node they are connected to.

I test with queue_master_locator=random again, it seems much better. Under 2000/s connection creation it used less than 4GB memory

@absolute8511 did you test queue_master_locator=random with Khepri or with Mnesia? Did you observe better performance with Khepri?

[ansd]

created rabbitmq/rabbitmq-website#1826

[absolute8511]

I test both with Khepri and with Mnesia, and it showed not much different for memory usage