Khepri projection memory footprint keeps growing with high binding churn

[jdennison-iforium]

Describe the bug

Hello,

We have recently upgraded our RabbitMQ cluster from RabbitMQ 3.7.5 (Erlang 20.3.4) to RabbitMQ 4.1.1 (Erlang 27.3.4).

Since the upgrade, we have noticed that RabbitMQ's memory utilisation is continually growing.

Upon investigation, we have noted:

• 'rabbitmq-diagnostics memory_breakdown' indicates that 'metadata_store_ets' consumes most of the memory: 1.1684 gb (59.11%)
• 'rabbitmq-diagnostics observer' indicates the 'rabbit_khepri_topic_trie' ETS table consumes 1.0855 GB

My suspicion is that 'rabbit_khepri_topic_trie' is continually growing larger due to one of our sub-systems utilising a high-churn of routing-keys on a particular exchange.

This suspicion seems to be validated by running:

• rabbitmqctl eval 'ets:tab2list(rabbit_khepri_topic_trie).'

which indicates that most of the rows in this table are relating to exchange of the sub-system in question.

We have not made any changes to this sub-system other than upgrading RabbitMQ. Prior to the upgrade, RabbitMQ memory consumption was not continually growing.

I note that RabbitMQ 4.1.1 is using Khepri for metadata storage instead of Mnesia used in 3.7.5.

It seems to me that Khepri metadata storage is not coping with the high-churn of routing-keys like Mnesia did.

To give an idea of routing-key churn:

We have ~2500 routing keys currently applied to the exchange, but the size of the 'rabbit_khepri_topic_trie' table is '4,335,432' rows.

Reproduction steps

1. Create high-churn of different routing-keys on an exchange.
2. Observe rabbit_khepri_topic_trie ETS table continually grow.

Expected behavior

I would expect memory utilisation to correlate with the number of routing-keys currently applied to an exchange, rather than grow indefinitely.

Additional context

No response

[mkuratczyk]

Thanks. I believe I can easily reproduce this. Seems like the binding deletion operation doesn't remove all entries related to the binding, so with high churn - the leftovers pile up.

Reproduction, starting from a fresh empty RabbitMQ:

1. Create a topic binding: rabbitmqadmin declare binding --source amq.topic --destination-type exchange --destination amq.direct --routing-key test
2. Check the contents of rabbit_khepri_topic_trie:

1> ets:tab2list(rabbit_khepri_topic_trie).
[{topic_trie_edge,{trie_edge,{resource,<<"/">>,exchange,
                                       <<"amq.topic">>},
                             #Ref<0.3750588171.3818389508.35802>,bindings},
                  {bindings,#{{binding,{resource,<<"/">>,exchange,
                                                 <<"amq.topic">>},
                                       <<"test">>,
                                       {resource,<<"/">>,exchange,<<"amq.direct">>},
                                       []} =>
                                  []}}},
 {topic_trie_edge,{trie_edge,{resource,<<"/">>,exchange,
                                       <<"amq.topic">>},
                             root,<<"test">>},
                  #Ref<0.3750588171.3818389508.35802>}]

3. Delete the binding: rabbitmqadmin delete binding --source amq.topic --destination-type exchange --destination amq.direct --routing-key test
4. Check the contents (expected empty but it's not empty):

(rabbit@carrot)2> ets:tab2list(rabbit_khepri_topic_trie).
[{topic_trie_edge,{trie_edge,{resource,<<"/">>,exchange,
                                       <<"amq.topic">>},
                             root,<<"test">>},
                  #Ref<0.3750588171.3818389508.35802>}]

[mkuratczyk]

If you need a quick workaround for now, a restart of the node should clean up these entries, so the memory usage will go down.

[jdennison-iforium]

Thanks for such a quick response! Yes, are monitoring the nodes and restarting periodically for now.

Fortunately, now we have set the following flag to reduce memory fragmentation, it isn't growing as fast as it was.

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+MMscs 2048"