Space consuming after upgrade to 4.0.5 from 3.13.x in directory msg_store_persistent #13233
-
Community Support Policy
RabbitMQ version used4.0.5 Erlang version used26.2.x Operating system (distribution) usedRocky Linux release 9.5 How is RabbitMQ deployed?RPM package rabbitmq-diagnostics status outputSee https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics Logs from node 1 (with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 2 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs Logs from node 3 (if applicable, with sensitive values edited out)See https://www.rabbitmq.com/docs/logging to learn how to collect logs rabbitmq.confSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location Steps to deploy RabbitMQ clusterDownload latest version from: https://github.com/rabbitmq/rabbitmq-server/releases RPM for Rocky / CentOS Steps to reproduce the behavior in questionUpgrade with RPM from 3.13.x to 4.0.5 advanced.configSee https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location Application code# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKSWhat problem are you trying to solve?Disk space consumed abnormally. |
Beta Was this translation helpful? Give feedback.
Answered by
mkuratczyk
Feb 26, 2025
Replies: 1 comment 14 replies
-
|
Hi, I think you mean to continue the discussion started in #13230 which was closed early. But please copy over relevant information to this discussion as well. I think it is a relevant information that your disk space is taken up almost entirely by the If you could provide the listing of the rdq files ( I am commenting because we've also seen cases on 4.0.x when the msg_store process does not delete rdq files which are not used any more (no queue references messages in them). The msg_store process should delete such file after the last message was dereferenced from it or after an unclean shutdown during a recovery. Obviously we couldn't reproduce it on a test env. |
Beta Was this translation helpful? Give feedback.
14 replies
-
|
We faced the same in our RMQ server running v4.0.5. Our use case can sometime involve processing messages of large size (3-10MB). It would consume 800+ GB in few hours when message rate is high. And when the queues get cleared, storage will remain there. It does decrease but very slowly. So we suspect this problem could be related to high rate with large messages. We came across this discussion and decided to downgrade to 3.12.14. This solved space consumption problem. We haven't seen this issue in 3.12 (CQv2) so far with same level of load. |
Beta Was this translation helpful? Give feedback.
-
|
This is not a solution, but a workaround (you don't plan to stay on 3.12 forever, do you?). Please provide as much detail about your configuration and workload as you can. If you can reproduce it using https://perftest.rabbitmq.com/ - that'd be ideal and would ensure we can investigate and fix it. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like we managed to reproduce the problem |
Beta Was this translation helpful? Give feedback.
-
|
PR for reference: #13430 |
Beta Was this translation helpful? Give feedback.
Answer selected by
mkuratczyk
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
PR for reference: #13430