[Questions] RabbitMQ process leaves log files after several days of work

[kamilbro5]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.1

Erlang version used

27.3.x

Operating system (distribution) used

Red Hat Enterprise Linux 9.6

How is RabbitMQ deployed?

RPM package

rabbitmq.conf

default_vhost = test
#$default_user = *
#default_pass = *

# Log Settings
#log.dir = /app/logs/rmq/node1
#log.file = ******88
#log.file.level = debug
#log.file.rotation.date = $D0
#log.file.rotation.count = 120
listeners.tcp.default = 4443
#log.connection.level = info
#log.channel.level = info
#log.queue.level = info
#log.mirroring.level = info
#log.federation.level = info
#log.upgrade.level = info
#log.default.level = info

# SSL Settings
listeners.ssl.default = 9443
ssl_options.certfile = /etc/******.crt
ssl_options.keyfile = /etc/***********.key
ssl_options.fail_if_no_peer_cert = false
ssl_options.verify = verify_none
ssl_options.versions.1 = tlsv1.2
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
ssl_options.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.2 = DHE-RSA-AES128-SHA256
ssl_options.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = DHE-RSA-AES256-SHA256
ssl_options.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.6 = ECDHE-RSA-AES128-SHA256
ssl_options.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.8 = ECDHE-RSA-AES256-SHA384

# MGMT Settings
#management.tcp.port = 9060
management.ssl.port = 8443
management.ssl.idle_timeout =  120000
management.ssl.inactivity_timeout =  120000
management.ssl.request_timeout = 10000
management.ssl.certfile = /etc/*******.crt
management.ssl.keyfile = /etc/***********.key
management.ssl.versions.1 = tlsv1.2
management.ssl.honor_cipher_order = true
management.ssl.honor_ecc_order = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate = true
management.csp.policy = default-src 'self' *.************ 'unsafe-eval' 'unsafe-inline' frame-ancestors 'none'
management.hsts.policy = max-age=31536000; includeSubDomains
management.cors.allow_origins.1 = *.*******
management.cors.allow_origins.2 = *.*********
management.cors.max_age = 3600
management.headers.content_type_options = nosniff
management.headers.xss_protection = 1; mode=block
management.headers.frame_options = SAMEORIGIN
management.ssl.ciphers.1 = DHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.2 = DHE-RSA-AES128-SHA256
management.ssl.ciphers.3 = DHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.4 = DHE-RSA-AES256-SHA256
management.ssl.ciphers.5 = ECDHE-RSA-AES128-GCM-SHA256
management.ssl.ciphers.6 = ECDHE-RSA-AES128-SHA256
management.ssl.ciphers.7 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.8 = ECDHE-RSA-AES256-SHA384
management.sample_retention_policies.global.minute  = 5
management.sample_retention_policies.global.hour    = 60
management.sample_retention_policies.global.day     = 1200
management.sample_retention_policies.basic.minute       = 5
management.sample_retention_policies.basic.hour         = 60
management.sample_retention_policies.detailed.10        = 5

default_user_tags.administrator = true
default_user_tags.management = true

# DISK limits
disk_free_limit.relative = 0.80

# MEM Settings
vm_memory_calculation_strategy = rss
vm_memory_high_watermark.relative = 0.87
vm_memory_high_watermark_paging_ratio = 0.75

# LDAP Settings
auth_backends.1 = internal
auth_backends.2 = cache
auth_cache.cached_backend = ldap
auth_cache.cache_ttl = 300000
auth_ldap.servers.1  = *************8
auth_ldap.user_dn_pattern = CN=${username},CN=Users,DC=*********,DC=*********
auth_ldap.use_ssl    = true
auth_ldap.port       = 636
auth_ldap.log        = false
auth_ldap.timeout    = 30000
auth_ldap.idle_timeout = 18000


# Quorum queue properties
quorum_queue.continuous_membership_reconciliation.enabled = true
quorum_queue.continuous_membership_reconciliation.target_group_size = 3
quorum_queue.continuous_membership_reconciliation.auto_remove = true

# Custom config from databags
disk_free_limit.relative = 0.80
cluster_partition_handling  = autoheal
message_interceptors.incoming.set_header_timestamp.overwrite = false

Steps to deploy RabbitMQ cluster

yum install rabbitmq

Steps to reproduce the behavior in question

systemctl start rabbitmq-server

Rabbitmq works few days correctly.
After few days Rabbitmq not append lines to main log files. Process not use file log:
response from # fuser -u /app/logs/rmq/node1/log is empty
Log /app/logs/rmq/node1/log before this issue is cut without problem cause

advanced.config

[
  {rabbit, [
      {config_entry_decoder, [
             {passphrase,{ file, "/app/rmq/node1/*******"} },
             {cipher, blowfish_cfb64},
             {hash, sha256},
             {iterations, 10000}
         ]},
      {credential_validator, [{validation_backend, rabbit_credential_validator_password_regexp},
                                   {regexp, <<"^[a-zA-Z0-9\S$]{2,100}">>}]
        },
      {default_user, admin},
      {default_pass,
             {encrypted, <<"********************">> }},
      {log,
             [{file,
                 [{count,120},
                  {date,"$D0"},
                  {level,info},
                  {file,"log"}]},
              {categories,
                 [{default,[{level,info}]},
                  {federation,[{level,info}]},
                  {queue,[{file,"queue.log"},{level,info}]},
                  {mirroring,[{file,"mirror.log"},{level,info}]},
                  {channel,[{file,"channel.log"},{level,info}]},
                  {connection,[{file,"connection.log"},{level,info}]}]}]}
    ]},
   {rabbitmq_auth_backend_ldap,[
     {group_lookup_base, "DC=*********,DC=*********"},
     {ssl_options, [
                    {cacertfile,"/etc/**********.pem"},
                    {verify, verify_peer},
                    {fail_if_no_peer_cert, false}

     ]},
     {vhost_access_query, {'or', [{in_group, "CN=*********,OU=*********,OU=*********,OU=*********,DC=*********,DC=*********"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=*********,OU=*********,DC=*********,DC=*********"},
                          {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=*********,OU=*********,DC=*********,DC=*********"}]} },
     {tag_queries, [{administrator, {in_group, "CN=*********,OU=*********,OU=*********,OU=*********,DC=*********,DC=*********"}},
                    {monitoring, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=*********,OU=*********,DC=*********,DC=*********"}},
                    {management, {in_group, "CN=!${vhost}_Users,OU=RabbitMQ,OU=*********,OU=*********,DC=*********,DC=*********"}}
                    ] }    ]},
    {lager, [
        {error_logger_hwm, 1024}
    ]}
].

What problem are you trying to solve?

After restart logging works correctly few days.
Logs from serwer /var/log/messages:

Oct 25 23:46:48 ********* szef-client[115313]: {{:badarg, [{:ets, :lookup_element, [:mnesia_stats, :trans_log_writes, 2], [error_info: %{cause: :id, module: :erl_stdlib_errors}]}, {:mnesia_lib, :read_counter, 1, [file: ~c"mnesia_lib.erl", line: 607]}, {:mnesia_dumper, :get_log_writes, 0, [file: ~c"mnesia_dumper.erl", line: 63]}, {:rabbit_mnesia, :"-execute_mnesia_transaction/1-fun-0-", 1, [file: ~c"src/rabbit_mnesia.erl", line: 838]}, {:worker_pool_worker, :"-run/2-fun-0-", 3, [file: ~c"worker_pool_worker.erl", line: 64]}]}, {:gen_server2, :call, [#PID<14153.15262795.0>, {:submit, #Function<2.79321401/0 in :rabbit_mnesia>, #PID<14153.15263087.0>, :single}, :infinity]}}

-rw-r--r-- 1 rabbitmq rabbitmq 268101363 Oct 25 23:46:48 log

[michaelklishin]

We don't enough details to conclude anything. Maybe the partition the log files are on ran out of disk space.

We do not guess in this community.

Two things that can be recommended even then:

• You are three patch releases behind => upgrade to 4.1.5 when it comes out in a couple of days
• Take a look at what log rotation settings are used. Note that on Linux the recommended log rotation option is logrotate. Set it up to rotate log files daily, for example

[kamilbro5]

i monitor disk space 24h, out of disk space is not a problem.
We don't use linux logrotate. native application rabbitmq rotate log files
{log,
[{file,
[{count,120},
{date,"$D0"},
{level,info},
{file,"log"}]},

[michaelklishin]

That's a very standard daily rotation setup. I don't recall a single complaint about it in 3.13.x and onwards, and those from the earlier days had to do with settings missing (compared to older series before the logging library was replaced).

And yes, you should use logrotate, the RPM package sets it up with weekly rotation by default.

[lukebakken]

@kamilbro5 at the very least you MUST provide your full, unedited RabbitMQ configuration files. If you have anything sensitive, replace those characters with #.

[kamilbro5]

Hi, i attached full config to ticket in section detail rabbitmq.conf and section detail advanced.config

[lukebakken]

@kamilbro5 I have no idea why you are configuring logging via advanced.config, because it won't work since RabbitMQ changed logging subsystems. You should be using rabbitmq.conf. I have some information for you here:

https://github.com/lukebakken/rabbitmq-server-14829

@kamilbro5 if you used a specific example to configure logging via the docs site, rabbitmq.com, please point it out so we can fix it.

[michaelklishin]

The Logging guide has been updated to use rabbitmq.conf years ago.

[kamilbro5]

thx guys, i will change logging to rabbitmq.conf :)