Nodes are running according to the management dashboard but rabbitmqctl commands display errors #2883
Description
I started a cluster of three nodes yesterday. The rabbitmqctl commands were able to display the status of the cluster as well as stopping and starting the nodes.
Today, when I run the command rabbitmqctl cluster_status I got the following error:
Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
However, in the management dashboard, all the nodes appear to be running and clients are able to send messages to the server.
Still, I run the command rabbitmqctl start_app and it appears to start the node according to the logs but with some errors. The rabbitmqctl cluster_status doesn't recognize the app running in the target node. Also, rabbitmqctl stop_app doesn't stop any of the nodes that appear to be running in the management console.
The relevant log excerpt seems to be: (full log below)
2021-03-11 08:51:46.067 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.068 [info] <0.11297.3>
Starting RabbitMQ 3.8.9 on Erlang 23.2.4
Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
Licensed under the MPL 2.0. Website: https://rabbitmq.com
2021-03-11 08:51:46.069 [info] <0.63.0> supervisor: {<0.63.0>,user_sup}, errorContext: child_terminated, reason: eio, offender: [{pid,<0.64.0>},{mod,user_sup}]
2021-03-11 08:51:46.070 [error] <0.63.0> Supervisor {<0.63.0>,user_sup} had child at module user_sup at <0.64.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.070 [error] <0.63.0> ** Generic server <0.63.0> terminating
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46.070 [error] <0.63.0> CRASH REPORT Process <0.63.0> with 0 neighbours exited with reason: eio in gen_server:handle_common_reply/8 line 805
2021-03-11 08:51:46.070 [error] <0.49.0> Supervisor kernel_sup had child user started with user_sup:start() at <0.63.0> exit with reason eio in context child_terminated
Full log
2021-03-11 08:51:44.660 [info] <0.11250.3> RabbitMQ is asked to start...
2021-03-11 08:51:44.660 [info] <0.44.0> Application rabbit_common started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.136 [debug] <0.11256.3> Adding Defaults
2021-03-11 08:51:45.136 [debug] <0.11256.3> Right Hand Side Substitutions
2021-03-11 08:51:45.136 [debug] <0.11256.3> Applying Datatypes
2021-03-11 08:51:45.137 [debug] <0.11256.3> Validation
2021-03-11 08:51:45.151 [debug] <0.11256.3> Applied 1:1 Mappings
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.writer_gc_threshold" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> Running translation for rabbit.loopback_users
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.ciphers" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.auth_mechanisms" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.auth_backends" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_vhost" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_user" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_pass" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_permissions" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_user_tags" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.connection_max" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.vm_memory_high_watermark" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.disk_free_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_partition_handling" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> Running translation for rabbit.cluster_formation.peer_discovery_backend
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.node_type" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.randomized_startup_delay_range" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> Running translation for rabbit.cluster_nodes
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_dns.hostname" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.queue_master_locator" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "syslog.dest_host" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "syslog.protocol" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.process_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.port_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.gc_ms_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.schedule_ms_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.heap_word_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.busy_port" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.busy_dist_port" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.segment_max_entries" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.wal_max_size_bytes" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.wal_max_entries" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.wal_hibernate_after" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.wal_max_batch_size" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.snapshot_chunk_size" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.data_dir" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_amqp1_0.default_vhost" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_cache.cached_backend" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.servers" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.dn_lookup_bind" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.other_bind" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_event_exchange.vhost" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.cacerts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.cert" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.dh" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.key" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.cors_allow_origins" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.cors_max_age" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.content_security_policy" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.strict_transport_security" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.sample_retention_policies" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_mqtt.default_user" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.default_pass" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.vhost" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.exchange" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.subscription_ttl" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.rabbit.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_ec2_region" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_use_private_ip" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_access_key" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_secret_key" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_autoscaling" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_ec2_tags" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.node_cleanup.cleanup_interval" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.node_cleanup.cleanup_only_log_warning" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.http_proxy" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.https_proxy" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.proxy_exclusions" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_host" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_port" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_scheme" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_acl_token" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.cluster_name" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_auto" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_nic" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_nodename" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_domain" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> Running translation for rabbit.cluster_formation.peer_discovery_consul.consul_include_nodes_with_warnings
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_ttl" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_deregister_after" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_use_longname" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_tags" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_meta" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_lock_prefix" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.lock_wait_time" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.endpoints" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_host" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_scheme" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_prefix" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.cluster_name" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_node_ttl" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.lock_wait_time" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_username" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_password" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.ciphers" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_host" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_scheme" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_token_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_cert_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_namespace_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_service_name" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_address_type" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_hostname_suffix" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_prometheus.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_prometheus.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.default_vhost" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.default_topic_exchange" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.providers" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.tcp_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.cowboy_opts.idle_timeout" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.tcp_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.cowboy_opts.idle_timeout" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> Applied Translations
2021-03-11 08:51:45.171 [info] <0.44.0> Application rabbitmq_prelaunch started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.185 [info] <0.44.0> Application os_mon started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.209 [info] <0.11281.3> ra: meta data store initialised. 3 record(s) recovered
2021-03-11 08:51:45.296 [info] <0.44.0> Application ra started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.297 [info] <0.44.0> Application sysmon_handler started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.307 [info] <0.11297.3> Stopping application 'syslog'
2021-03-11 08:51:45.307 [info] <0.11297.3> Stopping application 'lager'
2021-03-11 08:51:45.322 [debug] <0.11312.3> Lager installed handler error_logger_lager_h into error_logger
2021-03-11 08:51:45.325 [debug] <0.11315.3> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-03-11 08:51:45.325 [debug] <0.11318.3> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-03-11 08:51:45.325 [debug] <0.11321.3> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-03-11 08:51:45.325 [debug] <0.11324.3> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-03-11 08:51:45.325 [debug] <0.11327.3> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-03-11 08:51:45.325 [debug] <0.11330.3> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-03-11 08:51:45.325 [debug] <0.11333.3> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-03-11 08:51:45.325 [debug] <0.11336.3> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-03-11 08:51:45.325 [debug] <0.11339.3> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-03-11 08:51:45.325 [debug] <0.11342.3> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-03-11 08:51:45.325 [debug] <0.11345.3> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-03-11 08:51:45.325 [debug] <0.11348.3> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-03-11 08:51:45.355 [info] <0.44.0> Application lager started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.360 [info] <0.11297.3> Log file opened with Lager
2021-03-11 08:51:45.363 [debug] <0.11351.3> Lager installed handler {lager_file_backend,
"/exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252_upgrade.log"} into rabbit_log_upgrade_lager_event
2021-03-11 08:51:45.412 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.423 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-03-11 08:51:45.423 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-03-11 08:51:45.823 [debug] <0.11307.3> Lager installed handler lager_backend_throttle into lager_event
2021-03-11 08:51:46.067 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.068 [info] <0.11297.3>
Starting RabbitMQ 3.8.9 on Erlang 23.2.4
Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
Licensed under the MPL 2.0. Website: https://rabbitmq.com
2021-03-11 08:51:46.069 [info] <0.63.0> supervisor: {<0.63.0>,user_sup}, errorContext: child_terminated, reason: eio, offender: [{pid,<0.64.0>},{mod,user_sup}]
2021-03-11 08:51:46.070 [error] <0.63.0> Supervisor {<0.63.0>,user_sup} had child at module user_sup at <0.64.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.070 [error] <0.63.0> ** Generic server <0.63.0> terminating
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46.070 [error] <0.63.0> CRASH REPORT Process <0.63.0> with 0 neighbours exited with reason: eio in gen_server:handle_common_reply/8 line 805
2021-03-11 08:51:46.070 [error] <0.49.0> Supervisor kernel_sup had child user started with user_sup:start() at <0.63.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.072 [info] <0.11297.3>
node : rabbit@cs04r-sc-vserv-252
home dir : /exports/rmq/2021-03-10
config file(s) : /exports/rmq/2021-03-10/rabbitmq.conf
cookie hash : YutjBenRobSKMf92dFCNTQ==
log(s) : /exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252.log
: /exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252_upgrade.log
database dir : /exports/rmq/2021-03-10/mnesia/rabbit@cs04r-sc-vserv-252
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step pre_boot defined by app rabbit
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step rabbit_core_metrics defined by app rabbit
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step rabbit_alarm defined by app rabbit
2021-03-11 08:51:46.130 [info] <0.11425.3> Memory high watermark set to 1515 MiB (1589518336 bytes) of 3789 MiB (3973795840 bytes) total
2021-03-11 08:51:46.137 [info] <0.11427.3> Enabling free disk space monitoring
2021-03-11 08:51:46.137 [info] <0.11427.3> Disk free limit set to 50MB
2021-03-11 08:51:46.142 [info] <0.11297.3> Running boot step code_server_cache defined by app rabbit
2021-03-11 08:51:46.142 [info] <0.11297.3> Running boot step file_handle_cache defined by app rabbit
2021-03-11 08:51:46.142 [info] <0.11430.3> Limiting to approx 927 file handles (832 sockets)
2021-03-11 08:51:46.142 [info] <0.11431.3> FHC read buffering: OFF
2021-03-11 08:51:46.142 [info] <0.11431.3> FHC write buffering: ON
2021-03-11 08:51:46.144 [info] <0.11297.3> Running boot step worker_pool defined by app rabbit
2021-03-11 08:51:46.144 [info] <0.11414.3> Will use 1 processes for default worker pool
2021-03-11 08:51:46.144 [info] <0.11414.3> Starting worker pool 'worker_pool' with 1 processes in it
2021-03-11 08:51:46.145 [info] <0.11297.3> Running boot step database defined by app rabbit
2021-03-11 08:51:46.384 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.480 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.481 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.481 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.489 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.489 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.520 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.520 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.520 [info] <0.11297.3> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step database_sync defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step feature_flags defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step codec_correctness_check defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step external_infrastructure defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_registry defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_queue_location_random defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_event defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_priority_queue defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step kernel_ready defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-03-11 08:51:46.524 [info] <0.11526.3> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-03-11 08:51:46.524 [info] <0.11297.3> Running boot step guid_generator defined by app rabbit
2021-03-11 08:51:46.534 [info] <0.11297.3> Running boot step rabbit_node_monitor defined by app rabbit
2021-03-11 08:51:46.539 [info] <0.11530.3> Starting rabbit_node_monitor
2021-03-11 08:51:46.539 [info] <0.11297.3> Running boot step delegate_sup defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step rabbit_memory_monitor defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step core_initialized defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step upgrade_queues defined by app rabbit
2021-03-11 08:51:46.564 [info] <0.11297.3> Running boot step rabbit_connection_tracking defined by app rabbit
2021-03-11 08:51:46.564 [info] <0.11297.3> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_policies defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_policy defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_vhost_limit defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step recovery defined by app rabbit
2021-03-11 08:51:46.567 [info] <0.11559.3> Making sure data directory '/exports/rmq/2021-03-10/mnesia/rabbit@cs04r-sc-vserv-252/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-03-11 08:51:46.575 [info] <0.11559.3> Starting message stores for vhost '/'
2021-03-11 08:51:46.575 [info] <0.11563.3> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-03-11 08:51:46.578 [info] <0.11559.3> Started message store of type transient for vhost '/'
2021-03-11 08:51:46.578 [info] <0.11567.3> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-03-11 08:51:46.580 [info] <0.11559.3> Started message store of type persistent for vhost '/'
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step empty_db_check defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Will not seed default virtual host and user: have definitions to load...
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step rabbit_looking_glass defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step background_gc defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step connection_tracking defined by app rabbit
2021-03-11 08:51:46.834 [info] <0.11297.3> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.844 [info] <0.11297.3> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step routing_ready defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step pre_flight defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step notify_cluster defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step networking defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step definition_import_worker_pool defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11414.3> Starting worker pool 'definition_import_pool' with 1 processes in it
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step cluster_name defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step direct_client defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.44.0> Application rabbit started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.847 [info] <0.11530.3> rabbit on node 'rabbit@cs04r-sc-vserv-254' up
2021-03-11 08:51:47.175 [notice] <0.11577.3> queue 'transient.destination' in vhost '/': candidate -> leader in term: 2 machine version: 0
2021-03-11 08:51:48.160 [info] <0.11606.3> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-03-11 08:51:48.160 [info] <0.11606.3> Management plugin: using rates mode 'basic'
2021-03-11 08:51:48.221 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.235 [info] <0.44.0> Application cowlib started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.247 [info] <0.44.0> Application cowboy started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.260 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.272 [info] <0.44.0> Application amqp_client started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.283 [info] <0.11606.3> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2021-03-11 08:51:48.283 [info] <0.11606.3> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2021-03-11 08:51:48.309 [info] <0.11674.3> Management plugin: HTTP (non-TLS) listener started on port 15672
2021-03-11 08:51:48.309 [info] <0.11780.3> Statistics database started.
2021-03-11 08:51:48.309 [info] <0.11779.3> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-03-11 08:51:48.310 [info] <0.44.0> Application rabbitmq_management started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.341 [info] <0.44.0> Application prometheus started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.722 [notice] <0.11582.3> queue 'queue.status' in vhost '/': candidate -> leader in term: 26 machine version: 0
2021-03-11 08:52:40.848 [info] <0.11530.3> rabbit on node 'rabbit@cs04r-sc-vserv-253' up
2021-03-11 08:53:28.777 [info] <0.12374.3> MQTT retained message store: rabbit_mqtt_retained_msg_store_dets
2021-03-11 08:53:28.779 [info] <0.12392.3> started MQTT TCP listener on [::]:1883
2021-03-11 08:53:28.781 [info] <0.44.0> Application rabbitmq_mqtt started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.799 [info] <0.12395.3> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-03-11 08:53:28.799 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.810 [info] <0.12502.3> rabbit_stomp: default user 'guest' enabled
2021-03-11 08:53:28.812 [info] <0.12518.3> started STOMP TCP listener on [::]:61613
2021-03-11 08:53:28.813 [info] <0.11606.3> Ready to start client connection listeners
2021-03-11 08:53:28.813 [info] <0.44.0> Application rabbitmq_stomp started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.817 [info] <0.12533.3> started TCP listener on [::]:5672
2021-03-11 08:53:28.819 [notice] <0.11304.3> Changed loghwm of /exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252.log to 500000000000000000000000000000000000000000000000
2021-03-11 08:53:29.139 [notice] <0.12359.3> mqtt_node: candidate -> leader in term: 29 machine version: 0
2021-03-11 08:53:29.236 [info] <0.11606.3> Server startup complete; 6 plugins started.
* rabbitmq_stomp
* rabbitmq_prometheus
* rabbitmq_mqtt
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_management_agent
And this is from the crash.log
2021-03-11 08:51:46 =SUPERVISOR REPORT====
Supervisor: {<0.63.0>,user_sup}
Context: child_terminated
Reason: eio
Offender: [{pid,<0.64.0>},{mod,user_sup}]
2021-03-11 08:51:46 =ERROR REPORT====
** Generic server <0.63.0> terminating
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46 =CRASH REPORT====
crasher:
initial call: supervisor_bridge:user_sup/1
pid: <0.63.0>
registered_name: []
exception exit: {eio,[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,805}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
ancestors: [kernel_sup,<0.47.0>]
message_queue_len: 0
messages: []
links: [<0.49.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 1598
stack_size: 28
reductions: 1181
neighbours:
2021-03-11 08:51:46 =SUPERVISOR REPORT====
Supervisor: {local,kernel_sup}
Context: child_terminated
Reason: eio
Offender: [{pid,<0.63.0>},{id,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]
To be able to use rabbitmqctl I had to kill the processes and start the servers again but this is not ideal.
Thank you.