-
Notifications
You must be signed in to change notification settings - Fork 4k
Description
Hi there,
I'm running a 3 Node RMQ-Cluster. The Cluster should communicate over a dedicated switch-network (192.168.0/24 eth1). The eth0 is the public interface which is used for internet access.
The cluster is working, but I got a strange issue while using the rabbitmqctl.
I'm trying to get some information about RMQ (for munin) by calling for example:
rabbitmqctl list_queues
The response is:
Timeout: 60.0 seconds ...
Listing queues for vhost / ...
test 0
But the process is hanging and exists after the timeout. So I checked everything and found out that the firewall is blocking an incoming request of the other nodes. But not on the configured LAN-Interface (eth1) but on the public interface (eth0).
I looks like that the epmd on the othere nodes trying to do a callback to the node on the requesting server:
Sep 22 14:17:51 db1 kernel: [60437.488302] [UFW BLOCK] IN=eth0 OUT= MAC=<hidden> SRC=<public-ip> DST=<public-ip> LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=30019 DF PROTO=TCP SPT=53691 DPT=4369 WINDOW=29200 RES=0x00 SYN URGP=0
So I found out that the port 4369 is used by epmd. So I checked everything about binding to the correct interface (see config below).
I also tried many combinations of the rabbitmqctl commands, like giving the node-name:
rabbitmqctl list_queues -n rabbit@db2.testdomain.com
rabbitmqctl list_queues -n rabbit@db3.testdomain.com
I'm really stucked with that currently. Am I missing something in my config.
Thank you in advance.
Dominik
-
RabbitMQ version: 3.7.7
-
Erlang version: 21.0.9
-
RabbitMQ plugin information:
[E*] rabbitmq_management 3.7.7
[e*] rabbitmq_management_agent 3.7.7
[e*] rabbitmq_web_dispatch 3.7.7
-
Operating system: Debian 9.5 (Kernel 4.9.0-8)
-
/etc/rabbitmq/rabbitmq.config
[
{rabbit, [
{tcp_listeners, [{"192.168.190.10", 5672}]}
]}
,{kernel,[{inet_dist_use_interface,{ 192,168,190,10 }}]}
,{rabbitmq_management,
[
{listener, [{port, 15672}, {ip, "192.168.190.10"}]},
{path_prefix, "/rabbitmq"}
]}
].
- /etc/rabbitmq/rabbitmq-env.conf
export RABBITMQ_USE_LONGNAME=true
export NODENAME=rabbit@db1.domain.com
export NODE_IP_ADDRESS=192.168.190.10
export ERL_EPMD_ADDRESS=192.168.190.10
- Logfile
2018-09-22 13:55:41.432 [info] <0.297.0>
Starting RabbitMQ 3.7.7 on Erlang 21.0.9
Copyright (C) 2007-2018 Pivotal Software, Inc.
Licensed under the MPL. See http://www.rabbitmq.com/
2018-09-22 13:55:41.432 [info] <0.297.0>
node : rabbit@db1.testdomain.com
home dir : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash : YrPdN3lVso9h3XJkme/cGg==
log(s) : /var/log/rabbitmq/rabbit@db1.testdomain.com.log
: /var/log/rabbitmq/rabbit@db1.testdomain.com_upgrade.log
database dir : /var/lib/rabbitmq/mnesia/rabbit@db1.testdomain.com
2018-09-22 13:55:41.899 [info] <0.357.0> Memory high watermark set to 25687 MiB (26935299276 bytes) of 64218 MiB (67338248192 bytes) total
2018-09-22 13:55:41.901 [info] <0.359.0> Enabling free disk space monitoring
2018-09-22 13:55:41.901 [info] <0.359.0> Disk free limit set to 50MB
2018-09-22 13:55:41.902 [info] <0.361.0> Limiting to approx 924 file handles (829 sockets)
2018-09-22 13:55:41.902 [info] <0.362.0> FHC read buffering: OFF
2018-09-22 13:55:41.902 [info] <0.362.0> FHC write buffering: ON
2018-09-22 13:55:41.905 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.918 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.963 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.963 [info] <0.297.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2018-09-22 13:55:41.964 [info] <0.297.0> Priority queues enabled, real BQ is rabbit_variable_queue
2018-09-22 13:55:41.979 [info] <0.389.0> Starting rabbit_node_monitor
2018-09-22 13:55:41.996 [info] <0.297.0> Management plugin: using rates mode 'basic'
2018-09-22 13:55:41.998 [info] <0.421.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@db1.testdomain.com/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2018-09-22 13:55:42.004 [info] <0.421.0> Starting message stores for vhost '/'
2018-09-22 13:55:42.004 [info] <0.425.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2018-09-22 13:55:42.005 [info] <0.421.0> Started message store of type transient for vhost '/'
2018-09-22 13:55:42.005 [info] <0.428.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2018-09-22 13:55:42.006 [info] <0.421.0> Started message store of type persistent for vhost '/'
2018-09-22 13:55:42.030 [info] <0.468.0> started TCP Listener on 192.168.190.10:5672
2018-09-22 13:55:42.031 [info] <0.297.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@db1.testdomain.com'
2018-09-22 13:55:42.031 [info] <0.297.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@db1.testdomain.com'
2018-09-22 13:55:42.032 [info] <0.389.0> rabbit on node 'rabbit@db2.testdomain.com' up
2018-09-22 13:55:42.057 [info] <0.389.0> rabbit on node 'rabbit@db3.testdomain.com' up
2018-09-22 13:55:42.155 [info] <0.521.0> Management plugin started. Port: 15672
2018-09-22 13:55:42.155 [info] <0.627.0> Statistics database started.
2018-09-22 13:55:42.179 [notice] <0.107.0> Changed loghwm of /var/log/rabbitmq/rabbit@db1.testdomain.com.log to 50
2018-09-22 13:55:42.217 [info] <0.7.0> Server startup complete; 3 plugins started.
* rabbitmq_management
* rabbitmq_web_dispatch
* rabbitmq_management_agent
- netstat -ntlap | grep beam
tcp 0 0 192.168.190.10:5672 0.0.0.0:* LISTEN 29797/beam.smp
tcp 0 0 192.168.190.10:25672 0.0.0.0:* LISTEN 29797/beam.smp
tcp 0 0 192.168.190.10:15672 0.0.0.0:* LISTEN 29797/beam.smp
tcp 0 0 127.0.0.1:46215 127.0.0.1:4369 ESTABLISHED 29797/beam.smp
tcp 0 0 192.168.190.10:25672 192.168.190.12:34321 ESTABLISHED 29797/beam.smp
tcp 0 0 192.168.190.10:55719 192.168.190.11:25672 ESTABLISHED 29797/beam.smp
- ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet <hidden> netmask 255.255.255.192 broadcast 159.69.62.191
inet6 <hidden> prefixlen 64 scopeid 0x20<link>
inet6 <hidden> prefixlen 64 scopeid 0x0<global>
ether <hidden> txqueuelen 1000 (Ethernet)
RX packets 366350 bytes 174925955 (166.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 276713 bytes 126308539 (120.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xef100000-ef120000
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.190.10 netmask 255.255.255.0 broadcast 192.168.190.255
inet6 <hidden> prefixlen 64 scopeid 0x20<link>
ether <hidden> txqueuelen 1000 (Ethernet)
RX packets 5461633 bytes 639394879 (609.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 5340752 bytes 592625441 (565.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xef0c0000-ef0e0000
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 1790262 bytes 187188161 (178.5 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1790262 bytes 187188161 (178.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0