Skip to content

epmd using wrong interface #1707

@domhaas

Description

@domhaas

Hi there,

I'm running a 3 Node RMQ-Cluster. The Cluster should communicate over a dedicated switch-network (192.168.0/24 eth1). The eth0 is the public interface which is used for internet access.

The cluster is working, but I got a strange issue while using the rabbitmqctl.
I'm trying to get some information about RMQ (for munin) by calling for example:
rabbitmqctl list_queues

The response is:

Timeout: 60.0 seconds ...
Listing queues for vhost / ...
test	0

But the process is hanging and exists after the timeout. So I checked everything and found out that the firewall is blocking an incoming request of the other nodes. But not on the configured LAN-Interface (eth1) but on the public interface (eth0).
I looks like that the epmd on the othere nodes trying to do a callback to the node on the requesting server:
Sep 22 14:17:51 db1 kernel: [60437.488302] [UFW BLOCK] IN=eth0 OUT= MAC=<hidden> SRC=<public-ip> DST=<public-ip> LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=30019 DF PROTO=TCP SPT=53691 DPT=4369 WINDOW=29200 RES=0x00 SYN URGP=0

So I found out that the port 4369 is used by epmd. So I checked everything about binding to the correct interface (see config below).
I also tried many combinations of the rabbitmqctl commands, like giving the node-name:

rabbitmqctl list_queues -n rabbit@db2.testdomain.com
rabbitmqctl list_queues -n rabbit@db3.testdomain.com

I'm really stucked with that currently. Am I missing something in my config.
Thank you in advance.

Dominik


  • RabbitMQ version: 3.7.7

  • Erlang version: 21.0.9

  • RabbitMQ plugin information:

[E*] rabbitmq_management               3.7.7
[e*] rabbitmq_management_agent         3.7.7
[e*] rabbitmq_web_dispatch             3.7.7
  • Operating system: Debian 9.5 (Kernel 4.9.0-8)

  • /etc/rabbitmq/rabbitmq.config

[
 {rabbit, [
   {tcp_listeners, [{"192.168.190.10", 5672}]}
  ]}
 ,{kernel,[{inet_dist_use_interface,{  192,168,190,10  }}]}
 ,{rabbitmq_management,
  [
    {listener, [{port, 15672}, {ip, "192.168.190.10"}]},
    {path_prefix, "/rabbitmq"}
  ]}
].
  • /etc/rabbitmq/rabbitmq-env.conf
export RABBITMQ_USE_LONGNAME=true
export NODENAME=rabbit@db1.domain.com
export NODE_IP_ADDRESS=192.168.190.10
export ERL_EPMD_ADDRESS=192.168.190.10
  • Logfile
2018-09-22 13:55:41.432 [info] <0.297.0> 
 Starting RabbitMQ 3.7.7 on Erlang 21.0.9
 Copyright (C) 2007-2018 Pivotal Software, Inc.
 Licensed under the MPL.  See http://www.rabbitmq.com/
2018-09-22 13:55:41.432 [info] <0.297.0> 
 node           : rabbit@db1.testdomain.com
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.config
 cookie hash    : YrPdN3lVso9h3XJkme/cGg==
 log(s)         : /var/log/rabbitmq/rabbit@db1.testdomain.com.log
                : /var/log/rabbitmq/rabbit@db1.testdomain.com_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@db1.testdomain.com
2018-09-22 13:55:41.899 [info] <0.357.0> Memory high watermark set to 25687 MiB (26935299276 bytes) of 64218 MiB (67338248192 bytes) total
2018-09-22 13:55:41.901 [info] <0.359.0> Enabling free disk space monitoring
2018-09-22 13:55:41.901 [info] <0.359.0> Disk free limit set to 50MB
2018-09-22 13:55:41.902 [info] <0.361.0> Limiting to approx 924 file handles (829 sockets)
2018-09-22 13:55:41.902 [info] <0.362.0> FHC read buffering:  OFF
2018-09-22 13:55:41.902 [info] <0.362.0> FHC write buffering: ON
2018-09-22 13:55:41.905 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.918 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.963 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2018-09-22 13:55:41.963 [info] <0.297.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2018-09-22 13:55:41.964 [info] <0.297.0> Priority queues enabled, real BQ is rabbit_variable_queue
2018-09-22 13:55:41.979 [info] <0.389.0> Starting rabbit_node_monitor
2018-09-22 13:55:41.996 [info] <0.297.0> Management plugin: using rates mode 'basic'
2018-09-22 13:55:41.998 [info] <0.421.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@db1.testdomain.com/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2018-09-22 13:55:42.004 [info] <0.421.0> Starting message stores for vhost '/'
2018-09-22 13:55:42.004 [info] <0.425.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2018-09-22 13:55:42.005 [info] <0.421.0> Started message store of type transient for vhost '/'
2018-09-22 13:55:42.005 [info] <0.428.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2018-09-22 13:55:42.006 [info] <0.421.0> Started message store of type persistent for vhost '/'
2018-09-22 13:55:42.030 [info] <0.468.0> started TCP Listener on 192.168.190.10:5672
2018-09-22 13:55:42.031 [info] <0.297.0> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@db1.testdomain.com'
2018-09-22 13:55:42.031 [info] <0.297.0> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@db1.testdomain.com'
2018-09-22 13:55:42.032 [info] <0.389.0> rabbit on node 'rabbit@db2.testdomain.com' up
2018-09-22 13:55:42.057 [info] <0.389.0> rabbit on node 'rabbit@db3.testdomain.com' up
2018-09-22 13:55:42.155 [info] <0.521.0> Management plugin started. Port: 15672
2018-09-22 13:55:42.155 [info] <0.627.0> Statistics database started.
2018-09-22 13:55:42.179 [notice] <0.107.0> Changed loghwm of /var/log/rabbitmq/rabbit@db1.testdomain.com.log to 50
2018-09-22 13:55:42.217 [info] <0.7.0> Server startup complete; 3 plugins started.
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent
  • netstat -ntlap | grep beam
tcp        0      0 192.168.190.10:5672     0.0.0.0:*               LISTEN      29797/beam.smp      
tcp        0      0 192.168.190.10:25672    0.0.0.0:*               LISTEN      29797/beam.smp      
tcp        0      0 192.168.190.10:15672    0.0.0.0:*               LISTEN      29797/beam.smp      
tcp        0      0 127.0.0.1:46215         127.0.0.1:4369          ESTABLISHED 29797/beam.smp      
tcp        0      0 192.168.190.10:25672    192.168.190.12:34321    ESTABLISHED 29797/beam.smp      
tcp        0      0 192.168.190.10:55719    192.168.190.11:25672    ESTABLISHED 29797/beam.smp
  • ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet <hidden>  netmask 255.255.255.192  broadcast 159.69.62.191
        inet6 <hidden>  prefixlen 64  scopeid 0x20<link>
        inet6 <hidden> prefixlen 64  scopeid 0x0<global>
        ether <hidden>  txqueuelen 1000  (Ethernet)
        RX packets 366350  bytes 174925955 (166.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 276713  bytes 126308539 (120.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xef100000-ef120000  

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.190.10  netmask 255.255.255.0  broadcast 192.168.190.255
        inet6 <hidden>  prefixlen 64  scopeid 0x20<link>
        ether <hidden>  txqueuelen 1000  (Ethernet)
        RX packets 5461633  bytes 639394879 (609.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5340752  bytes 592625441 (565.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16  memory 0xef0c0000-ef0e0000  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 1790262  bytes 187188161 (178.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1790262  bytes 187188161 (178.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussions/mailing list materialThis belongs to Discussions or the mailing list (rabbitmq-users on Google Groups)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions