Router thread failed to connect to broker for metadata

[yukselao]

Hello,

I have 2 test machines. One of them (host A) has 3 kafka broker instances (9092,9093,9094) + 1 zookeeper instance (2181)...

bruce is working on host B. And I want to push messages to unix domain socket on host B. And I want to consume them on host A.

What kind of configuration do we need to do that?

my configuration details:

I added:

to /etc/bruce/bruce_conf.xml file on host B. (bruce host)

Our network team open ports 9092,9093,9094 and 2181 between host A and host B.
Is it enough for communication or do we need another open ports?

bruce --msg_buffer_max 65536 --receive_socket_name /var/run/bruce/bruce.socket --config_path /etc/bruce/bruce_conf.xml --log_echo --log_level LOG_DEBUG --topic_autocreate --status_port 2122

bruce[21788]: Log started
bruce[21788]: Using "bruce" for client ID since none was specified with --client_id option. This is a workaround for a bug in Kafka 0.9.0.0 that causes broker to crash on receipt of produce request with empty client ID. See https://issues.apache.org/jira/browse/KAFKA-3088 for details.
bruce[21788]: Version: [1.0.28]
bruce[21788]: Config file: [/etc/bruce/bruce_conf.xml]
bruce[21788]: UNIX domain datagram input socket [/var/run/bruce/bruce.socket]
bruce[21788]: UNIX domain datagram input socket mode
bruce[21788]: Using Kafka protocol version [0]
bruce[21788]: Listening on status port 2122
bruce[21788]: Web interface loopback only: false
bruce[21788]: Buffered message limit 65536 kbytes
bruce[21788]: Max input message size 65536 bytes
bruce[21788]: Allow large UNIX datagrams: false
bruce[21788]: Max failed delivery attempts 5
bruce[21788]: Not running as daemon
bruce[21788]: Client ID [bruce]
bruce[21788]: Required ACKs -1
bruce[21788]: Replication timeout 10000 milliseconds
bruce[21788]: Shutdown send grace period 30000 milliseconds
bruce[21788]: Kafka dispatch restart grace period 5000 milliseconds
bruce[21788]: Metadata refresh interval 15 minutes
bruce[21788]: Kafka socket timeout 60 seconds
bruce[21788]: Pause rate limit initial 5000 milliseconds
bruce[21788]: Pause rate limit max double 4
bruce[21788]: Minimum pause delay 5000 milliseconds
bruce[21788]: Discard reporting interval 600 seconds
bruce[21788]: Debug directory [/home/bruce/debug]
bruce[21788]: Message debug time limit 3600 seconds
bruce[21788]: Message debug byte limit 2147483648
bruce[21788]: Skip comparing metadata on refresh: false
bruce[21788]: Discard logfile creation is disabled
bruce[21788]: Discard report bad msg prefix size: 256 bytes
bruce[21788]: Automatic topic creation enabled
bruce[21788]: Using new input datagram format
bruce[21788]: Using new output format
bruce[21788]: Pool block size is 128 bytes
bruce[21788]: Server started
bruce[21788]: Starting input thread
bruce[21788]: Starting router thread
bruce[21788]: Input thread 21789 started
bruce[21788]: Router thread 21790 startedbruce[21788]: Input thread opening UNIX domain datagram socket

bruce[21788]: Router thread sending initial metadata request
bruce[21788]: Input thread finished initialization, forwarding messages
bruce[21788]: Main thread detected successful input thread initializationbruce[21788]: Router thread getting metadata from broker broker_host_1 port 9092

bruce[21788]: Started web interface, waiting for signals and errors
bruce[21788]: Metadata sanity check passed
bruce[21788]: Router thread starting dispatcher during initialization
bruce[21788]: Starting connector thread for broker index 0 (Kafka ID 1)
bruce[21788]: Router thread finished initialization
bruce[21788]: Connector thread 21793 (index 0 broker 1) started
bruce[21788]: Connector thread 21793 (index 0 broker 1) connecting to host localhost port 9092
bruce[21788]: Starting pause on failure to connect to broker localhost port 9092
bruce[21788]: Connector thread 21793 (index 0 broker 1) finished on error
bruce[21788]: Router thread detected pause: waiting 5000 milliseconds before responding
bruce[21788]: All connector threads finished shutting down
bruce[21788]: Router thread shutting down dispatcher on pause
bruce[21788]: Sending fast shutdown request to connector thread (index 0 broker 1)
bruce[21788]: Waiting for shutdown ACK from connector thread (index 0 broker 1)
bruce[21788]: Got shutdown finished notification from connector thread (index 0 broker 1)
bruce[21788]: Router thread waiting for dispatcher shutdown
bruce[21788]: Start waiting for dispatcher shutdown status
bruce[21788]: Finished waiting for dispatcher shutdown status
bruce[21788]: Dispatcher terminated on error
bruce[21788]: Router thread getting metadata in response to pause
bruce[21788]: Router thread getting metadata from broker localhost port 9092
bruce[21788]: Router thread failed to connect to broker for metadata
bruce[21788]: Metadata request failed for all known brokers, waiting 5000 milliseconds before retry (1)
bruce[21788]: Router thread getting metadata from broker localhost port 9092
bruce[21788]: Router thread failed to connect to broker for metadata
bruce[21788]: Metadata request failed for all known brokers, waiting 6579 milliseconds before retry (1)

I defined broker_host_1 for connection but it tries to connect localhost why?

[dspeterson]

I assume that the section of bruce's config file specifies broker_host_1 port 9092, since bruce successfully connects to that broker and gets metadata. The metadata response gives the hostnames and ports of all brokers, which is how bruce knows which brokers to connect to for message transmission. The following log message:

bruce[21788]: Connector thread 21793 (index 0 broker 1) connecting to host localhost port 9092

suggests that according to the metadata response, one of the brokers is running on localhost port 9092 (clearly wrong), so bruce tries unsuccessfully to connect to a broker running locally. That suggests that your Kafka cluster is misconfigured. Take a look at the server.properties file for each broker and verify that everything is correct. In particular, take a look at the the settings for host.name, advertised.host.name, and advertised.port. I suspect that one or more of these settings are wrong. In my broker config, all 3 settings are commented out. For troubleshooting purposes, bruce comes with a program called mdrequest, which connects to a broker, sends a metadata request, and writes JSON to standard output showing the response. This may help. You can build and run it as follows, from the root of bruce's source tree:

./build_all -v
out/debug/bruce/kafka_proto/v0/mdrequest --broker_host broker_host_1 --broker_port 9092

Let me know if you still have problems.