producer can't send message to the topic #85

geekhuyang · 2016-03-25T09:25:15Z

I set up the kafka cluster according to the instruction,but when I send the message to the created topic,
the error comes

yqian1991 · 2016-04-11T02:57:00Z

Hmm, I had the same issue, if you check docker-compose logs, I guess the leader is set to docker-machine ip : 9092, which caused this problem. maybe the problem of docker-compose.yml

geekhuyang · 2016-04-26T03:01:41Z

@yqian1991 Thanks. your meaning is the var KAFKA_ADVERTISED_HOST_NAME in docker-compose should not be the docker host ip?

geekhuyang · 2016-04-26T15:23:40Z

I think the ./broker-list.sh result is no correct,but I have no idea about the reason.

@wurstmeister In this case,I should set the KAFKA_ADVERTISED_HOST_NAME value is which above?
thanks

pluswave · 2016-05-11T03:25:51Z

same issue on my side. I guess it should be some error with specific combination of host os / docker version / kafka-docker version. So I try to describe this and the reproduce method as detail as possible here.

Host os is Centos 7, Docker version 1.10.3, build 20f81dd, java version 1.8.0 (openjdk), kafka-docker is latest git ( f32ebdc ) . the change is

~ $ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 73dd854..4f81af8 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -9,6 +9,6 @@ kafka:
   links: 
     - zookeeper:zk
   environment:
-    KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103
+    KAFKA_ADVERTISED_HOST_NAME: 192.168.1.100
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock

the ip address 192.168.1.100 is the host address.

step to reproduce:

docker-compose up -d
install kafka 0.9.0.1 locally by download and install tarball from here and unzip it.
cd kafka_2.11-0.9.0.1

docker ps to find port forwarding for kafka and zookeeper. in my case

 CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                       PORTS                                                 NAMES
9352fee6defe        kafkadocker_kafka        "start-kafka.sh"         17 hours ago        Up 17 hours                0.0.0.0:32785->9092/tcp                               kafkadocker_kafka_1
2a7851f7af61        wurstmeister/zookeeper   "/bin/sh -c '/usr/sbi"   17 hours ago        Up 17 hours                22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:32784->2181/tcp   kafkadocker_zookeeper_1

create topic:

~$ bin/kafka-topics.sh --create --topic test --zookeeper 192.168.1.100:32784 --partitions 1 --replication-factor 1

show topic info:

~$ bin/kafka-topics.sh --describe  --zookeeper 192.168.1.100:32784
Topic:test PartitionCount:1    ReplicationFactor:1 Configs:
  Topic: test Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

produce some message:

$ bin/kafka-console-producer.sh --broker-list="192.168.1.100:32785" --topic test
abcd
[2016-05-11 11:21:42,527] WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:42,625] WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:42,727] WARN Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:42,829] WARN Error while fetching metadata with correlation id 3 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:42,931] WARN Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,033] WARN Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,135] WARN Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,237] WARN Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,339] WARN Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,441] WARN Error while fetching metadata with correlation id 9 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,543] WARN Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,645] WARN Error while fetching metadata with correlation id 11 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,747] WARN Error while fetching metadata with correlation id 12 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,849] WARN Error while fetching metadata with correlation id 13 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:43,951] WARN Error while fetching metadata with correlation id 14 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:44,053] WARN Error while fetching metadata with correlation id 15 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:44,155] WARN Error while fetching metadata with correlation id 16 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:44,257] WARN Error while fetching metadata with correlation id 17 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:44,359] WARN Error while fetching metadata with correlation id 18 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2016-05-11 11:21:44,462] WARN Error while fetching metadata with correlation id 19 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

more debug steps:

port detection

telnet 192.168.1.100 32785
Trying 192.168.1.100...
Connected to 192.168.1.100.
Escape character is '^]'.


Connection closed by foreign host.

zookeeper client:

ls /
[consumers, config, controller, isr_change_notification, admin, brokers, zookeeper, controller_epoch]
ls /brokers 
[seqid, topics, ids]
ls /brokers/ids
[1001]
get /brokers/ids/1001
{"jmx_port":-1,"timestamp":"1462872416451","endpoints":["PLAINTEXT://192.168.1.100:32785"],"host":"192.168.1.100","version":2,"port":32785}
cZxid = 0x18
ctime = Tue May 10 17:26:56 CST 2016
mZxid = 0x18
mtime = Tue May 10 17:26:56 CST 2016
pZxid = 0x18
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15499fd99150000
dataLength = 139
numChildren = 0
ls /config 
[topics, clients, changes]
ls /config/topics
[test]
get /config/topics/test
{"version":1,"config":{}}
cZxid = 0x1e
ctime = Tue May 10 17:30:00 CST 2016
mZxid = 0x1e
mtime = Tue May 10 17:30:00 CST 2016
pZxid = 0x1e
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 25
numChildren = 0

pluswave · 2016-05-11T09:05:14Z

some clue about this issue for my case:

inside the kafka server container (please note I just restart the service so the port mapping changed):

docker exec -it kafkadocker_kafka_1 bash
bash-4.3# netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 :::33304                :::*                    LISTEN      103/java
tcp        0      0 :::9092                 :::*                    LISTEN      103/java
tcp        0      1 ::ffff:172.17.0.8:46190 ::ffff:192.168.1.100:32791 SYN_SENT    103/java
tcp        0      0 ::ffff:172.17.0.8:57736 ::ffff:172.17.0.7:2181  ESTABLISHED 103/java

the SYN_SENT indicate the cause (but not root cause). from the kafka server (docker instance), it can't connect to DockerHost:ForwardPort to any kafka server in cluster. Instead, it can only connect to some BridgedIp:9092 but it is NOT advertised by zookeeper. However from outside world, the bridged network is not visible but only fowarded port and docker host.

The problem now is : how to fix it ?

pluswave · 2016-05-17T02:15:14Z

ok should be this issue: moby/moby#16137

wurstmeister · 2016-06-04T17:29:33Z

Hi,

sorry for the late reply, this could indeed be a firewall issue. sounds very similar to #30

@madfrog2047 the output of broker-list.sh depends on how you started start-kafka-shell.sh, which is where it is supposed to be used (see https://github.com/wurstmeister/kafka-docker/blob/master/broker-list.sh#L4)

bikeholik · 2016-06-23T06:39:46Z

I've got it working disabling firewalld completely (on Fedora 23) for test purposes...

lidel · 2016-07-26T10:50:36Z

@madfrog2047 Just out of curiosity, did you set a (custom) low tickTime in ZooKeeper by any chance? I had the same error with tickTime=10 and it is gone with tickTime=500 (embedded Kafka+ZooKeeper in JUnits).

BenoitPotty · 2016-08-19T12:55:43Z

Hi Guys,

Had the same issue as you, and disabling the Firewall worked fine for me. Did anyone went further on searching which rule we should add to Firewall and be able to leave it active?

Thanks a lot.

junjiemars · 2017-03-15T02:25:39Z

set advertised.host.name=<container's-ip> in Kafka's server.properties solve my problem.

Hubbitus · 2017-09-05T22:00:07Z

It seams related to moby/moby#26594

As workaround you may use HOSTNAME_COMMAND instead of providing IP statically.
So f.e. If I add in environment section of docker-compose.yml file line:

HOSTNAME_COMMAND: "getent hosts `hostname` | cut -d' ' -f1"

it works for me.

ppromsiri · 2018-01-22T09:37:48Z

Hi All

I found workaround
add KAFKA_ADVERTISED_LISTENERS:
in section environment docker-compose.yml

it works. Let's try

jeven2016 · 2018-01-31T08:49:10Z

yes, the KAFKA_ADVERTISED_LISTENERS is very useful, and following is my yml file for reference:

version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper
ports:
- "5804:2181"
kafka:
image: wurstmeister/kafka
ports:
- "5805:9092"
environment:
KAFKA_ADVERTISED_LISTENERS: 10.196.10.6:9092
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock

dustinfarris · 2018-02-20T15:37:51Z

@jeven2016 where did that IP address come from? "10.196.10.6"

sscaling · 2018-03-29T07:45:16Z

Documentation has been added in #291.

Wiki page: https://github.com/wurstmeister/kafka-docker/wiki/Connectivity

vvscher · 2018-06-21T09:13:46Z

I assume it's the issue of the kafka itself.

I've fixed it by:

mapping "kafka" (host name of the kafka inside the docker) to 127.0.0.1 in hosts file (for some reason kafka doesn't always work properly when I call it by the host different from advertised).
and publishing a message to this topic from console producer (to auto create the topic) before the service tries to publish anything.

I cannot call it a decent fix, but it definitely works

LuisIvanMD · 2019-06-02T03:02:54Z

Hi Guys,

Had the same issue as you, and disabling the Firewall worked fine for me. Did anyone went further on searching which rule we should add to Firewall and be able to leave it active?

Thanks a lot.

Thar works for me

JOJO0527 · 2020-06-13T03:00:09Z

I've got it working disabling firewalld completely (on Fedora 23) for test purposes...

it works for me too! thanks

ynqa mentioned this issue Sep 26, 2016

[question] kafka cannot consume any messages just after established #127

Closed

sscaling closed this as completed Mar 29, 2018

kilianpg mentioned this issue May 15, 2018

BUG: org.apache.kafka.clients.NetworkClient : Error while fetching metadata with correlation id 0 : {springCloudBusOutput=LEADER_NOT_AVAILABLE} Arquisoft/InciManager_e3b#54

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

producer can't send message to the topic #85

producer can't send message to the topic #85

geekhuyang commented Mar 25, 2016

yqian1991 commented Apr 11, 2016

geekhuyang commented Apr 26, 2016 •

edited

Loading

geekhuyang commented Apr 26, 2016 •

edited

Loading

pluswave commented May 11, 2016 •

edited

Loading

pluswave commented May 11, 2016 •

edited

Loading

pluswave commented May 17, 2016

wurstmeister commented Jun 4, 2016

bikeholik commented Jun 23, 2016

lidel commented Jul 26, 2016 •

edited

Loading

BenoitPotty commented Aug 19, 2016

junjiemars commented Mar 15, 2017

Hubbitus commented Sep 5, 2017

ppromsiri commented Jan 22, 2018

jeven2016 commented Jan 31, 2018

dustinfarris commented Feb 20, 2018

sscaling commented Mar 29, 2018

vvscher commented Jun 21, 2018 •

edited

Loading

LuisIvanMD commented Jun 2, 2019

JOJO0527 commented Jun 13, 2020

producer can't send message to the topic #85

producer can't send message to the topic #85

Comments

geekhuyang commented Mar 25, 2016

yqian1991 commented Apr 11, 2016

geekhuyang commented Apr 26, 2016 • edited Loading

geekhuyang commented Apr 26, 2016 • edited Loading

pluswave commented May 11, 2016 • edited Loading

pluswave commented May 11, 2016 • edited Loading

pluswave commented May 17, 2016

wurstmeister commented Jun 4, 2016

bikeholik commented Jun 23, 2016

lidel commented Jul 26, 2016 • edited Loading

BenoitPotty commented Aug 19, 2016

junjiemars commented Mar 15, 2017

Hubbitus commented Sep 5, 2017

ppromsiri commented Jan 22, 2018

jeven2016 commented Jan 31, 2018

dustinfarris commented Feb 20, 2018

sscaling commented Mar 29, 2018

vvscher commented Jun 21, 2018 • edited Loading

LuisIvanMD commented Jun 2, 2019

JOJO0527 commented Jun 13, 2020

geekhuyang commented Apr 26, 2016 •

edited

Loading

geekhuyang commented Apr 26, 2016 •

edited

Loading

pluswave commented May 11, 2016 •

edited

Loading

pluswave commented May 11, 2016 •

edited

Loading

lidel commented Jul 26, 2016 •

edited

Loading

vvscher commented Jun 21, 2018 •

edited

Loading