librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14

zbweng · 2013-07-25T12:24:41Z

Hi, Magnus.
I find that librdkafka 0.8 can not produce messages to the kafka brokers when the leader of the topic partition failed. After I check the new leader is elected, the problem is not yet solved.
I use the rdkafka_example.c in the example folder and create a topic with 3 replicas and 1 partition.
After I restart the producer (rdkafka_example.c, I use it as a producer), the problem is solved.
I am not sure whether it is a bug for librdkafka 0.8.
Thanks.

edenhill · 2013-08-19T22:17:30Z

I will push a fix for this soon.

)

zbweng · 2013-08-21T03:13:40Z

Hi, Magnus.
I test for this issue on your latest code again. However, I find that this issue is not yet solved.
I use pkill to stop the leader of a topic partition, then rdkafka cannot produce messages.

edenhill · 2013-08-21T16:43:25Z

You are right.
librdkafka polls for topic leader as soon as a produce request for a topic fails, but the new leader has probably not been elected yet.
I'll add a timer to retry the poll until a leader is found.

Config property: Config property: "topic.metadata.refresh.fast.cnt" (def 10) Config property: "topic.metadata.refresh.fast.interval.ms" (def 250ms)v

edenhill · 2013-08-26T20:37:49Z

Give it another shot now on master branch.
It now regularily refreshes the broker list (metadata), and after a leader failure it aggressively refreshes it 10 times with a 250ms interval (default config).

zbweng · 2013-08-31T10:06:41Z

Great! I test for this issue again and find that it is solved.

mazharshaikh86 · 2015-12-03T07:41:06Z

Hi All,
Can I find all these fixed on latest 0.8. 6 release?

"https://github.com/edenhill/librdkafka/releases/tag/0.8.6"

Regards,
Mazhar Shaikh.

edenhill · 2015-12-03T07:47:27Z

The release notes indicate what has been fixed, yes. or what are you asking?

mazharshaikh86 · 2015-12-03T13:44:18Z

I'm using "librdkafka_2.10-0.8.2.1" and im facing the same problem.

Cannot find info on this bug in release notes.

I want to know that, if I use "0.8.6" version, then will this solve problem.

Below are few logs generated by stub code.

geo redundent kafka process (zookeeper + Broker)

Machine 1 : advertised.host.name=sysctrl1.vsepx.broker.com
Machine 2 : advertised.host.name=sysctrl2.vsepx.broker.com

% Sent 3 bytes to topic topic1 partition -1, outQlen[94]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Request metadata for all topics: periodic refresh]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Requesting metadata for all topics]
log_cb : rdkafka#producer-0: [7] [SEND] [sysctrl2.vsepx.broker.com:9092/bootstrap: Sent MetadataRequest (25 bytes, CorrId 8)]
log_cb : rdkafka#producer-0: [7] [SEND] [sysctrl2.vsepx.broker.com:9092/bootstrap: Sent 1 bufs]
log_cb : rdkafka#producer-0: [7] [SEND] [sysctrl2.vsepx.broker.com:9092/bootstrap: Sent 0 bufs]
log_cb : rdkafka#producer-0: [7] [RECV] [sysctrl2.vsepx.broker.com:9092/bootstrap: Received MetadataResponse (8 bytes, CorrId 8, rtt 2.91ms)]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: ===== Received metadata from sysctrl2.vsepx.broker.com:9092/bootstrap =====]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: 0 brokers, 0 topics]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: couldn't connect to ipv4#166.45.146.65:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [sysctrl1.vsepx.broker.com:9092/0: failed: err: Local: Communication failure with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/0: Purging bufq with 0 buffers]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/bootstrap: broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/bootstrap: couldn't connect to ipv4#166.45.146.65:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [sysctrl1.vsepx.broker.com:9092/bootstrap: failed: err: Local: Communication failure with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/bootstrap: Purging bufq with 0 buffers]

edenhill · 2015-12-03T13:50:47Z

There is no such librdkafka version as "librdkafka_2.10-0.8.2.1". That looks more like a kafka version.

It fails to TCP connect to broker sysctrl1.vsepx.broker.com with address 166.45.146.65 on port 9092.
This is not a problem within librdkafka but rather with your network or on the broker.
You can try to connect manually with telnet to the broker from the host running librdkafka: telnet 166.45.146.65 9092, that will most likely also fail.

Check:

iptables / firewall rules. Is your host allowed to access the broker
Is the broker running on the server and on that port
network connectivity

mazharshaikh86 · 2015-12-03T14:01:46Z

I'm using "kafka_2.10-0.8.2.1.tgz" for my C Project which has is needed to support geo redundant kafka process (zookeeper + Broker).

Machine 1 : Producer1 : Broker IP"sysctrl1.vsepx.broker.com:9092,sysctrl2.vsepx.broker.com:9092"

Machine 2 : Broker1 : advertised.host.name=sysctrl1.vsepx.broker.com
Machine 3 : Broker2 : advertised.host.name=sysctrl2.vsepx.broker.com

Scenario:
1. Broker1 is UP, Broker 2 is Down.
2. Producer1 is sending messages to Broker1 successfully
3. Broker1 goes down.
4. Broker2 comes UP
5. Producer1 connection with broker 1 fails, But Connection with Broker2 is established.
6. Producer1 exchanges meta data with Broker2, But messages are not being sent to Broker2, Instead the producer1 is still trying to send the same to Broker1 (which is down).
7. All Messages Fail.

Now, If I kill and restart Producer1, then all the new messages are succesfully sent to Broker2.

mazharshaikh86 · 2015-12-03T14:07:04Z

./rdkafka_example

Usage: ./rdkafka_example -C|-P|-L -t [-p ] [-b host1:port1,host2:port2,..]

librdkafka version 0.8.6 (0x00080600)

edenhill · 2015-12-03T14:09:40Z

log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: ===== Received metadata from sysctrl2.vsepx.broker.com:9092/bootstrap =====]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: 0 brokers, 0 topics]

Broker sysctrl2 is returning 0 brokers, 0 topics, in the metadata reply.
That is weird. Can you check if there are any exceptions in the broker 2 log around the time of this event?

mazharshaikh86 · 2015-12-03T14:29:56Z

Broker2 Logs :

[2015-12-03 09:28:23,842] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2015-12-03 09:28:24,052] ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 2220; ClientId: rdkafka; Topics: topic1 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:520)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:503)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:503)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:542)
at kafka.server.KafkaApis.handle(KafkaApis.scala:62)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:701)
[2015-12-03 09:28:24,052] ERROR [KafkaApi-0] error when handling request Name: TopicMetadataRequest; Version: 0; CorrelationId: 1558; ClientId: rdkafka; Topics: topic1 (kafka.server.KafkaApis)
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:171)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:520)
at kafka.server.KafkaApis$$anonfun$19.apply(KafkaApis.scala:503)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:47)
at scala.collection.SetLike$class.map(SetLike.scala:93)
at scala.collection.AbstractSet.map(Set.scala:47)
at kafka.server.KafkaApis.getTopicMetadata(KafkaApis.scala:503)
at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:542)
at kafka.server.KafkaApis.handle(KafkaApis.scala:62)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:701)
[2015-12-03 09:28:24,573] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2015-12-03 09:28:24,579] INFO Registered broker 0 at path /brokers/ids/0 with address sysctrl2.vsepx.broker.com:9092. (kafka.utils.ZkUtils$)
[2015-12-03 09:28:24,593] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-12-03 09:28:24,718] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],topic1,14
[2015-12-03 09:28:24,982] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],topic1,14

edenhill · 2015-12-03T14:31:37Z

There's your error:
kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0

mazharshaikh86 · 2015-12-03T14:31:49Z

librdkafka logs (repeat2)

% Sent 3 bytes to topic topic1 partition -1, outQlen[127]
log_cb : rdkafka#producer-0: [7] [RECV] [sysctrl2.vsepx.broker.com:9092/bootstrap: Received MetadataResponse (473 bytes, CorrId 12, rtt 142.35ms)]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: ===== Received metadata from sysctrl2.vsepx.broker.com:9092/bootstrap =====]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: 1 brokers, 1 topics]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Broker #0/1: sysctrl2.vsepx.broker.com:9092 NodeId 0]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic #0/1: topic1 with 16 partitions]
log_cb : rdkafka#producer-0: [7] [PARTCNT] [No change in partition count for topic topic1]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 8 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [8] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 11 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [11] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 2 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [2] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 5 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [5] with 8 messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 14 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [14] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 4 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [4] with 8 messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 13 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [13] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 7 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [7] with 7 messages (17 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 1 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [1] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 10 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [10] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 9 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [9] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 3 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [3] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 12 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [12] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 15 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [15] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 6 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [6] with 8 messages (20 bytes) queued]
log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 0 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [0] with 8 messages (19 bytes) queued]
log_cb : rdkafka#producer-0: [7] [PARTCNT] [Partitioning 0 unassigned messages in topic topic1 to 16 partitions]
log_cb : rdkafka#producer-0: [7] [UAS] [0/0 messages were partitioned in topic topic1]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/bootstrap: broker in state DOWN connecting]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/bootstrap: couldn't connect to ipv4#192.168.108.140:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [sysctrl1.vsepx.broker.com:9092/bootstrap: failed: err: Local: Communication failure with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/bootstrap: Purging bufq with 0 buffers]
log_cb : rdkafka#producer-0: [7] [CONNECT] [sysctrl1.vsepx.broker.com:9092/0: couldn't connect to ipv4#192.168.108.140:9092: Connection refused]
log_cb : rdkafka#producer-0: [7] [BROKERFAIL] [sysctrl1.vsepx.broker.com:9092/0: failed: err: Local: Communication failure with broker: (errno: Connection refused)]
log_cb : rdkafka#producer-0: [7] [BUFQ] [sysctrl1.vsepx.broker.com:9092/0: Purging bufq with 0 buffers]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [8]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [11]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [2]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [5]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [14]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [4]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [13]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [7]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [1]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [10]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [9]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [3]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [12]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [15]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [6]]
log_cb : rdkafka#producer-0: [7] [BRKTP] [sysctrl1.vsepx.broker.com:9092/0: Undelegating topic1 [0]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 no longer leader for topic topic1 [0]]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [No broker is leader for topic topic1 [0]]

mazharshaikh86 · 2015-12-03T14:32:23Z

broker logs contd:

[2015-12-03 09:28:24,573] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2015-12-03 09:28:24,579] INFO Registered broker 0 at path /brokers/ids/0 with address sysctrl2.vsepx.broker.com:9092. (kafka.utils.ZkUtils$)
[2015-12-03 09:28:24,593] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2015-12-03 09:28:24,718] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],topic1,14
[2015-12-03 09:28:24,982] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [topic1,1],[topic1,0],[topic1,10],[topic1,15],[topic1,9],[topic1,4],[topic1,8],[topic1,3],[topic1,13],[topic1,12],[topic1,2],[topic1,6],[topic1,7],[topic1,5],[topic1,11],topic1,14
[2015-12-03 09:30:10,195] INFO Closing socket connection to /12.1.1.81. (kafka.network.Processor)

mazharshaikh86 · 2015-12-03T14:35:45Z

From producer1 logs :

log_cb : rdkafka#producer-0: [7] [METADATA] [sysctrl2.vsepx.broker.com:9092/bootstrap: Topic topic1 partition 11 Leader 0]
log_cb : rdkafka#producer-0: [7] [BRKDELGT] [Broker sysctrl1.vsepx.broker.com:9092/0 is now leader for topic topic1 [11] with 8 messages (19 bytes) queued]

Here, "sysctrl2.." is the leader, but in the next log , "sysctrl1.." is shown as leader

edenhill · 2015-12-03T15:34:07Z

The last log says that broker 0 is leader for partition 11

mazharshaikh86 · 2015-12-04T07:04:01Z

broker.id=0;
broker.id is same in broker1 and broker2 (server.properties). Only change is in "advertised.host.name"

mazharshaikh86 · 2015-12-04T09:33:22Z

Even if i use 1 broker. Same issue is observed.

Steps:

Start Broker1 with "advertised.host.name=sysctrl1"
Start load with producer1, (Producer was successful in sending data to broker)
Stop broker1 (let producer be running)
Change Broker1 advertised.host.name to "sysctrl2". ("advertised.host.name=sysctrl2")
Start Broker1 again.
Notice that producer1 is not able to send data to broker. (this is the issue)
Restart Producer1, and data flow is normal again.

hostnames sysctrl1 & sysctrl2 resolves to same IP.

Here only the "advertised.host.name" is changed, and producer is not able to handle the same.

Please comment on the scenario.

Thank you.

edenhill · 2015-12-04T09:34:38Z

In your example you have the same hostname for both step 1 and 4, did you mean sysctrl2 on step 4?

mazharshaikh86 · 2015-12-04T09:36:28Z

yes sir. sorry for mistake.
I changed the hostname to sysctrl2. (on step4)

edenhill · 2015-12-04T09:41:39Z

Okay, this is quite an odd thing to do and I dont understand why you are doing it.

But what this looks like to the client is:

sees broker 0 hostname sysctrl1
broker 0 disconnects
sees broker 0 hostname sysctrl2

librdkafka 0.8.6 does not support broker hostname updates, but the master branch does (at least somewhat).

Which version of librdkafka are you using? (if you checked out from git please specify the exact git sha by git log | head -1

mazharshaikh86 · 2015-12-04T10:04:10Z

I am using only one broker (no cluster).
But when the machine hosting the broker goes down.
Another machine at different location (geo redundent) will be started, hence the IP & Hostname will change.

Here, as kafka consumer is dependent on offsets, we need the same "log.dirs" files to restore all the data.

I have downloaded librdkafka from the below location
https://github.com/edenhill/librdkafka/releases/tag/0.8.6

Please let me know if there is an alternate way to achive this model.

edenhill · 2015-12-04T10:05:17Z

You should try using the latest master, it should support broker name changes.

Regarding your geo setup, that is a much bigger discussion which is not really related to librdkafka, so I can't help you there.

mazharshaikh86 · 2015-12-04T10:08:49Z

Thank you Magnus :)

I will get the latest master and give a try.

ghost assigned edenhill Jul 30, 2013

edenhill added a commit that referenced this issue Aug 20, 2013

Re-delegate all affected topic+partitions when a broker fails (issue #14

edf5336

)

edenhill closed this as completed Aug 20, 2013

edenhill reopened this Aug 21, 2013

edenhill added a commit that referenced this issue Aug 26, 2013

Proper broker metadata refresh on leader failure. (issue #14)

c12ceab

Config property: Config property: "topic.metadata.refresh.fast.cnt" (def 10) Config property: "topic.metadata.refresh.fast.interval.ms" (def 250ms)v

zbweng closed this as completed Aug 31, 2013

xiaojun789 mentioned this issue Nov 8, 2013

crashes on too large messages #24

Closed

winbatch mentioned this issue Dec 27, 2013

static linking of librdkafka.a needs dynamic pthreads linking #34

Closed

winbatch mentioned this issue Feb 11, 2014

Crash on exit #72

Closed

affranchi mentioned this issue Mar 26, 2015

Seg fault with multi brokers #224

Closed

DEvil0000 mentioned this issue Oct 12, 2015

rdkafka stopping connection retries on RdKafka::ERR__ALL_BROKERS_DOWN #373

Closed

edenhill added the question label Dec 4, 2015

ago3x mentioned this issue Jan 6, 2016

autocommit offset does not work if i use high level api #480

Closed

trthulhu mentioned this issue Feb 5, 2016

rdkafka crashes when topic is too long (255 characters) sometimes #529

Closed

wucanhui mentioned this issue Feb 22, 2016

rdkafka_partition.c:165:rd_kafka_toppar_remove: assert: rktp->rktp_removed==0 #558

Closed

hendrik-schumacher mentioned this issue Apr 21, 2016

librdkafka hanging on rd_kafka_destroy() #624

Closed

mlala-pan mentioned this issue Aug 5, 2016

Consuming from multiple topics partitions from a single thread causing an assert #748

Closed

trthulhu mentioned this issue Aug 18, 2016

hang on rd_kafka_destroy() in repetitive, quick succession #747

Closed

9 tasks

trthulhu mentioned this issue Sep 7, 2016

Core rd_kafka_buf_grow assertion failure during rapid usage of consumer #781

Closed

9 tasks

longtong mentioned this issue Sep 27, 2016

legacy consumer coredump on 0.8.6 #801

Closed

9 tasks

greatt1n mentioned this issue Jan 16, 2017

Hangs on rd_kafka_flush #999

Closed

benli123 mentioned this issue Mar 17, 2017

librdkafka crashes due to seemingly racing condition #1125

Closed

9 tasks

ameyapg mentioned this issue May 2, 2018

Crash observed consistently while calling begin() ( seems to have an invalid/corrupted iterator) on the vector of pointers to partition metadata of one of the topics #1790

Closed

7 tasks

hulaxiaodai mentioned this issue Nov 7, 2018

【producer】deadlock on metadata requery #2092

Closed

4 tasks

matthrae mentioned this issue Nov 8, 2018

Assertion `!*"refcnt sub-zero"' after reconnect #2088

Closed

yexinxing mentioned this issue Mar 8, 2019

program crashed for rd_kafka_tppar_t was used after destroy #1846

Closed

7 tasks

pradeepp2019 mentioned this issue Apr 23, 2019

Query on Async programming (Seastar) with librdkafka #2292

Closed

MockingJayWong mentioned this issue Jul 9, 2019

rd_kafka_topic_new after using admin api #2399

Closed

7 tasks

hyhtemple mentioned this issue Oct 14, 2019

use the produce of the librdkafka c++ interface ,the program report "Program terminated with signal 6, Aborted" #2571

Closed

aelanthi mentioned this issue Dec 16, 2019

confluent python kafka client still crashes after 1.0.1 upgrade #2658

Closed

7 tasks

hyhtemple mentioned this issue Jun 30, 2020

It is crashed ,when use method of the producer in the thread #2947

Closed

dshivashankar1994 mentioned this issue Jul 6, 2020

rd_refcnt_sub0: Assertion `!*"refcnt sub-zero"' failed - when the subscribed topic is deleted #2963

Closed

Furuta-Masakazu-quick mentioned this issue Sep 29, 2020

Producer detects "All broker connections are down" when rolling restarts #3090

Closed

7 tasks

taj-uddin mentioned this issue Apr 15, 2021

High CPU usage in librdkafka1.4.0 producer #2986

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14

librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14

zbweng commented Jul 25, 2013

edenhill commented Aug 19, 2013

zbweng commented Aug 21, 2013

edenhill commented Aug 21, 2013

edenhill commented Aug 26, 2013

zbweng commented Aug 31, 2013

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14

librdkafka 0.8 can not produce messages when the leader of the topic partition failed #14

Comments

zbweng commented Jul 25, 2013

edenhill commented Aug 19, 2013

zbweng commented Aug 21, 2013

edenhill commented Aug 21, 2013

edenhill commented Aug 26, 2013

zbweng commented Aug 31, 2013

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

./rdkafka_example

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

mazharshaikh86 commented Dec 3, 2015

edenhill commented Dec 3, 2015

mazharshaikh86 commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015

edenhill commented Dec 4, 2015

mazharshaikh86 commented Dec 4, 2015