Difference between isr info from librdkafka vs kafka-list-topic.sh #147
Comments
|
could you try my tool with |
|
|
Oh, it doesnt output ISRs, nevermind. So either the cluster <-> ZK state is desynced, or there is a handling error in rdkafka. |
|
Do you know if the broker asks Zk for the info or has its own cache? How do you re sync? Bounce the brokers to recache from Zk? |
|
1: dont know, but probably cached |
|
Oh no! I've stumped the master! Now what am I to do?? :( |
|
Do the tcpdump dance!
|
|
This was reported from another user (who provided a tcpdump!). |
|
Was that user able to figure out how to re sync? On Thursday, September 18, 2014, Magnus Edenhill notifications@github.com
|
|
Not yet, working on root-causing it. |
|
I noticed that in my case, only 1 of the 4 brokers was ever missing in the ISRs reported by Kafka Brokers (via librdkafka). That JIRA indicated that a preferred-replica-election should fix the problem. I did this: controlled-shutdown of offending broker 21. Then actual shutdown of broker 21. Once this was done, librdkafka metadata showed the correct ISRs, since this offending broker really was not in any ISRs. I then restarted broker 21 and let its replicas catch back up. Once it caught up, zookeeper reported that all ISRs were in sync. I then checked librdkafka's metadata, and broker 21 was not listed in any ISR. I then ran a preferred-replica-election. broker 21 was then promoted to leader of some partitions. librdkafka then only showed broker 21 being in the ISRs for which it was also the leader. Any partition that has a replica on broker 21 does not show up in the ISR unless broker 21 is the leader. |
|
@edenhill or @winbatch, want to comment on the most recent post here? Specifically: |
|
Done! |
Hi, we've been noticing that we're getting different results between the 2 tools. The difference always seems to be that rdkafka shows fewer in sync than the kafka provided tool (never the reverse). One difference is that rdkafka goes to the brokers for this while the kafka provided tool goes to zookeeper. Could this indicate some sort of mismatch between the 2?
kafka-list-topic.sh --zookeeper ${ZOOKEEPER_CLUSTER}:${ZOOKEEPER_PORT}
topic: ESM_METRICS_NETWORK partition: 0 leader: 1052189105 replicas: 1052189116,1052189105,1052189106 isr: 1052189105,1052189106,1052189116
topic: ESM_METRICS_NETWORK partition: 1 leader: 1052189106 replicas: 1052189117,1052189106,1052189116 isr: 1052189106,1052189117,1052189116
topic: ESM_METRICS_NETWORK partition: 2 leader: 1052189117 replicas: 1052189104,1052189116,1052189117 isr: 1052189117,1052189104,1052189116
topic: ESM_METRICS_NETWORK partition: 3 leader: 1052189105 replicas: 1052189105,1052189117,1052189104 isr: 1052189105,1052189104,1052189117
topic: ESM_METRICS_NETWORK partition: 4 leader: 1052189106 replicas: 1052189106,1052189104,1052189105 isr: 1052189106,1052189104,1052189105
Magnus tool:
rdkafka_example -L -b kafkalog1.messaging.nimbus.masked.com:5757 -t ESM_METRICS_NETWORK
Metadata for ESM_METRICS_NETWORK (from broker -1: kafkalog1.messaging.nimbus.masked.com:5757/bootstrap):
5 brokers:
broker 1052189104 at d146537-023.dc.masked.com:5757
broker 1052189117 at d146537-025.dc.masked.com:5757
broker 1052189105 at d146537-021.dc.masked.com:5757
broker 1052189116 at d146537-024.dc.masked.com:5757
broker 1052189106 at d146537-022.dc.masked.com:5757
1 topics:
topic "ESM_METRICS_NETWORK" with 5 partitions:
The text was updated successfully, but these errors were encountered: