rdkafka_broker.c:2755:rd_kafka_fetch_reply_handle: assert failed

[Kshitij29]

Description

I get a lot of partition count change messages as follows:

`
%4|1534226113.072|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [14] is unknown (partition_cnt 6)

%4|1534226113.072|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [13] is unknown (partition_cnt 6)

%4|1534226113.072|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [12] is unknown (partition_cnt 6)

%5|1534226114.589|PARTCNT|rdkafka#consumer-9| [thrd:main]: Topic abc.com partition count changed from 6 to 15

%5|1534226121.090|PARTCNT|rdkafka#consumer-9| [thrd:main]: Topic abc.com partition count changed from 15 to 6

%4|1534226121.090|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [14] is unknown (partition_cnt 6)

%4|1534226121.090|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [13] is unknown (partition_cnt 6)

%4|1534226121.090|LEADER|rdkafka#consumer-9| [thrd:main]: abc.com [12] is unknown (partition_cnt 6)

%5|1534226121.177|PARTCNT|rdkafka#consumer-9| [thrd:main]: Topic abc.com partition count changed from 6 to 15
`
However using kafka-topics.sh, I get that partition count is constant at 15.

After these partition count change messages, my daemon crashes with the following error:
*** rdkafka_broker.c:2755:rd_kafka_fetch_reply_handle: assert: tver && rd_kafka_toppar_s2i(tver->s_rktp) == rktp *** Abort trap: 6

How to reproduce

Created 10 consumers in 10 separate threads using github.com/confluentinc/confluent-kafka-go .
All consumer threads are created by a main thread. All subscribe to same topic (here "abc.com"). Seeing the issue as soon as the main thread starts.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): v0.11.4
• Apache Kafka version: v0.10.1.0
• librdkafka client configuration: "group.id": abc.com, "auto.offset.reset": "earliest", "auto.commit.interval.ms": 5000, "api.version.request": "false", "broker.version.fallback": 0.10.1.0, "broker.address.family": "v4", "queued.max.messages.kbytes": 100,
• Operating system: macOS Sierra 10.12.6
• Provide logs (with debug=.. as necessary) from librdkafka: In Description
• Provide broker log excerpts: NA
• Critical issue: yes

[edenhill]

It seems like your cluster is desynchronized, different brokers returning different partition counts for a topic.

librdkafka queries the brokers for topic metadata directly, while kafka-topics.sh queries zookeeper. A client should only query the brokers and not zookeeper.

As for the crash; there are some known problems when the partition count decreases (which should never happen in a healthy cluster), and this crash is most likely related to that. We'll look into fixing the crash but please do try to fix your cluster.

[Kshitij29]

Thanks for the prompt reply @edenhill . I'll check if the cluster is properly synchronised. However, if this desynchronisation is persistent in our cluster due to some reason, I'll wait for your fix.

[edenhill]

Fixed on master

[Kshitij29]

Thanks for the solution @edenhill . The application does not crash now even with out-of-sync brokers.

[edenhill]

Thank you!