can't get the "dr_cb" callback when the network is recoveryed.

[vfhky]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

When the remote kafka network is recovery, my program cannot get the "dr_cb" callback.

How to reproduce

step1. set queue.buffering.max.messages equal 2 and register my callback function to "dr_cb".
step2. set the remote kafka network being unusable by firewall.
step3. publish two messages to kafka local queue.
step4. recovery the remote kafka network.
step5. the two messages in local queue has been send to remote kafka automatically, but my program cannot get the "dr_cb" callback.
step6. publish the third message to kafka, then get the error:-184, Local: Queue full.

i think if the network is recoveryed, it will get the "dr_cb" callback to clear the two messages in local queue. Thus i will get no error when i publish the third message to kafka in step 6.

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): v1.1.0
• Apache Kafka version: <REPLACE with e.g., 0.10.2.3>
• librdkafka client configuration: queue.buffering.max.messages:2, request.required.acks:1,
• Operating system: Ubuntu 12.04 LTS
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[edenhill]

Please reproduce on the latest version v1.3.0, and provide output from debug=msg,queue,topic

[edenhill]

You should also use dr_msg_cb(), not that it matters in this case, but it is a richer interface and dr_cb() is deprecated.

Also make sure to call rd_kafka_poll() at frequent intervals to serve delivery reports.

[vfhky]

You should also use dr_msg_cb(), not that it matters in this case, but it is a richer interface and dr_cb() is deprecated.

Also make sure to call rd_kafka_poll() at frequent intervals to serve delivery reports.

thanks to your guide, and it worked.
But i get another qustion that my consumer consumered the two messges later than the callback from dr_cb about six seconds. By theory, the dr_cb callbak would be work only when the two messages has been send to the remote kafka from local queue.
I am looking forward to your answer, thanks!

[edenhill]

The consumer will be able to consume the messages as soon as they've been written to the topic log according to the acks setting, typically within a handful o milliseconds.
There is no state shared between producer and consumer.

The 6s delay is most likely because of the initial consumer group rebalance.