It appears that producer does not flush its buffer at exit

[Pehat]

Hi,
it's not actually an issue but rather a feature request. I have the following code:

#!/usr/local/bin/python3
from kafka import KafkaProducer

producer = KafkaProducer(bootstrap_servers=["10.20.30.40:5060"])
for i in range(1000):
    producer.send('topic', key=i.to_bytes(), value=i.to_bytes())
# that's all

When consumer reads the topic, it gets no messages. I've made an investigation and found that the reason is producer does not flush its message buffer before the script terminates. Appending producer.flush() to the script solves my problem, but not in all cases. For example, what if I've got an unhandled exception and flushing does not happen? Please consider handling KafkaProducer.__del__() and/or using KafkaProducer via the context (with statement).

[dpkp]

You should always run producer.close() before exiting if you want to guarantee the message queue is flushed. If you want to manage exceptions, try / except / finally is always available.

Nonetheless, the producer does flush at exit (and at __del__), it's just that it does not block (much) so messages may get dropped. see https://github.com/dpkp/kafka-python/blob/master/kafka/producer/kafka.py#L263-L267

Much preferred that you call close() directly and set an explicit timeout that you are comfortable with given the context of your application.