in producer, max_block_ms should larger than request_timeout_ms by default

[hicqu]

Suppose we have a kafka producer initialized with default arguments, 60s as max_block_ms and 30s as request_timeout_ms. Then we run a loop:

while True:
    producer.send(topic, value)

After we kill all processes in kafka-cluster, I want the producer raise an exception KafkaTimeoutError.

but when max_block_ms > request_timeout_ms, the exception will never be raised out.

Only when max_block_ms < request_timeout_ms, Exception will raised:

KafkaTimeoutError: Failed to allocate memory within the configured max blocking time

My analyse:

when kafka producer try to send, it will block at most request_timeout_ms, after that it will drop those messages and clean send buffer. So, the producer can put new messages into the send buffer. The result is we can't receive any notice but our messages are droped exactly.

So, I think request_timeout_ms should never larger than max_block_ms. What do you think?

[dpkp]

Take a look at #986 -- can you provide a PR implementing the same checks for max_block_ms ?

[dpkp]

I don't agree with your analysis. max_block_ms is designed primarily to constrain the time that the 'send' method may block (blocking may be caused by memory allocation issues and/or metadata refresh requirements). The request_timeout_ms on the other hand is a constraint on how long the core IO system will wait for a response from a server. If you have a small max_block_ms you may get an exception raised by producer.send while, for example, metadata is requested + loaded. But this is ok because the metadata is refreshing asynchronously via a background thread, so you simply retry producer.send later. The point is that you can control how long to block on producer.send if you are in a time-sensitive path like a web-request and would rather drop messages than increase response latency.