Producer OOM precaution

[ZhuYouzhi]

Hi, Edenhill.

In current librdkafka, if producer produce data at a rate faster than broker can handle, then data will accumulate in rk->rk_op, will may cause OOM (This does happen in my case).

To prevent this, this patch adds a new parameter max_payload_size, if sum of payload exceeds max_payload_size, comming msgs will be dropped.

[edenhill]

Hi Zhu!

Thanks for your contribution.

I added a similar setting to the 0.7 branch a couple of weeks ago.
There are two differences:

• instead of limiting on total payload size it limits on the total number of messages.
• it lets the application know (by -1 return from rd_kafka_produce() and errno ENOBUFS) that it has dropped the message, allowing the application to take appropriate actions, whatever it may be (spool, backpressure, log, ..).

See this commit:
06fd28e

Configuration: producer.max_outq_msg_cnt

I've merged the 0.7 branch down to master, including this commit.
Let me know if the existing solution (limit on messages, not size) is good enough for you.

[ZhuYouzhi]

Hi, Edenhill.

"it lets the application know (by -1 return from rd_kafka_produce() and errno ENOBUFS) that it has dropped the message, allowing the application to take appropriate actions, " is cool. Thanks.

There are two little question I'd like to discuss:
(1)Limit only on queue len may be not enough, because msg size may vary. Limit on both queue size and queue length may be a good idea.
(2) As for droppping message(due to exceed max_outq_msg_cnt), current patch can cause memory allocated for payload leaks, if I get it right.

[edenhill]

1. This is true, but it will add the cost of an additional atomic variable barrier/sync.
   Dont you think the application can assume an average, or safe, messsage size and use that to calculate max_outq_msg_cnt accordingly?

2. I dont think your patch leaks memory as op_destroy() will clean up allocated resources.
   But the max message size check is done too late, it should be checked before any memory has been allocated, like the max_outq_msg_cnt does.

[ZhuYouzhi]

Hi, Edenhill. Thanks for you reply