ProduceRequest failed: Local: Timed out in queue: explicit actions Retry,MsgNotPersisted

[write2jaydeep]

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

Hi @edenhill
In our production, we have two different machines sending messages to the same Kafka cluster. One of them working fine.
Another one is having a problem and the Produce request is being failed with error "ProduceRequest failed: Local: Timed out in queue: explicit actions Retry,MsgNotPersisted"

what could be the probable cause?

do you think socket.nagle.disable=true or queue.buffering.max.ms=1 can create trouble?

Note: we have a high traffic of 2k msgs/second in peak hour

Thank you!

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.5.0
• Apache Kafka version: 0.10.1
• librdkafka client configuration: `security.protocol=sasl_plaintext
• builtin.features=sasl_gssapi
• queue.buffering.max.ms=1
• socket.nagle.disable=true`
• Operating system: Linux
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts
• Critical issue

[write2jaydeep]

Hi @edenhill
I can understand it could be a burden for you guys to reply to all problems.
This problem is on high priority for me, anybody please guide me? I request you to please help me

Thanks
Jaydeep

[rolandyoung]

It looks as rdkafka is reporting that the broker failed to respond in time to a request and then the broker actually went down. I doubt that any configuration on the client side will fix this. If you have a reproducible error, you should try to reproduce it using the current release of librdkafka. You might also want to use a more up-to-date version of the broker software.
If the broker logs show no problems, you may have a problem with the network connection.

[write2jaydeep]

Thanks a lot @rolandyoung
We have another machine with the same version 1.5.0 sending to the same broker and topic, it works very well. It is production system so actually, I can't upgrade librdkafka release.
What parameter do I need to tune on the client-side or broker side if it is problem with network connections?

[rolandyoung]

If you have two clients running the same software connecting to the same broker and one works fine but the other fails badly, with symptoms of network issues, then you do not need to tune the client, you need to fix the network.
(I am sorry, but you will not receive free support from an open source project for an out-of-date release.)

[write2jaydeep]

Thank you @rolandyoung I really appreciate your valuable time.
I will see possibilities to upgrade librdkafka to the latest release.

Thanks once again.