Rdkafka consumer settings for high latency environments

[sverch]

Our use case for kafka involves high latency (on the order of a second round trip time) and sudden bursts of messages on the order of 300k. We are seeing slowness in that situation, and I was looking into different kafka settings that might help mitigate this.

I saw that max.poll.records is set to 500 by default. I thought this might solve our problem (or would at least be worth testing), but I saw that it is not supported in rdkafka and found these two issues:

• Add support for max.poll.records consumer configuration confluentinc/librdkafka#1653
• No such configuration property: "max.poll.records" confluentinc/confluent-kafka-go#380

It seems like those use cases were slightly different, in that they wanted to limit the batch size due to memory concerns. In our case, processing 300k messages at a batch size of 500 would spend about ten minutes waiting for network round trips.

I also found this documentation that is related, but not quite for this case:

• Documentation on increasing the batch size for the producer
• Documentation on how to optimize for low latency

Specifically, I'm wondering:

• Is it possible for this setup to be slow, or are the polls pipelined behind the scenes somehow?
• Are there specific settings that would help with a high latency consumer and large bursts of messages? (Note, the processing itself is fast).