KAFKA-8832:Limit the maximum size read by a fetch request on the kafka server.

[lordcheng10]

We should limit the maximum size read by a fetch request on the kafka server.

[lordcheng10]

I found that kafka is not on the server side, limiting the amount of data read per fetch request. This may cause the kafka server program to report an error: OutOfMemory. Due to unreasonable client configuration, fetch.message.max.bytes configuration is too large, such as 100M, because the kafka server receives a lot of fetch requests at a certain moment, causing the server to report an error: OutOfMemory。So I think this is a bug。

[lordcheng10]

Looking forward to your attention！ @guozhangwang @stanislavkozlovski @junrao @gwenshap @ijuma

[guozhangwang]

@lordcheng10 since you are proposing to add a new config to the Kafka brokers, it needs to go through a KIP discussion as a public API change, could you file one KIP?

Also I'm not sure if OOM is caused by the large min.bytes configs, since during fetch request handling Kafka brokers are usually doing nio zero-copy, i.e. it would not allocate more memory to host the fetched data before sending it to the IO layer, could you elaborate a bit more how you've found out that OOM is related to the large value of min.bytes?

[lordcheng10]

Also I'm not sure if OOM is caused by the large min.bytes configs, since during fetch request handling Kafka brokers are usually doing nio zero-copy, i.e. it would not allocate more memory to host the fetched data before sending it to the IO layer, could you elaborate a bit more how you've found out that OOM is related to the large value of min.bytes?

Thank you for your attention, I put a screenshot of the error log at the time in jira, I think this can explain some problems. https://issues.apache.org/jira/browse/KAFKA-8832

[lordcheng10]

@guozhangwang Thank you for your attention! We all know that when the client fetches the data, the kafka server reads the disk data when sending the response. The method is org.apache.kafka.common.record.FileRecords#writeTo, but the socket will use the buffer. When a fetch data is too large, there will be a lot of network buffers in the memory, so the accumulated memory consumption will be large.
Jira recorded the error log at the time：https://issues.apache.org/jira/browse/KAFKA-8832

[lordcheng10]

@lordcheng10 since you are proposing to add a new config to the Kafka brokers, it needs to go through a KIP discussion as a public API change, could you file one KIP?

Also I'm not sure if OOM is caused by the large min.bytes configs, since during fetch request handling Kafka brokers are usually doing nio zero-copy, i.e. it would not allocate more memory to host the fetched data before sending it to the IO layer, could you elaborate a bit more how you've found out that OOM is related to the large value of min.bytes?
@guozhangwang
I did a test and then dumped the jvm memory at the time. Found that there is a lot of buffer data in memory： https://issues.apache.org/jira/browse/KAFKA-8832