Real time server query performance hit when catching up Kafka backlogs

[dongxiaoman]

Right now our upstream Kafka log has a "batching" behavior that every a few minutes there is a sudden spike in Kafka traffic (due to upstream Flink job's checkpointing behavior). This cause a periodical Kafka backlog for our Pinot server. For example, every 30 minutes, we will suddenly have lots of Kafka backlog to catch up.

This is fine but the query performance suffers a lot during that backlog period. Based on our quick chart analysis it could be I/O bound.
Below are some of the charts we can share, showing correlation.

[dongxiaoman]

Right now maybe adding an optional rate limit in the Kafka consumption of Pinot Server is a good way to go?

[mcvsubbu]

If we add a rate limit, it should be stream-generic, or perhaps a combination of generic plus kafka-specific. Yes, we do have such patterns as well, but we have not seen a query performance drop due to this.

I think the rate limiter should be in terms of bytes (and not messages).

@npawar any comments?

[snleee]

@jtao15 Let's take this item after #6111