Pinot server memory usage #14225
Description
We are running a Pinot cluster in production with 4 servers (16 GB ,4 Cores). There are multiple realtime tables consuming from Kafka topics. We are observing the 60 to 70 % of the total memory is being used during the peak load. The issue is memory usage doesn't come down even after the load stabilises after sometime.
This is causing OOM errors when the utilisation steadily increases and doesn't steps down. There is no real benefit of increasing the memory as this occurs only when the Peak wps rate increases beyond a level.
How do we identify the exact processes which causes the memory spike? Is there a way to make sure the memory usage stabilises/comes down when wps comes down?
Also is this behaviour somehow related to GC or any configuration?
