[Question] CPU and Memory Utilization too High for Manager/startup module [Distributed Architecture startup , collector , warehouse]

[kaushaldeokar]

Question

1. Currently Hertzbeat uses apache arrow and hertzbeat has both in-memory heap and external memory usage .

It uses apache arrow to write in external memory .

Our scenario : we have distributed actictecture where we spin multi-collectors and warehouses . data get converted to arrow object in collector after collection and is passed to manager via netty here in manager data gets deserialized and stored into memory and suspect manager memory is increased due to this and later the data send to warehouse for persistence via kafka topic .

if we add more monitors the memory grows drastically -> need to solve this

here manager memory is increased and not decreased later .

tried to clear the objects from memory before sending to kafka topic but it didnot work .

here is sample code from manager where it pushes the data to kafka .

attached the ss for Heap and Code as well