Possible memory leak in follower

[korthout]

Describe the bug

Medic benchmark CW35 is showing signs of a possible memory leak.

See Process memory usage and JVM memory usage. Note that this occurs on 1 of the followers (zeebe-2, leader is zeebe-1). There have not been any role changes.

See Grafana medic-cw-35-01f895df7-benchmark for more details.

[deepthidevaki]

Observed high direct memory use in both followers

2021-08-31 17:10:02.351 CEST
Exception in thread "Broker-2-zb-actors-3" java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) at io.camunda.zeebe.util.allocation.DirectBufferAllocator.allocate(DirectBufferAllocator.java:20) at io.camunda.zeebe.util.allocation.BufferAllocators.allocateDirect(BufferAllocators.java:16) at io.camunda.zeebe.dispatcher.DispatcherBuilder.initAllocatedBuffer(DispatcherBuilder.java:147) at io.camunda.zeebe.dispatcher.DispatcherBuilder.build(DispatcherBuilder.java:93) at io.camunda.zeebe.logstreams.impl.log.LogStreamImpl.openAppender(LogStreamImpl.java:280) at io.camunda.zeebe.logstreams.impl.log.LogStreamImpl.createWriter(LogStreamImpl.java:208) at io.camunda.zeebe.logstreams.impl.log.LogStreamImpl.lambda$newLogStreamBatchWriter$1(LogStreamImpl.java:115) at io.camunda.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) at io.camunda.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) at io.camunda.zeebe.util.sched.ActorTask.execute(ActorTask.java:122) at io.camunda.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:94) at io.camunda.zeebe.util.sched.ActorThread.doWork(ActorThread.java:78) at io.camunda.zeebe.util.sched.ActorThread.run(ActorThread.java:191)

zeebe-1 is the leader for all partition.

Heap OOM could be just a consequence of OOM in direct memory.

[deepthidevaki]

Also found many StreamProcessor instance in the heap dump. Most of them are already closed (checked their state), but not garbage collected.

Cyclic references??

[deepthidevaki]

[deepthidevaki]

Here https://github.com/camunda-cloud/zeebe/blob/a3156320fdb56f7fb6c252b37da32a75dfebd059/broker/src/main/java/io/camunda/zeebe/broker/engine/impl/PartitionCommandSenderImpl.java#L34
we are adding TopologyPartitionListener as a listener to TopologyManager. But we never remove it when we close the leader.

[deepthidevaki]

It is not just the topology listener. StreamProcessor and LogStream are not garbage collected due to "several" listeners registered but never removed when a role is closed. So after a each transition, it is accumulated and eventually gits OOM on direct memory.