On each metadata change, the topology of partition followers grows in the Gateway's broker topology

[romansmirnov]

Describe the bug

In a long-running scenario, where the Gateway is up and running for a long time, the heap memory consumption of BrokerTopologyManager grows over time:

Especially, the property partitionFollowers grows the most:

Whenever there is a metadata change, the Gateway gets notified about it by the respective broker. The Gateway will update the topology information accordingly. Thereby, it will update the list of followers for a specific partition every time. Basically, it will always add the broker to the list of followers for a partition:

https://github.com/camunda-cloud/zeebe/blob/df800d41fcc307817b692d17d9c87247f2825516/gateway/src/main/java/io/camunda/zeebe/gateway/impl/broker/cluster/BrokerClusterStateImpl.java#L90-L97

It only removes the broker from the list of followers for a specific partition if the broker transitions to another role then FOLLOWER. Meaning, if a broker remains all the time as FOLLOWER for a specific partition but other metadata changes (like terms or the healthy state) overtime frequently, the broker is added multiple times to the list of followers for that specific partition:

The same applies to the property partitionInactiveNodes:

https://github.com/camunda-cloud/zeebe/blob/df800d41fcc307817b692d17d9c87247f2825516/gateway/src/main/java/io/camunda/zeebe/gateway/impl/broker/cluster/BrokerClusterStateImpl.java#L99-L106

To Reproduce

• Let the cluster with a Gateway run for a long period whereby the metadata changes over time frequently

Expected behavior

• The followers are listed only once for each partition so that the retained size of the object of BrokerTopologyManager doesn't grow over time.

Environment:

• Zeebe Version: 1.4.0-SNAPSHOT

[npepinpe]

Thanks for the clear report, seems like a low hanging fruit and we should do it (under the assumption it's a quick fix).