Unstable cluster on bigger state

[Zelldon]

Describe the bug

I have setup an stable cluster on gke (no preemtable nodes) and expected to run a benchmark for a while. Unfortunately it died after around 12 hours.

Investigating this I saw that we had unexpected often Leader changes in the cluster.

These caused then also a growing of the disk space. Because we are not able to compact if we have to often a leader change.

As we can see above only a few restarts are really related due to restarts. The restarts were caused by reaching the memory limit. Probably rocksdb reached there limits again.

We had the problem with our helm charts that the gateway used only one thread to process the requests, which means we were not able to work fast on the created jobs. See related issue #4524 . This caused an growing running count.

As you can see we have around 2 million running instances. This big state causes also a bigger snapshot size.

This might be related to the leader changes, because it will take a long time to replicate these snapshots. Furthermore it increases the disk usage as well.

To Reproduce

Start an usual benchmark with non preemtable nodes and start less workers.
Steps to reproduce the behavior

Expected behavior
A stable running system.

Environment:

• OS: k8
• Zeebe Version: 0.24.0-SNAPSHOT
• Configuration: elastic, standalone gw

[Zelldon]

Happens again with a new benchmark. We are not able to work on all jobs, which means we building up a bigger state. This causes also bigger snapshot sizes.

We see a lot of leader changes in the benchmark and missing heartbeats.

In the end we went out of disk space.

[npepinpe]

Downgrading, as while this is important to figure out, it does not impact our own production environment (thanks pre-emptible nodes), and as such isn't critical to our OKRs.