Weird throughput pattern

[Zelldon]

Describe the bug

Due to recent discussions I have run a benchmark with different process models, actually I just started a benchmark with make all. This causes to run process models with only start-end, process model with intermediate timer catch event (PT1S) and a process model

Models

Observation

We can observe a weird throughput pattern, which is recurring every ~20 mins

If we take a look what happens on the drops/spikes we can see that a lot of jobs are completed at once

Interesting is that the state seems to grow and then shrink again, after all jobs are completed

If we take a look at the created - completed instance, we can see that we accumulate instances/jobs, which are then at some point released/completed again.

This sounds like it is related to issues like #7955 #8132

To Reproduce

just run a benchmark with make all, but make sure to configure the starters to a rate of 100.

Expected behavior

We expect to complete 300 process instances per second and that the throughput graph is stable.

Log/Stacktrace

Nothing

Environment:

• OS:
• Zeebe Version: develop
• Configuration: benchmark

[Zelldon]

After scaling the workers down it looks much better

It seems that due to lot of workers we can disrupt the throughput, question would be for me whether we want to investigate here more or just document it better for users, which will stumble over this.

[menski]

I think we need more investigation to better understand the client behavior in this case, i.e. what are the metrics of the clients, why are there bursts of activation and completion. Is there an impact of back-off in the client.

[menski]

Maybe a potential topics for chaos days, i.e. the number of workers should not impact the throughput of the cluster /cc @Zelldon

[deepthidevaki]

I see similar behaviours in our weekly benchmark.

It is not consistent. Sometimes it occurs frequently, othertimes the throughput is flat.

[pihme]

Is the timer set to 20 mins by any chance? Then it might be related to the blocking looping over all timers

[Zelldon]

Lets try to reproduce again, if it still happens lets try with our new shiny due date feature flag and document the result here. If it is not failing let's just close it.

[Zelldon]

I'm not able to reproduce this. I run make all which deploys the timers, simple starter and normal starters.

Throughput looks stable. I will close this.