Fix timer scheduling

[Zelldon]

Description

If we already scheduled a Task for a created Timer, we shouldn't schedule another Task on next timer creation. This PR fixes that.

The previous behavior could cause severe issues, like exponential increasing backlog if a lot of timers are created per second etc. see #10532 and #10526

Related issues

closes #10532

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

• The changes are backwards compatibility with previous versions
• If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

• There are unit/integration tests that verify all acceptance criterias of the issue
• New tests are written to ensure backwards compatibility with further versions
• The behavior is tested manually
• The change has been verified by a QA run
• The impact of the changes is verified by a benchmark

Documentation:

• The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
• New content is added to the release announcement
• If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

[Zelldon]

I have run some benchmarks to compare the fix. I was not able yet to run the E2E tests, but I will do with a next release candidate.

Benchmarks

Benchmarks only timers (rate 100 PI/s)

Timer Benchmark with bug

Timer Benchmark with fix

We can see that it is much more stable in the processing and fewer events are skipped. The backpressure is reduced by half.

Mixed Benchmark with bug

For that I used the medic cw 39 (and checked the begin of the benchmark). The begin of the mixed looks always okayisch later it shows higher backpressure. But we can see in the begin already issues with the latencies.

Mixed Benchmark with fix

In the fixed mixed benchmark we see in general processing not much difference, but in the latency which looks much stable. I will run this benchmark for a while. After ca 5 hours the mixed often failed, so lets see. But I think we can already merge it.

[github-actions]

Test Results

   933 files  +  1     933 suites  +1   2h 13m 24s ⏱️ + 8m 25s
7 460 tests  - 30  7 450 ✔️  - 30  10 💤 ±0  0 ❌ ±0 
7 648 runs   - 30  7 638 ✔️  - 30  10 💤 ±0  0 ❌ ±0 

Results for commit e769bcd. ± Comparison against base commit 751ac05.

[Zelldon]

I retagged the docker image such that I can run the E2E tests and it looks great now

Previously:

[saig0]

@Zelldon looks good. 👍

[saig0]

bors merge

[zeebe-bors-camunda]

Build failed:

• Test summary

[Zelldon]

bors r+

[zeebe-bors-camunda]

Build failed:

• Test summary

[Zelldon]

bors r+

NOW

[zeebe-bors-camunda]

Build succeeded:

• Go linting
• Java checks
• Test summary

[backport-action]

Successfully created backport PR #10535 for release-8.1.0.