Triggering due timer events causes periodic latency spikes

[romansmirnov]

Describe the bug

Basically, it is the same as #11591 but related to timer events (and not messages). In a nutshell, Zeebe checks regularly for due timer events to eventually trigger them and continue with the process flow. The corresponding checker shares the actor of the Stream Processor which will block the Stream Processor while the checker runs. Also, the checker might submit a batch of commands to trigger timer events which will be executed by the Stream Processor in a row without anything else in between.

Expected behavior

• The Stream Processor and the checker do not share an actor so the Stream Processor continues processing while the checker collects timers to trigger.
• (This might be partially already the case, needs to be checked.) The checker only submits a batch with a limited number of commands. For example, when the checker runs it will collect the first 10 due timer events and submit them as a batch to the log stream. And then continues with collecting the next 10 due timer events, and so on until there are due timer events. That way, triggering the timer events would interleave with any incoming commands from users/clients.
• Instead of writing 10 commands, it could write just one command containing the 10 due timer events to trigger. (Might not be easily possible in terms of rolling upgrades, an old version of Zeebe would not be able to process such a command.)

Hints

• A simple prototype to avoid sharing the actor can be found here (in case of expired messages): poc: async message time to live checker #11550
• Things to consider: When the checker and the Stream Processor do not share an actor, they may run concurrently. That means, the checker reads from RocksDB, and the Stream Processor (mostly) writes to RocksDB. While RocksDB itself is thread-safe, the Zeebe layer may not (like TransactionContext, ...).
• Also, while reading the state by the checker, the state might change.
• The same pattern should be applied to the job's timeline and backoff checker.

Environment:

• Zeebe Version: 8.1

---

• related to SUPPORT-15902
• related to Partitions gets unhealthy if many timers are scheduled #8991

[Zelldon]

Related to #8991

[megglos]

related to #11591

[megglos]

Sync with @abbasadel :

• @abbasadel checks in with Nico once MessageTimeToLiveChecker clogs the log stream with commands #11762 is completed on whether the ZPA team can continue with this right after

[korthout]

Discussed this issue in the ZPA triage:

• not urgent for the 8.2 release (other issues have priority)
• it is an important issue and should be fixed
• marking it is as later as we will first focus on the 8.2 release

[megglos]

as this issue strongly relates to the mission of the ZDP team, we would take this one over to drive it forward

[megglos]

@abbasadel Ole might need support at least for alignment and dicussion from a ZPA engineer