Enable running Message TTL Checker async to stream processor by default

[korthout]

Description

We fixed

• MessageTimeToLiveChecker blocks Stream Processor  #11761

by introducing an experimental feature flag zeebe.broker.experimental.features.enableMessageTtlCheckerAsync which allows running the Message TTL Checker asynchronous to the stream processor when enabled.

By default, this feature is disabled. As 8.2 is now released, we should enable the feature by default so we can build up confidence in it and eventually remove the feature flag altogether.

Left over from

• Discarding expired messages causes periodic latency spikes #11591

[korthout]

ZPA Triage:

• trial clusters will run on alpha as well, perhaps we should investigate to run this first on benchmarks
• that's only useful if the benchmarks actually use a non-zero TTL for messages
• otherwise, let's just flip the switch on the next alpha
• alternatively, we could attempt to roll it out to small groups by changing the setting for a small group of new trial clusters
• we should at least have a way to verify that this is not leading to issues once rolled out

---

@Zelldon Are there any chaos experiments running on testbench that use a non-zero message TTL?

EDIT: I've handed Ole the responsibility for confidence in this feature, so I no longer need to know about any chaos experiments that cover this

[korthout]

Today, I talked with @oleschoenburg about this topic:

Now:

• There is a need to have this same feature flag available for timers Triggering due timer events causes periodic latency spikes #11594
• Ole will work on this, and I will review it
• we have already verified the following:
  • that due date timer checker is threadsafe
  • that there is a yielding behavior that should result in similar non-flooding of the log, like message ttl checker's batch limit

Further iterations:

• Ole wants to introduce a new feature flag that allows running all scheduled tasks async to the stream processor and remove the message ttl specific feature flag Scheduled tasks should not block processing #12302
• Ole would then be responsible for switching the default setting of this feature flag and testing it
• ZPA will remove any left-over state mutations from the scheduled tasks
• Nico will raise the priority of these changes at the ZPA planning session

This means that we can lower the priority of this issue to later, as it will likely be removed anyways as part of

• Scheduled tasks should not block processing #12302

[megglos]

@korthout this can be closed now, given #12302 is done, right?

[korthout]

@megglos Yes we close this 👍

✅ All scheduled tasks are run async by default:

• feat: optionally run all scheduled tasks outside of processing actor #13524

Thanks again @oleschoenburg 🙇