feat: run timer due date checker concurrently to processing #12403
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lenaschoenburg
force-pushed
the
os-11594-async-due-timer-triggering
branch
from
efdf474
to
771a079
Compare
lenaschoenburg
added
backport stable/8.1
backport stable/8.2
This state is used by the timer due date checker scheduled task and should not be shared with regular processing.
Test Results1 054 files ± 0 1 054 suites ±0 1h 58m 9s ⏱️ - 2m 20s Results for commit 1093893. ± Comparison against base commit 1d32130. This pull request removes 566 and adds 460 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
This copies the scheme already used for async message TTL checking and adds a new experimental feature flag. It is disabled by default for safety but enabled in tests so that we can find potential issues faster.
This uses the new config `enableTimerDueDateCheckerAsync` to switch between running the timer due date checker scheduled task on the processing actor or another actor. The reasoning is the same as for running message TTL checking outside of the processing actor: Some use cases have a lot of messages/timers that need to be checked and that potentially expire at the same time. This should not block regular processing so these tasks must not block the processing actor. Timer due date checking already supported an experimental feature to interrupt a single check run after a certain amount of time has passed. Conceptually this is similar to the batching that we implemented for message TTL checking. This feature continues to function, regardless of whether async checking is enabled or not.
lenaschoenburg
force-pushed
the
os-11594-async-due-timer-triggering
branch
from
771a079
to
1093893
Compare
korthout
approved these changes
Apr 17, 2023
|
bors r+ |
|
Build succeeded: |
|
Backport failed for Please cherry-pick the changes locally. git fetch origin stable/8.1
git worktree add -d .worktree/backport-12403-to-stable/8.1 origin/stable/8.1
cd .worktree/backport-12403-to-stable/8.1
git checkout -b backport-12403-to-stable/8.1
ancref=$(git merge-base 1d321300618431b49259f490ced2202b5f78a321 1093893e7b0aa27849677218d1ac32738e77fa3b)
git cherry-pick -x $ancref..1093893e7b0aa27849677218d1ac32738e77fa3b |
|
Successfully created backport PR for |
zeebe-bors-camunda bot
added a commit
that referenced
this pull request
Apr 18, 2023
12467: [Backport stable/8.1] feat: run timer due date checker concurrently to processing r=oleschoenburg a=oleschoenburg Manual backport of #12403. No tricky merge conflicts, just minor differences in naming and the list of available feature flags. 12468: [Backport 8.1]: skip unnecessary blacklist check r=megglos a=Zelldon ## Description Backport #12306 <!-- Please explain the changes you made here. --> ## Related issues <!-- Which issues are closed by this PR or are related --> closes to #12041 Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com> Co-authored-by: Meggle (Sebastian Bathke) <sebastian.bathke@camunda.com>
zeebe-bors-camunda bot
added a commit
that referenced
this pull request
Apr 18, 2023
12467: [Backport stable/8.1] feat: run timer due date checker concurrently to processing r=oleschoenburg a=oleschoenburg Manual backport of #12403. No tricky merge conflicts, just minor differences in naming and the list of available feature flags. Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
zeebe-bors-camunda bot
added a commit
that referenced
this pull request
Apr 19, 2023
12467: [Backport stable/8.1] feat: run timer due date checker concurrently to processing r=oleschoenburg a=oleschoenburg Manual backport of #12403. No tricky merge conflicts, just minor differences in naming and the list of available feature flags. Co-authored-by: Ole Schönburg <ole.schoenburg@gmail.com>
lenaschoenburg
added
the
version:8.2.3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This follows a pattern we established for message TTL checking and introduces a new experimental feature flag
zeebe.broker.experimental.features.enableTimerDueDateCheckerAsync. When enabled, timer due dates are checked outside of the processing actor.This ensures that a large number of expired timers does not block other processing and thus cause latency spikes.
Again similar to message TTL checking, there is another experimental configuration that enables yielding of the timer due date checker after a fixed amount of time has passed. This is conceptually similar to
ttlCheckerBatchLimitin that it ensures that a single run of a scheduled task does not block for too long and produces a limited amount of new records.Yielding continues to function the same, regardless of whether async checking is enabled or not.
closes #11594