Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timer creations will cause to schedule multiple Tasks #10532

Closed
Zelldon opened this issue Sep 28, 2022 · 0 comments · Fixed by #10534
Closed

Timer creations will cause to schedule multiple Tasks #10532

Zelldon opened this issue Sep 28, 2022 · 0 comments · Fixed by #10534
Assignees
@Zelldon
Labels
area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) kind/bug Categorizes an issue or PR as a bug severity/critical Marks a stop-the-world bug, with a high impact and no existing workaround version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0

Comments

@Zelldon
Copy link
Member

Zelldon commented Sep 28, 2022

Had a look together with @saig0 at the DueDateChecker, because I was wondering why so many Tasks are scheduled and same batches are written. We found out that timer creations can cause new task scheduling, even if already a task is scheduled.

See https://github.com/camunda/zeebe/blob/main/engine/src/main/java/io/camunda/zeebe/engine/processing/scheduled/DueDateChecker.java#L53 here the checkerRunning is not set.

Previously we used the scheduledTimer to indicate that already a timer is running. https://github.com/camunda/zeebe/blob/8.0.6/engine/src/main/java/io/camunda/zeebe/engine/processing/scheduled/DueDateChecker.java#L47

This issue caused a load of scheduled timers, especially if we have many timer creations. This again caused many (and same timer Triggers per second), which caused a big backlog and high backpressure.
I will run a benchmark to see how it behaves, but I think it can explain the high backpressure in the mixed benchmarks.

Originally posted by @Zelldon in #10526 (comment)
@Zelldon Zelldon added kind/bug Categorizes an issue or PR as a bug severity/high Marks a bug as having a noticeable impact on the user with no known workaround severity/critical Marks a stop-the-world bug, with a high impact and no existing workaround area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) and removed severity/high Marks a bug as having a noticeable impact on the user with no known workaround labels Sep 28, 2022
@Zelldon Zelldon self-assigned this Sep 28, 2022
@Zelldon Zelldon mentioned this issue Sep 28, 2022
Merged
10 tasks
zeebe-bors-camunda bot added a commit that referenced this issue Sep 28, 2022
@zeebe-bors-camunda @Zelldon
merge: #10534
2db19d2 
10534: Fix timer scheduling r=saig0 a=Zelldon

## Description

If we already scheduled a Task for a created Timer, we shouldn't schedule another Task on next timer creation. This PR fixes that.

The previous behavior could cause severe issues, like exponential increasing backlog if a lot of timers are created per second etc. see #10532 and #10526

<!-- Please explain the changes you made here. -->

## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #10532



Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
zeebe-bors-camunda bot added a commit that referenced this issue Sep 28, 2022
@zeebe-bors-camunda @Zelldon
merge: #10534
71cec2f 
10534: Fix timer scheduling r=Zelldon a=Zelldon

## Description

If we already scheduled a Task for a created Timer, we shouldn't schedule another Task on next timer creation. This PR fixes that.

The previous behavior could cause severe issues, like exponential increasing backlog if a lot of timers are created per second etc. see #10532 and #10526

<!-- Please explain the changes you made here. -->

## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #10532



Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
@backport-action backport-action mentioned this issue Sep 28, 2022
Merged
zeebe-bors-camunda bot added a commit that referenced this issue Sep 28, 2022
@zeebe-bors-camunda @Zelldon
merge: #10535
7eb5aa1 
10535: [Backport release-8.1.0] Fix timer scheduling r=Zelldon a=backport-action

# Description
Backport of #10534 to `release-8.1.0`.

relates to #10532 #10526 #10532

Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
zeebe-bors-camunda bot added a commit that referenced this issue Sep 28, 2022
@zeebe-bors-camunda @Zelldon
merge: #10535
02f43d2 
10535: [Backport release-8.1.0] Fix timer scheduling r=Zelldon a=backport-action

# Description
Backport of #10534 to `release-8.1.0`.

relates to #10532 #10526 #10532

Co-authored-by: Christopher Zell <zelldon91@googlemail.com>
@Zelldon Zelldon mentioned this issue Sep 29, 2022
Closed
@Zelldon Zelldon added the version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0 label Oct 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Zelldon Zelldon

Labels
area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) kind/bug Categorizes an issue or PR as a bug severity/critical Marks a stop-the-world bug, with a high impact and no existing workaround version:8.1.0 Marks an issue as being completely or in parts released in 8.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix timer scheduling camunda/camunda
1 participant
@Zelldon