backup: report when job is slow due to overload #142352
Open
+236
−7
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
|
This change is |
dt
force-pushed
the
backup-ac-msg
branch
3 times, most recently
from
d60a989 to
8537276
Compare
Contributor
Author
|
@msbutler @jeffswenson I tried to refactor this into something that should be easy to replicate in other jobs; what do you think of the new structure? |
dt
force-pushed
the
backup-ac-msg
branch
5 times, most recently
from
372800a to
975b01b
Compare
Release note: none. Epic: none.
When a backup is running more slowly because admission control is delaying its progress, it can be easy to miss: one needs to look at specifically the overload dashboard and even then how much a specific backup or changefeed or other job is specifically being impacted is not very clear. This change extends backup to use its collected trace infomation, which includes information about how long the backup spent doing work/waiting for work to be done as well as how long the backup's work spent delayed due to AC, and then logs a warning to the backup job's message log if that ratio exceeds 50%. Release note: none. Epic: none.
Release note: none. Epic: none.
Randomized map iteration makes it harder to compare two dumps side by side since they are presented in different orders. There isn't any logical relationship between span names to preserve, so just alphabetizing them is as good as any other stable ordering. Release note: none. Epic: none.
Contributor
Author
|
Probably want to dig into #143920 before advancing this |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously when looking at a job making less progress than normal, it was sometimes difficult to determine if it was being slowed due to admission control protecting the cluster from overload. One can go to the overload dashboard and see if something is being slowed due to AC by looking at the admission delay micros/sec charts, but this just tells you that something or somethings are being slowed, not how much specifically the job you were looking at is being slowed in particular. Additionally needing to navigate to the overload dashboard to find out if that is or is not why a specific job is hanging is a step that is not intuitive. Instead, if a single (backup) job detect that its work is being substantially delayed due to admission delays in its recorded traces, it can directly surface this fact and inform the user that it in particular was delayed by a certain amount, making it quicker and easier for an operator that is looking at a slow/stuck backup job to determine what is wrong if it is stuck due to the cluster being overloaded.
Release note: none.
Epic: none.