Alerting: Support hysteresis command expression #75189
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuri-tceretian
force-pushed
the
yuri-tceretian/expr-hysteresis-alerting
branch
from
d64d02a
to
5e1b397
Compare
yuri-tceretian
force-pushed
the
yuri-tceretian/expr-hysteresis-alerting
branch
from
72aa808
to
24d23fd
Compare
yuri-tceretian
requested review from
gillesdemey,
VikaCep,
konrad147,
soniaAguilarPeiron,
rwwiv,
JacobsonMT,
grobinson-grafana,
papagian,
zserge and
yangkb09
and removed request for
a team
yuri-tceretian
added
add to changelog
no-backport
public/app/features/alerting/unified/components/expressions/Expression.tsx
Outdated
Show resolved
Hide resolved
|
3 tasks
|
last commit : skip hysteresis values being validated when unchecking hysteresis. fix-hysteresis-validation.mp4 |
|
/deploy-to-hg |
# Conflicts: # pkg/expr/threshold_test.go
|
/deploy-to-hg |
|
My last commit is a fix for the style for hysteresis fields in dashboard panel expressions. Before the change:
After the change:
 |
|
After a conversation with @yuri-tceretian we agreed on hiding hsyteresis in panel expressions. |
# Conflicts: # pkg/tests/api/alerting/testing.go
|
/deploy-to-hg |
|
|
JacobsonMT
approved these changes
Jan 4, 2024
3 tasks
zserge
pushed a commit
that referenced
this pull request
Jan 22, 2024
Backend: * Update the Grafana Alerting engine to provide feedback to HysteresisCommand. The feedback information is stored in state.Manager as a fingerprint of each state. The fingerprint is persisted to the database. Only fingerprints that belong to Pending and Alerting states are considered as "loaded" and provided back to the command. - add ResultFingerprint to state.State. It's different from other fingerprints we store in the state because it is calculated from the result labels. - add rule_fingerprint column to alert_instance - update alerting evaluator to accept AlertingResultsReader via context, and update scheduler to provide it. - add AlertingResultsFromRuleState that implements the new interface in eval package - update getExprRequest to patch the hysteresis command. * Only one "Recovery Threshold" query is allowed to be used in the alert rule and it must be the Condition. Frontend: * Add hysteresis option to Threshold in UI. It's called "Recovery Threshold" * Add test for getUnloadEvaluatorTypeFromCondition * Hide hysteresis in panel expressions * Refactor isInvalid and add test for it * Remove unnecesary React.memo * Add tests for updateEvaluatorConditions --------- Co-authored-by: Sonia Aguilar <soniaaguilarpeiron@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this feature?
This PR adds support for a new type of command introduced in #70998. The hysteresis command requires information about the results of the previous evaluation. This PR updates the state manager and scheduler to facilitate this information for the evaluation engine:
The structure AlertingResultsFromRuleState implements the logic that determines what metric\dimension\state is considered as "loaded":
Why do we need this feature?
To support hysteresis expression in alerting. This will help users reduce flapping on the alert rules when the metric value frequently crosses threshold boundaries.
Who is this feature for?
Alerting
Which issue(s) does this PR fix?:
Related #6202
Special notes for your reviewer:
recoveryThreshold. if it is disabled the expr engine does not create HysteresisCommand struct and code in this PR does not do anything.UI screen recording
hysteresis_UI.mp4
When viewing the rule
Please check that: