Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alerting: During legacy migration reduce the number of created silences #78505

Merged
merged 4 commits into from Jan 24, 2024

Commits on Jan 10, 2024

  1. Alerting: During legacy migration reduce the number of created silences

    During legacy migration every migrated rule was given a label rule_uid=<uid>.
    This was used to silence DatasourceError/DatasourceNoData alerts for
    migrated rules that had either ExecutionErrorState/NoDataState set to
    keep_state, respectively.
    
    This could potentially create a large amount of silences and a high cardinality
    label. Both of these scenarios have poor outcomes for CPU load and latency in
    unified alerting.
    
    Instead, this change creates one label per ExecutionErrorState/NoDataState when
    they are set to keep_state as well as two silence rules, if rules with said
    labels were created during migration. These silence rules are:
    
    - __legacy_silence_error_keep_state__ = true
    - __legacy_silence_nodata_keep_state__ = true
    
    This will drastically reduce the number of created silence rules in most cases
    as well as not create the potentially high cardinality label `rule_uid`.
    JacobsonMT committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    6297513 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2024

  1. Configuration menu
    Copy the full SHA
    06170ce View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f5a6991 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4ad340a View commit details
    Browse the repository at this point in the history