Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1828427: Add CI test to check for critical alerts post upgrade success #24786

Merged
merged 2 commits into from Apr 27, 2020
Merged

Bug 1828427: Add CI test to check for critical alerts post upgrade success #24786

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3b8cb3c
Add CI test to check for crit alerts post upgrade
jottofar Mar 27, 2020
3a92334
Ignore KubeAPIErrorBudgetBurn alert
jottofar Apr 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Ignore KubeAPIErrorBudgetBurn alert 
KubeAPIErrorBudgetBurn alert is firing due to Bug 1821661. Once this
bug is fixed this change will be backed out.
  • Loading branch information
@jottofar
jottofar committed Apr 27, 2020
commit 3a9233400053c036838bdbf7f992874b7a0805fd
3 changes: 2 additions & 1 deletion test/e2e/upgrade/alert/alert.go
View file
Open in desktop
Expand Up @@ -84,7 +84,8 @@ func (t *UpgradeTest) Test(f *framework.Framework, done <-chan struct{}, upgrade
watchdogQuery := fmt.Sprintf(`count_over_time(ALERTS{alertstate="firing",alertname="Watchdog", severity="none"}[%dm])`, alertCheckSleepMinutes)

// Query to check for any critical severity alerts that have occurred within the last alertPeriodCheckMinutes.
jottofar marked this conversation as resolved.
Show resolved Hide resolved
criticalAlertQuery := fmt.Sprintf(`count_over_time(ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|KubeAPILatencyHigh",alertstate="firing",severity="critical"}[%dm]) >= 1`, alertPeriodCheckMinutes)
// TODO Remove KubeAPIErrorBudgetBurn from ignore list once Bug 1821661 is fixed.
criticalAlertQuery := fmt.Sprintf(`count_over_time(ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|KubeAPILatencyHigh|KubeAPIErrorBudgetBurn",alertstate="firing",severity="critical"}[%dm]) >= 1`, alertPeriodCheckMinutes)

tests := map[string]bool{
watchdogQuery: true,
Expand Down