Bug 1809232: prevent AlertmanagerReceiversNotConfigured false-positive #723
Conversation
|
@paulfantom: This pull request references Bugzilla bug 1809232, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
bugzilla/invalid-bug
|
In this case bugzilla doesn't matter as this targets |
openshift-ci-robot
added
the
do-not-merge/hold
|
Will this fire the moment a user logs into the console after an install? |
|
It will fire 30m after prometheus is up and running. We can decrease this to a much lower value, but 0m is sometimes causing false-positives during an upgrade. |
assets/prometheus-k8s/rules.yaml
Outdated
| @@ -591,6 +591,7 @@ spec: | |||
| occur. Check the OpenShift documentation to learn how to configure notifications | |||
| with Alertmanager. | |||
| expr: cluster:alertmanager_routing_enabled:max == 0 | |||
| for: 30m | |||
There was a problem hiding this comment.
Lets then adjust to 15m and that should ™️ give enough time to fire watchdog alert after upgrades. I would say we can also add note in bugzilla for QE to verify that when cluster first spins up and user logs in this alert is firing. WDYT?
|
We did another upgrade yesterday of our clusters, it seems it took 5 minutes in total for the alert to be resolved. So anything above 5 minutes should work to not have flaky alerts. |
|
Changed to /bugzilla refresh |
openshift-ci-robot
added
bugzilla/valid-bug
|
@paulfantom: This pull request references Bugzilla bug 1809232, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: lilic, paulfantom The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@paulfantom: All pull requests linked via external trackers have merged: openshift/cluster-monitoring-operator#723. Bugzilla bug 1809232 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Delaying alert by 30m prevents false-positive as Watchdog alert should already fire in this time and populate at least one of
alertmanager_notifications_totalmetrics./cc @openshift/openshift-team-monitoring
I wonder if we can shorten it to 15m or less as Watchdog should fire instantly. WDYT @brancz @simonpasquier ?