Bug 1784166: Alert on high number of etcd leader changes in a short time window #595
Conversation
openshift-ci-robot
added
the
bugzilla/valid-bug
|
@paulfantom: This pull request references Bugzilla bug 1784166, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
size/S
|
thanks for this @paulfantom looks fine to me, deferring LGTM for @smarterclayton to review, thanks! |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alaypatel07, paulfantom The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@paulfantom: All pull requests linked via external trackers have merged. Bugzilla bug 1784166 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
| expr: | | ||
| rate(etcd_server_leader_changes_seen_total{job=~".*etcd.*"}[15m]) > 3 | ||
| for: 15m | ||
| increase((max by (job) (etcd_server_leader_changes_seen_total{job=~".*etcd.*"}) or 0*absent(etcd_server_leader_changes_seen_total{job=~".*etcd.*"}))[15m:1m]) >= 3 |
There was a problem hiding this comment.
why is == 3 bad? With 3 masters and a rolling upgrade of the nodes, couldn't you get 3 leader changes?
Backporting fix from #592, which was merged in #591
/cc @smarterclayton @alaypatel07