WIP: Reduce tolerance for etcd leader elections #25430
Conversation
openshift-ci-robot
added
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ironcladlou The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci-robot
added
the
approved
| defer g.GinkgoRecover() | ||
| oc := exutil.NewCLI("etcd-leader-change").AsAdmin() | ||
| g.It("leader changes are not excessive", func() { | ||
| prometheus, err := client.NewE2EPrometheusRouterClient(oc) | ||
| o.Expect(err).ToNot(o.HaveOccurred()) | ||
| g.By("Examining the rate of increase in the number of etcd leadership changes for last five minutes") | ||
| result, _, err := prometheus.Query(context.Background(), "increase((max by (job) (etcd_server_leader_changes_seen_total) or 0*absent(etcd_server_leader_changes_seen_total))[15m:1m])", time.Now()) | ||
| result, _, err := prometheus.Query(context.Background(), "max by (job) (etcd_server_leader_changes_seen_total)", time.Now()) |
There was a problem hiding this comment.
Ah, sounds like the by (job) suggestion was originally from @smarterclayton. I'm still not clear on if/why it's needed.
There was a problem hiding this comment.
Is it because each instance keeps its own accounting here and they often diverge? Thus we want the max as reported by all instances. cc @hexfusion
|
/test e2e-azure |
|
/retest |
ironcladlou
force-pushed
the
detect-unexpected-elections
branch
from
5a05544
to
3098cac
Compare
|
/retest |
|
/test all |
|
@ironcladlou: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Replacing this with #25454 /close |
|
@ironcladlou: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
No description provided.