Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results #843

tjungblu · 2022-06-01T13:00:04Z

No description provided.

…ults

openshift-ci · 2022-06-01T13:00:26Z

@tjungblu: This pull request references Bugzilla bug 2092395, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.11.0) matches configured target release for branch (4.11.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @geliu2016

In response to this:

Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tjungblu · 2022-06-01T13:02:45Z

/cherry-pick release-4.10
/cherry-pick release-4.9

openshift-cherrypick-robot · 2022-06-01T13:02:46Z

@tjungblu: once the present PR merges, I will cherry-pick it on top of release-4.10 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.10
/cherry-pick release-4.9
/cherry-pick release-4.8
/cherry-pick release-4.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Elbehery · 2022-06-01T14:41:40Z

/retest

tjungblu · 2022-06-01T15:40:12Z

/retest-required

related to changes in openshift/cluster-etcd-operator#843

tjungblu · 2022-06-01T16:39:27Z

runbook update here: openshift/runbooks#55

tjungblu · 2022-06-01T16:48:52Z

/retest-required

tjungblu · 2022-06-02T07:18:13Z

/retest-required

tjungblu · 2022-06-02T09:04:53Z

/retest-required

tjungblu · 2022-06-02T14:27:07Z

@wking by popular demand of the etcd team, do you also want to take a look?

dusk125 · 2022-06-02T14:32:32Z

@wking by popular demand of the etcd team, do you also want to take a look?

Unless Trevor has any thoughts, lgtm

EmilyM1 · 2022-06-02T17:21:06Z

/lgtm

dusk125 · 2022-06-03T13:14:25Z

/approve

openshift-ci-robot · 2022-06-03T14:14:26Z

/retest-required

Remaining retests: 2 against base HEAD 7e439f3 and 8 for PR HEAD 05b302a in total

wking · 2022-06-03T16:06:09Z

manifests/0000_90_etcd-operator_03_prometheusrule.yaml

        runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md
+        summary: etcd cluster 99th percentile fsync durations are too high.


nit: etcdHighFsyncDurations is not claiming a specific percentile, so you might want to go more generic here in summary with something like "etcd cluster fsync durations are too high.". The description should definitely call out 99th percentile. That would give you space for extending your existing series of graduated etcdHighFsyncDurations with versions that set different thresholds for different percentiles, and still have them all rolled up into a single reporting entry.

Same feedback applies to some other summary, like the one for etcdHighCommitDurations.

that's from the upstream mixin: https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet#L168-L182

nit: etcdHighFsyncDurations is not claiming a specific percentile, so you might want to go more generic here in summary with something like "etcd cluster fsync durations are too high.".

hmm, are you sure? it does say histogram_quantile(0.99, ...) in the expression of the alert.

wking · 2022-06-03T16:08:54Z

manifests/0000_90_etcd-operator_03_prometheusrule.yaml

-        description: 'etcd cluster "{{ $labels.job }}": database size exceeds the
-          defined quota on etcd instance {{ $labels.instance }}, please defrag or
-          increase the quota as the writes to etcd will be disabled when it is full.'
+        description: 'etcd cluster "{{ $labels.job }}": database size exceeds the defined quota on etcd instance {{ $labels.instance }}, please defrag or increase the quota as the writes to etcd will be disabled when it is full.'


"... exceeds the defined quota..." is a bit premature. I'd just say "... is {{ $value }}% of the defined quota..." or some such.

that's also from upstream: https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet#L213-L226

but I agree, this is a rather strange description.

wking · 2022-06-03T16:10:26Z

manifests/0000_90_etcd-operator_03_prometheusrule.yaml

        runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdBackendQuotaLowSpace.md
+        summary: etcd cluster database is running full.


"running full" is not very specific about what is full. Maybe rephrase to "etcd cluster database size is near quota" or something?

same upstream mixing as above: https://github.com/etcd-io/etcd/blob/main/contrib/mixin/mixin.libsonnet#L213-L226

I think this one is actually fine, but diverges from it's alert description above. Should definitely be updated upstream.

wking · 2022-06-03T16:17:12Z

manifests/0000_90_etcd-operator_03_prometheusrule.yaml

-          leading to 50% increase in database size over the past four hours on etcd
-          instance {{ $labels.instance }}, please check as it might be disruptive.'
+        description: 'etcd cluster "{{ $labels.job }}": Observed surge in etcd writes leading to 50% increase in database size over the past four hours on etcd instance {{ $labels.instance }}, please check as it might be disruptive.'
+        summary: etcd cluster database growing very fast.
      expr: |
        increase(((etcd_mvcc_db_total_size_in_bytes/etcd_server_quota_backend_bytes)*100)[240m:1m]) > 50


unrelated nit, but putting etcd_server_quota_backend_bytes inside the increase could lead to false-positives if the user scales down the quota because they had overprovisioned. What you care about is consumption vs. the current quota, so maybe:

increase(etcd_mvcc_db_total_size_in_bytes[4h]) / etcd_server_quota_backend_bytes *100 > 50

with a description like:

etcd cluster "{{ $labels.job }}": etcd database size increased {{ $value }}% of the configured quota over the past four hours on etcd instance {{ $labels.instance }}. Please defrag (FIXME: doc link?) or increase the quota (FIXME: doc link?) as the writes to etcd will be disabled when it is full. Alternatively, investigate consumption and see if you can remove whatever's spewing this data into etcd (FIXME: reword. Doc link).

Good point, let's take that upstream too.

I think OCP customers currently don't have the option to change the size, so unless you go completely unmanaged etcd it's not possible to get into such scenario.

tjungblu · 2022-06-07T12:02:49Z

/retest-required

dusk125 · 2022-06-07T13:40:49Z

/lgtm

openshift-ci · 2022-06-07T13:41:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dusk125, EmilyM1, tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dusk125]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tjungblu · 2022-06-07T13:42:38Z

/retest-required

openshift-ci-robot · 2022-06-07T15:16:53Z

/retest-required

Remaining retests: 2 against base HEAD 7e439f3 and 8 for PR HEAD 5c7d414 in total

openshift-ci-robot · 2022-06-07T15:48:45Z

/retest-required

Remaining retests: 1 against base HEAD 7e439f3 and 7 for PR HEAD 5c7d414 in total

openshift-ci-robot · 2022-06-07T17:50:34Z

/retest-required

Remaining retests: 0 against base HEAD 7e439f3 and 6 for PR HEAD 5c7d414 in total

tjungblu · 2022-06-08T09:13:38Z

/retest-required

tjungblu · 2022-06-08T13:45:26Z

/retest-required

openshift-ci-robot · 2022-06-09T04:45:31Z

/retest-required

Remaining retests: 2 against base HEAD 82bf899 and 5 for PR HEAD 5c7d414 in total

tjungblu · 2022-06-09T08:32:47Z

/retest-required

openshift-ci-robot · 2022-06-09T09:45:35Z

/retest-required

Remaining retests: 1 against base HEAD 82bf899 and 4 for PR HEAD 5c7d414 in total

tjungblu · 2022-06-09T10:15:05Z

/retest-required

tjungblu · 2022-06-09T12:36:49Z

/retest-required

openshift-ci-robot · 2022-06-09T12:36:56Z

/retest-required

Remaining retests: 0 against base HEAD 82bf899 and 3 for PR HEAD 5c7d414 in total

tjungblu · 2022-06-09T14:33:50Z

.......

tjungblu · 2022-06-09T15:11:56Z

overriding the jobs now, dunno why prow wanted to retest the whole thing after all was green already...

/override ci/prow/e2e-aws-serial

openshift-ci · 2022-06-09T15:12:51Z

@tjungblu: tjungblu unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

In response to this:

overriding the jobs now, dunno why prow wanted to retest the whole thing after all was green already...

/override ci/prow/e2e-aws-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tjungblu · 2022-06-09T15:13:46Z

/retest-required

tjungblu · 2022-06-09T18:05:42Z

/retest-required

openshift-ci · 2022-06-09T18:13:09Z

@tjungblu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-five-control-plane-replicas	`5c7d414`	link	false	`/test e2e-gcp-five-control-plane-replicas`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2022-06-09T18:55:52Z

/retest-required

Remaining retests: 2 against base HEAD b71793a and 2 for PR HEAD 5c7d414 in total

openshift-ci · 2022-06-09T21:55:31Z

@tjungblu: All pull requests linked via external trackers have merged:

Bugzilla bug 2092395 has been moved to the MODIFIED state.

In response to this:

Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2022-06-09T21:56:14Z

@tjungblu: new pull request created: #850

In response to this:

/cherry-pick release-4.10
/cherry-pick release-4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot · 2022-06-09T21:56:51Z

@tjungblu: #843 failed to apply on top of branch "release-4.9":

Applying: Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results
Using index info to reconstruct a base tree...
M	jsonnet/custom.libsonnet
M	jsonnet/jsonnetfile.lock.json
M	jsonnet/vendor/github.com/etcd-io/etcd/contrib/mixin/mixin.libsonnet
M	manifests/0000_90_etcd-operator_03_prometheusrule.yaml
Falling back to patching base and 3-way merge...
Auto-merging manifests/0000_90_etcd-operator_03_prometheusrule.yaml
CONFLICT (content): Merge conflict in manifests/0000_90_etcd-operator_03_prometheusrule.yaml
Auto-merging jsonnet/vendor/github.com/etcd-io/etcd/contrib/mixin/mixin.libsonnet
CONFLICT (content): Merge conflict in jsonnet/vendor/github.com/etcd-io/etcd/contrib/mixin/mixin.libsonnet
Auto-merging jsonnet/jsonnetfile.lock.json
CONFLICT (content): Merge conflict in jsonnet/jsonnetfile.lock.json
Auto-merging jsonnet/custom.libsonnet
CONFLICT (content): Merge conflict in jsonnet/custom.libsonnet
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.10
/cherry-pick release-4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong res…

05b302a

…ults

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jun 1, 2022

openshift-ci bot requested review from geliu2016, Elbehery and hasbro17 June 1, 2022 13:00

tjungblu added a commit to tjungblu/runbooks that referenced this pull request Jun 1, 2022

Bug 2092395: Updating with latest threshold changes

196227b

related to changes in openshift/cluster-etcd-operator#843

tjungblu mentioned this pull request Jun 1, 2022

Bug 2092395: Updating with latest threshold changes openshift/runbooks#55

Merged

tjungblu requested a review from wking June 2, 2022 14:27

openshift-ci bot assigned EmilyM1 Jun 2, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 2, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 3, 2022

wking reviewed Jun 3, 2022

View reviewed changes

openshift-ci bot assigned dusk125 Jun 7, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 7, 2022

openshift-merge-robot merged commit 28a4ae4 into openshift:master Jun 9, 2022
10 of 11 checks passed

openshift-cherrypick-robot mentioned this pull request Jun 9, 2022

[release-4.10] Bug 2095579: etcdHighNumberOfFailedGRPCRequests alerts with wrong results #850

Merged

tjungblu deleted the alert branch November 28, 2022 16:43

		runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdHighFsyncDurations.md
		summary: etcd cluster 99th percentile fsync durations are too high.

		runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-etcd-operator/etcdBackendQuotaLowSpace.md
		summary: etcd cluster database is running full.

Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results #843

Bug 2092395: etcdHighNumberOfFailedGRPCRequests alerts with wrong results #843

Conversation

tjungblu commented Jun 1, 2022

openshift-ci bot commented Jun 1, 2022

tjungblu commented Jun 1, 2022 • edited

openshift-cherrypick-robot commented Jun 1, 2022

Elbehery commented Jun 1, 2022

tjungblu commented Jun 1, 2022

tjungblu commented Jun 1, 2022

tjungblu commented Jun 1, 2022

tjungblu commented Jun 2, 2022

tjungblu commented Jun 2, 2022

tjungblu commented Jun 2, 2022

dusk125 commented Jun 2, 2022

EmilyM1 commented Jun 2, 2022

dusk125 commented Jun 3, 2022

openshift-ci-robot commented Jun 3, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tjungblu commented Jun 7, 2022

dusk125 commented Jun 7, 2022

openshift-ci bot commented Jun 7, 2022

tjungblu commented Jun 7, 2022

openshift-ci-robot commented Jun 7, 2022

openshift-ci-robot commented Jun 7, 2022

openshift-ci-robot commented Jun 7, 2022

tjungblu commented Jun 8, 2022

tjungblu commented Jun 8, 2022

openshift-ci-robot commented Jun 9, 2022

tjungblu commented Jun 9, 2022

openshift-ci-robot commented Jun 9, 2022

tjungblu commented Jun 9, 2022

tjungblu commented Jun 9, 2022

openshift-ci-robot commented Jun 9, 2022

tjungblu commented Jun 9, 2022

tjungblu commented Jun 9, 2022

openshift-ci bot commented Jun 9, 2022

tjungblu commented Jun 9, 2022

tjungblu commented Jun 9, 2022

openshift-ci bot commented Jun 9, 2022 • edited

openshift-ci-robot commented Jun 9, 2022

openshift-ci bot commented Jun 9, 2022

openshift-cherrypick-robot commented Jun 9, 2022

openshift-cherrypick-robot commented Jun 9, 2022

tjungblu commented Jun 1, 2022 •

edited

openshift-ci bot commented Jun 9, 2022 •

edited