New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG 1875041: UPSTREAM: 94134: Make similar buckets for api and etcd request duration histogram #331
BUG 1875041: UPSTREAM: 94134: Make similar buckets for api and etcd request duration histogram #331
Conversation
…on histogram Make similar buckets for the apiserver_request_duration_seconds and the etcd_request_duration_seconds histogram so that the result is more comparable side by side. etcd_request_duration_seconds uses the default buckets provided by prometheus client library: DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10} apiserver_request_duration_seconds on the other hand uses more fine grained buckets, and the maximum bucket size is 60s. Both histograms should use similar bucket sizes so they are more comparable side by side.
@tkashem: This pull request references Bugzilla bug 1875041, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/retest |
1 similar comment
/retest |
/retest |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sttts, tkashem The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/retest Please review the full test history for this PR and help us cut down flakes. |
3 similar comments
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
/retest Please review the full test history for this PR and help us cut down flakes. |
@tkashem: All pull requests linked via external trackers have merged: Bugzilla bug 1875041 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What type of PR is this?
/kind bug
What this PR does / why we need it:
etcd_request_duration_seconds
uses the default buckets provided by prometheus client library.DefBuckets = []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10}
The maximum bucket size is
10s
. On the other hand,apiserver_request_duration_seconds
uses more fine grained bucket sizes and the maximum bucket size is60s
The left panel shows latency for
Deployment-DELETE
api (metric=apiserver_request_duration_seconds
), this is taking about40s
to complete. On the other hand, etcd latency (metric=etcd_request_duration_seconds
) for the same objectapps.Deployment-delete
is capped at10s
. Now the difference in latency is hard to account for. It cloud be latency from ectd but we can't answer this question by looking at the metrics.If the etcd metric has similar bucket sizes, we could account for the difference in latency.
This PR makes the bucket sizes for both metrics similar. Also, no existing bucket for
etcd_request_duration_seconds
was dropped.Does this PR introduce a user-facing change?:
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: