Bug 1907998: Collect /metrics/resources from scheduler #283
Conversation
smarterclayton
force-pushed
the
collect_pod_resources
branch
from
80e3e20
to
517278c
Compare
smarterclayton
force-pushed
the
collect_pod_resources
branch
from
517278c
to
710c063
Compare
|
/retest |
smarterclayton
changed the title
openshift-ci-robot
added
bugzilla/severity-high
|
@smarterclayton: This pull request references Bugzilla bug 1907998, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
This is ready for review @lilic / @s-urbaniak. I want to add a recording rule that sums resource requests and returns for workloads on the cluster and will do that in a follow up PR. Manually verified cardinality as described in the discussion in e2e run and verified the scrape returned the correct data. I believe low risk for 4.7 but you're call. |
|
Nice! 🎉 I agree the risk is low here, an in the worst case we revert this change as we don't use these metrics in any rules yet. /hold /retest |
openshift-ci-robot
added
the
do-not-merge/hold
|
/retest |
|
/hold cancel |
openshift-ci-robot
removed
the
do-not-merge/hold
|
/refresh |
|
@smarterclayton: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest |
|
/approve |
smarterclayton
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: lilic, s-urbaniak, smarterclayton The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
6 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
5 similar comments
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
As part of KEP 1916, resource consumption from pods will be tracked by the scheduler and reported. This commit allows that endpoint to be be scraped. It exposes roughly one series per pod since the average pod requests ~1.1 resources.
Verify that both metrics endpoints report, and fix the test because the metric we previously tested against 1.19 was removed in 1.20.
smarterclayton
force-pushed
the
collect_pod_resources
branch
from
710c063
to
0ce5b4d
Compare
|
Second commit deals with - kubernetes/kubernetes#94884 which in 1.20 removed the metric we test (just came up because i had the first PR to touch metrics) |
|
/retest |
|
@smarterclayton: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@smarterclayton: All pull requests linked via external trackers have merged: Bugzilla bug 1907998 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
As part of KEP 1916, resource consumption from pods will be tracked by the scheduler and reported. This commit will allow that endpoint to be scraped once that code is available. This enables those metrics to be scraped. It adds roughly 1 series per pod on average (pods usually declare ~1 resource request on average).
From the e2e run, showing the sum of requested resources for the cluster
Subsequent releases will add resource capacity dashboards.