-
Notifications
You must be signed in to change notification settings - Fork 38.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Work-around for missing memory metrics on CRI-O exited containers #88734
Conversation
HPA needs metrics for exited init containers before it will take action. By setting memory and CPU usage to zero for any containers that cAdvisor didn't provide statistics for, we are assured that HPA will be able to correctly calculate pod resource usage.
/cc @dashpole |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we do this for all metrics (e.g. disk as well) or do you think we should only do so for cpu and memory?
} else { | ||
memoryStats = &statsapi.MemoryStats{ | ||
Time: metav1.NewTime(cstat.Timestamp), | ||
WorkingSetBytes: uint64Ptr(0), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we default other fields to 0 to have consistent behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps. I was just trying to match the behavior in the CRI stats provider, here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L537
As to the question of populating other missing metrics like disk and network, I'm not sure. It's certainly worth thinking about. |
/priority important-soon |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dashpole, joelsmith The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@dashpole @derekwaynecarr thoughts on getting this into 1.18? |
@rphillips I am in favor. |
its a bug fix that impacts users. /milestone v1.18 |
/retest |
@joelsmith: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
@joelsmith, @dashpole @derekwaynecarr, we are running into this and are currently about to upgrade from 1.15 to 1.6. Can we get this cherry-picked back to 1.17 and 1.16? |
…734-upstream-release-1.16 Automated cherry pick of #88734: Work-around for missing metrics on CRI-O exited containers
…734-upstream-release-1.17 Automated cherry pick of #88734: Work-around for missing metrics on CRI-O exited containers
What type of PR is this?
/kind bug
What this PR does / why we need it:
HPA needs metrics for exited init containers before it will take action. By setting memory and CPU usage to zero for any containers that cAdvisor didn't provide statistics for, we are assured that HPA will be able to correctly calculate pod resource usage.
This PR essentially makes the kubelet's cAdvisor stats provider behave the same as its CRI stats provider (see #74336).
Which issue(s) this PR fixes:
No issue filed.
Special notes for your reviewer:
N/A
Does this PR introduce a user-facing change?:
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:
N/A