Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Pod-level Resource Usage Stats #55978

Closed
jingxu97 opened this issue Nov 17, 2017 · 5 comments
Closed

Add Pod-level Resource Usage Stats #55978

jingxu97 opened this issue Nov 17, 2017 · 5 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. milestone/needs-approval priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Milestone
v1.9

Comments

@jingxu97
Copy link
Contributor

jingxu97 commented Nov 17, 2017 •
edited

With all the features related to pod-level management (pod cgroup, pod-level resource management and local ephemeral storage isolation) added recently, it becomes necessary to provide more Pod-level metrics including CPU, memory and ephemeral storage usage consumed by individual pod. Pod-level accounting could help solve the following issues

  • Pod level monitoring can help account for pod resource usage overhead that is otherwise accounted to the node.
  • For local disk usage, ephemeral storage might be used by volumes (pod-level property) or consumed by containers. Although kubelet reports volume usage at pod-level, there is no way to tell whether the volume is using ephemeral storage or not. So users could not figure out how much total ephemeral storage is used by a pod.

The first step of exposing pod-level is Kubelet Summary API change:

  • CPU and memory: the usage information should be obtained by cAdvisor’s pod-level cgroup
  • Ephemeral-storage: the usage is calculated as the sum of all containers usage and all local ephemeral volume usage including emptyDir (where medium is not memory or hugepages) , configureMap, and downwardAPI.

The next step requires the following changes:

  1. Add pod-level metrics to master metric server so that other k8s components can use the metrics (e.g., kubectl top)
  2. Add to heaspter to scrape the metrics so that stackdriver can use them.
@jingxu97 jingxu97 mentioned this issue Nov 17, 2017
Merged
@k8s-github-robot k8s-github-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Nov 17, 2017
@dims
Copy link
Member

dims commented Nov 18, 2017

/sig node

@k8s-ci-robot k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Nov 18, 2017
@k8s-github-robot k8s-github-robot removed the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Nov 18, 2017
@jingxu97 jingxu97 mentioned this issue Nov 21, 2017
Merged
@jingxu97 jingxu97 added this to the v1.9 milestone Nov 21, 2017
@jingxu97 jingxu97 added kind/feature Categorizes issue or PR as related to a new feature. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Nov 21, 2017
k8s-github-robot pushed a commit that referenced this issue Nov 22, 2017
Merge pull request #55447 from jingxu97/Nov/podmetric
94a8d81 
Automatic merge from submit-queue (batch tested with PRs 55812, 55752, 55447, 55848, 50984). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add Pod-level local ephemeral storage metric in Summary API

This PR adds pod-level ephemeral storage metric into Summary API.
Pod-level ephemeral storage usage is the sum of all containers and local
ephemeral volume including EmptyDir (if not backed up by memory or
hugepages), configueMap, and downwardAPI.
Address issue #55978

**Release note**:
```release-note
Add pod-level local ephemeral storage metric in Summary API. Pod-level ephemeral storage reports the total filesystem usage for the containers and emptyDir volumes in the measured Pod.
```
@jberkus
Copy link

jberkus commented Nov 22, 2017 •
edited

Hey, since the related PR is approved-for-milestone, can a SIG maintainer mark this issue the same way? Thanks.

danehans pushed a commit to danehans/kubernetes that referenced this issue Nov 22, 2017
Merge pull request kubernetes#55969 from jingxu97/Nov/podlevelmetric
e481ff7 
Automatic merge from submit-queue (batch tested with PRs 51321, 55969, 55039, 56183, 55976). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Add pod-level metric for CPU and memory stats

This PR adds the pod-level metrics for CPU and memory stats. cAdvisor
can get all pod cgroup information so we can add this pod-level CPU and
memory stats information from the corresponding pod cgroup
Address issue kubernetes#55978

**Release note**:

```release-note
Add pod-level CPU and memory stats from pod cgroup information
```
@jberkus
Copy link

jberkus commented Nov 27, 2017

Should this issue be resolved, at this point, given the PR merges?

@k8s-github-robot
Copy link

[MILESTONENOTIFIER] Milestone Issue Needs Approval

@jingxu97 @kubernetes/sig-node-misc

Action required: This issue must have the status/approved-for-milestone label applied by a SIG maintainer. If the label is not applied within 0 days, the issue will be moved out of the v1.9 milestone.

Issue Labels
  • sig/node: Issue will be escalated to these SIGs if needed.
  • priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
  • kind/feature: New functionality.
Help

@jingxu97
Copy link
Contributor Author

The changes for Kubernetes 1.9 part is finished, so close this issue.

@lukaszgryglicki lukaszgryglicki mentioned this issue Mar 1, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature. milestone/needs-approval priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
v1.9
Development

No branches or pull requests
5 participants
@dims @jberkus @k8s-github-robot @jingxu97 @k8s-ci-robot