-
Notifications
You must be signed in to change notification settings - Fork 39.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Volume metrics exposed in /stats/summary not available in /metrics #34137
Comments
@timothysc I'd be happy to make a PR that adds this, but I'm not sure where I'd need to add this. Would this be a cadvisor patch or a k8s patch? What package might that take place in? |
/cc @timstclair |
Volume stats collection is currently only in the Kubelet, so this would be a Kubernetes patch. I believe we don't currently translate the Summary metrics (including volume) into prometheus metrics. Most stats are translated in cAdvisor directly, and some additional Kubelet specific stats are translated here. I'm not sure whether it makes sense to plumb the Summary through to that metrics collector or not. @dashpole may have more thoughts. |
I don't have a great way of monitoring PVC device usage without adding it to the kubelet |
Is this an accepted feature? |
Up vote for this feature! We also have prometheus set up but it can't monitor PV usage currently due to the missing metrics under the /metrics path. Since kubelet seems to have the data already (/stats/summary), it might be a good idea to expose them under /metrics? |
/sig storage |
I am working on this feature, assign the issue to myself. Will send out a proposal soon. |
@jingxu97 |
I have a proposal in the community kubernetes/community#855 |
@cofyc @javefang - Take a look at the proposal Jing linked above. I'm currently working on the first part of this which is adding PVC information to the kubelet stats summary. After that, we should be able to expose that via prometheus metrics. There was some discussion in the last sig meeting whether the metric exposed needs to be indexed by PVC name or if PodName + Pod Volume Name is sufficient. Can you provide some input on that for your use case? |
hi, what we plan to do in our production cluster are:
For (1), (2), (3), (4) we need volume metrics to be associated with PVC/PV objects. Because users create PVC objects to request PV. Pod name often is generated by controller (e.g. deployment), pod volume name is meaningless outside of Pod. For monitoring and alerting, these metrics is better to be associated with PVC, then uses can know which PVC they created is near full for example. I think VolumeStat indexed by PodName + Pod Volume Name is sufficient as long as VolumeStat includes PVC/PV references. But in |
@jingxu97 Awesome! I will try it in our clusters. |
@cofyc, @vjsamuel @javefang @micahhausler we continue to work on expose volume metrics in Kubernetes. Just want to collect more feedback on this. One question come up is whether users prefer to get volume usage information in
If we choose option 2, will users want to know which pod is using the PVC? (from PVC name to get Pod name?) Thanks! |
To us having the volume stats separate is easier for us to template the
Grafana dashboard. And knowing which pod mounted the PVC is good too so we
can link it to the pod.
Many thanks
Xinghong
…On Tue, 24 Oct 2017 at 01:20, Jing Xu ***@***.***> wrote:
@cofyc <https://github.com/cofyc>, @vjsamuel <https://github.com/vjsamuel>
@javefang <https://github.com/javefang> @micahhausler
<https://github.com/micahhausler> we continue to work on expose volume
metrics in Kubernetes. Just want to collect more feedback on this. One
question come up is whether users prefer to get volume usage information in
1. Inside of Pod stats, volume stats contains PVC reference
(namespace, PVC name)
2. PVC stats is exposed separately from Pod. So the metric has PVC
name (possibly PV name) and the usage/capacity information.
If we choose option 2, will users want to know which pod is using the PVC?
(from PVC name to get Pod name?) Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#34137 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABJRxHO1aP2YgsxA2HjtRTT2cgMCAmtAks5svS1SgaJpZM4KPOIH>
.
|
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
Was this completed in #51553? Or is this different? |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Hello, Thanks! /remove-lifecycle rotten |
I have the same issue on 1.10.0 |
I think after this PR #59170, you need to set up kube-state-metrics to get the metrics |
I have one requirement that is, |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Currently I am using k8s version 1.22.X with runtime as containerd and cgroup as systems. whereas when I use EKS with the same runtime and cgroup, I can see all the volume_stats metrics in /metrics route [http://localhost:8001/api/v1/nodes/ap-south-1.compute.internal/proxy/metrics] |
@novneet03: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Kubernetes Version: 1.3
On hitting
http://node:10255/stats/summary
I am able to see volume metrics like:but the same metrics are not available when I hit
http://node:10255/metrics
.Im under the assumption that all metrics are equally available on both endpoints. We use prometheus today to do our timeseries metric storage and because of this gap we are not able to see volume usage on prometheus.
The text was updated successfully, but these errors were encountered: