Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(metrics): drop cadvisor container metrics without metadata #2918

Merged
merged 1 commit into from Mar 8, 2023

Conversation

swiatekm-sumo
Copy link
Contributor

@swiatekm-sumo swiatekm-sumo commented Mar 8, 2023

Cadvisor container metrics are emitted for all cgroups, even ones not actually related to containers run by the kubelet. For some of these, the only label is the id containing the cgroup path:

$ kubectl get --raw "/api/v1/nodes/sumologic-control-plane/proxy/metrics/cadvisor" | grep container_cpu_usage_seconds_total
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="",cpu="total",id="/",image="",name="",namespace="",pod=""} 676.209622261 1678268968878
container_cpu_usage_seconds_total{container="",cpu="total",id="/docker/bc61f8197b2e99d91a3e3da589e9c13dbfd2756a4e7f9903a2b76c4e6b0b5fda/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod3d6b3bf5_5b1c_407c_8842_04565a00d150.slice/cri-containerd-3ac92040b58042a3828de42432948019484832e0f20e03da5c1b09b5bc7d8f13.scope",image="k8s.gcr.io/pause:3.6",name="3ac92040b58042a3828de42432948019484832e0f20e03da5c1b09b5bc7d8f13",namespace="kube-system",pod="kube-proxy-2ccmm"} 0.01729504 1678268965373
container_cpu_usage_seconds_total{container="",cpu="total",id="/docker/bc61f8197b2e99d91a3e3da589e9c13dbfd2756a4e7f9903a2b76c4e6b0b5fda/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-besteffort.slice/kubelet-kubepods-besteffort-pod9b9253b0_57a3_4ead_97c4_fb77f5c3c9a6.slice/cri-containerd-1c4169329d281834a35068176f49a6491b77d54bd0284670a8af040ce03883f0.scope",image="k8s.gcr.io/pause:3.6",name="1c4169329d281834a35068176f49a6491b77d54bd0284670a8af040ce03883f0",namespace="local-path-storage",pod="local-path-provisioner-9cd9bd544-79qtd"} 0.049625352 1678268959599
container_cpu_usage_seconds_total{container="",cpu="total",id="/docker/bc61f8197b2e99d91a3e3da589e9c13dbfd2756a4e7f9903a2b76c4e6b0b5fda/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod4f661fbe2c2911480854bb5e7af7e447.slice/cri-containerd-6c5627a36b6a91e1c8539e128ae6a878ddb9fc74b5775a2dc67df028fb55bcf1.scope",image="k8s.gcr.io/pause:3.6",name="6c5627a36b6a91e1c8539e128ae6a878ddb9fc74b5775a2dc67df028fb55bcf1",namespace="kube-system",pod="etcd-sumologic-control-plane"} 0.056957712 1678268962995
container_cpu_usage_seconds_total{container="",cpu="total",id="/docker/bc61f8197b2e99d91a3e3da589e9c13dbfd2756a4e7f9903a2b76c4e6b0b5fda/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod5536bf0d7cbf6795298304a1be19d4e4.slice/cri-containerd-055241d4e9b99c69abdd34c9bedf2619900409d22b3a00835880331fb50a93f6.scope",image="k8s.gcr.io/pause:3.6",name="055241d4e9b99c69abdd34c9bedf2619900409d22b3a00835880331fb50a93f6",namespace="kube-system",pod="kube-apiserver-sumologic-control-plane"} 0.058457856 1678268971417
container_cpu_usage_seconds_total{container="",cpu="total",id="/docker/bc61f8197b2e99d91a3e3da589e9c13dbfd2756a4e7f9903a2b76c4e6b0b5fda/kubelet.slice/kubelet-kubepods.slice/kubelet-kubepods-burstable.slice/kubelet-kubepods-burstable-pod70ba3f84_8f81_43fa_80e6_65ceb572df07.slice/cri-containerd-87f5879164f9484fabbbd1a21e69cfaf16ee889af78d9a3e3fcf482991e4150d.scope",image="k8s.gcr.io/pause:3.6",name="87f5879164f9484fabbbd1a21e69cfaf16ee889af78d9a3e3fcf482991e4150d",namespace="kube-system",pod="coredns-6d4b75cb6d-j2q9k"} 0.052671836 1678268961640
...

We've used this setting internally for a while as well.

Checklist

  • Changelog updated or skip changelog label added

@swiatekm-sumo swiatekm-sumo marked this pull request as ready for review March 8, 2023 10:15
@swiatekm-sumo swiatekm-sumo requested a review from a team as a code owner March 8, 2023 10:15
@swiatekm-sumo swiatekm-sumo merged commit 55a1bde into main Mar 8, 2023
@swiatekm-sumo swiatekm-sumo deleted the fix/metrics/kubelet-metrics branch March 8, 2023 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants