Kublet embedded cAdvisor outputs empty container_name fields

[IvanovOleg]

I am trying to get the container CPU usage information in grafana, but getting emty diagrams. Grafana has the next dashboard request:
sum by (container_name) (rate(container_cpu_usage_seconds_total{job=\"kubelet\", namespace=\"$namespace\", image!=\"\",container_name!=\"POD\",pod_name=\"$pod\"}[1m]))
Here is the part of the cAdvisor output:
container_cpu_usage_seconds_total{container_name="",cpu="total",id="/kubepods/burstable/pod6c90ee67-83c5-11e9-94aa-000d3a218149",image="",name="",namespace="default",pod_name="cassandra-1"} 23630.333620239
As you can see, there is the empty container_name field. Can anyone help?

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:53:53Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"abdda3f9fefa29172298a2e42f5102e777a8ec25", GitTreeState:"clean", BuildDate:"2019-05-08T13:46:28Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Azure
• OS (e.g: cat /etc/os-release): CoreOS stable (2079.3.0)
• Kernel (e.g. uname -a): Linux tmp-plus1connect-k8s-master-0 4.19.34-coreos #1 SMP Mon Apr 22 20:32:34 -00 2019 x86_64 Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz GenuineIntel GNU/Linux
• Install tools: Terraform
• Network plugin and version (if this is a network-related bug): Azure-CNI
• Others:

[dashpole]

That looks like a metric for the pod's cgroup. container_name=="POD" is the "pause" container for the pod.

[IvanovOleg]

@dashpole cadvisor outputs container_name="". In the grafana dashboard there is a filter container_name!="POD", not container_name=="POD". In the prometeus I see no container_name field.

[dashpole]

Right, there are two seperate metrics. One for the "pause container", which has container_name=="POD", and one for the pod cgroup, which has container_name==""

[IvanovOleg]

@dashpole Looks like grafana is able to build a separate graph for each container in the pod, but as long as container_name="" it shows nothing. As far as I know container_name field shoud not be empty. cAdvisor outputs container_name="" for all entries.

[bboreham]

Do you have a nonstandard Docker root directory? If so see #1932

[IvanovOleg]

@bboreham I am using the containerd runtime. Maybe because of that I have no metadata.

[daxwang]

same promblem, the container_name in some node cadvisor metrics is not empty, but a lot is empty.
is there any may to fix it?

[IvanovOleg]

@daxwang I've fixed that by updating containerd to the latest version

[cgiraldo]

@IvanovOleg which containerd version?

We are hitting the same problem in k3s v1.0.0 that uses containerd v1.3.0-k3s.4.

[IvanovOleg]

@cgiraldo It was containerd 1.2.7, CoreOs, kubernetes 1.14.6, and remote runtime configuration for kubelets that points to the containerd (no docker on hosts).

[chenkaiyue]

@daxwang I've fixed that by updating containerd to the latest version

@IvanovOleg Which containerd version you upgrade to?

[IvanovOleg]

@chenkaiyue 1.2.7

[jordy1024]

Right, there are two seperate metrics. One for the "pause container", which has container_name=="POD", and one for the pod cgroup, which has container_name==""

@dashpole
Excuse me, the container named POD is the famous pause container? Are there official documents related to it?
The condition container_name!="POD" means that the metadata of the pause container is filtered out, right?

[dashpole]

Excuse me, the container named POD is the famous pause container? Are there official documents related to it?
The condition container_name!="POD" means that the metadata of the pause container is filtered out, right?

Just to be precise, it filters out metric streams for the pause container, not metadata generally. Most people, if they want to graph the containers in their pod, don't want to see resource usage for the pause container, as it doesn't do much. The name of the pause container is an implementation detail of some container runtimes, but doesn't apply to all, and isn't guaranteed to stick around.

[jordy1024]

@dashpole Thank you,Let me understand it again and consult you if you have any questions;

[bboreham]

There's a related discussion here with a bit more background: containerd/cri#922