Skip to content

container_fs_* stats always 0 for all containers created by k8s #55397

@discordianfish

Description

@discordianfish

/kind bug

What happened:
All cadvisor container_fs_* stats are 0, except a few without pod/image labels which I assume are for the root cgroup:


container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="pool-small",container_name="prometheus-to-sd-exporter",device="/dev/sda1",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/kubepods/burstable/pode93369ea-c40b-11e7-bff5-42010af0018b/3f76690535157c729524b48dbaab3f1788d680ad5d859768dc06c05e9ccd5610",image="asia.gcr.io/google-containers/prometheus-to-sd@sha256:c6aaa681e77e55aa7f7017ca55265accde313f8e2e5484ee1d0a4d89ff741c48",instance="gke-latency-at-pool-small-3338bda2-3c8g",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-pool-small-3338bda2-3c8g",name="k8s_prometheus-to-sd-exporter_fluentd-gcp-v2.0.9-h7tl7_kube-system_e93369ea-c40b-11e7-bff5-42010af0018b_0",namespace="kube-system",pod_name="fluentd-gcp-v2.0.9-h7tl7"} | 0
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="pool-small",container_name="prometheus-to-sd-exporter",device="/dev/sda1",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/kubepods/burstable/podfc87c089-c3dd-11e7-bff5-42010af0018b/9f3cdcf5306a7a42fd88f8bbe690a15d964010eebcd3cd681d40f785a3dd8107",image="asia.gcr.io/google-containers/prometheus-to-sd@sha256:c6aaa681e77e55aa7f7017ca55265accde313f8e2e5484ee1d0a4d89ff741c48",instance="gke-latency-at-pool-small-3338bda2-2cg2",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-pool-small-3338bda2-2cg2",name="k8s_prometheus-to-sd-exporter_fluentd-gcp-v2.0.9-jb7ll_kube-system_fc87c089-c3dd-11e7-bff5-42010af0018b_0",namespace="kube-system",pod_name="fluentd-gcp-v2.0.9-jb7ll"} | 0
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="pool-small",container_name="sidecar",device="/dev/sda1",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/kubepods/burstable/poddee71937-c472-11e7-bff5-42010af0018b/0584c59990c2bd3efe011f4bf5d755659567b3abb2c1a42c7ef49c7b11a69a3a",image="asia.gcr.io/google_containers/k8s-dns-sidecar-amd64@sha256:9aab42bf6a2a068b797fe7d91a5d8d915b10dbbc3d6f2b10492848debfba6044",instance="gke-latency-at-pool-small-3338bda2-3c8g",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-pool-small-3338bda2-3c8g",name="k8s_sidecar_kube-dns-4031738344-k8dfb_kube-system_dee71937-c472-11e7-bff5-42010af0018b_0",namespace="kube-system",pod_name="kube-dns-4031738344-k8dfb"} | 0
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="pool-small",container_name="tiller",device="/dev/sda1",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/kubepods/besteffort/pod13ca351f-c484-11e7-bff5-42010af0018b/1315c8d1d2ada471cd0d4636f21766499feb5c66b558425adbf0cf19ec416865",image="gcr.io/kubernetes-helm/tiller@sha256:82677f561f8dd67b6095fe7b9646e6913ee99e1d6fdf86705adbf99a69a7d744",instance="gke-latency-at-pool-small-3338bda2-3c8g",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-pool-small-3338bda2-3c8g",name="k8s_tiller_tiller-deploy-3066893457-jcwwg_kube-system_13ca351f-c484-11e7-bff5-42010af0018b_0",namespace="kube-system",pod_name="tiller-deploy-3066893457-jcwwg"} | 0
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="small-preemptible",cloud_google_com_gke_preemptible="true",container_name="",device="/dev/root",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/",image="",instance="gke-latency-at-small-preemptible-0c981b61-1x4z",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-small-preemptible-0c981b61-1x4z",name="",namespace="",pod_name=""} | 66386
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="small-preemptible",cloud_google_com_gke_preemptible="true",container_name="",device="/dev/root",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/",image="",instance="gke-latency-at-small-preemptible-0c981b61-9489",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-small-preemptible-0c981b61-9489",name="",namespace="",pod_name=""} | 66386
container_fs_inodes_free{beta_kubernetes_io_arch="amd64",beta_kubernetes_io_fluentd_ds_ready="true",beta_kubernetes_io_instance_type="g1-small",beta_kubernetes_io_os="linux",cloud_google_com_gke_nodepool="small-preemptible",cloud_google_com_gke_preemptible="true",container_name="",device="/dev/root",failure_domain_beta_kubernetes_io_region="us-east1",failure_domain_beta_kubernetes_io_zone="us-east1-c",id="/",image="",instance="gke-latency-at-small-preemptible-0c981b61-99zh",job="kubernetes-cadvisor",kubernetes_io_hostname="gke-latency-at-small-preemptible-0c981b61-99zh",name="",namespace="",pod_name=""} | 66386

What you expected to happen:
Have these stats return right values..

How to reproduce it (as minimally and precisely as possible):
As far as I can tell, this should happen on a fresh install too.

Environment:

  • Kubernetes version (use kubectl version): 1.8.1
  • Cloud provider or hardware configuration: GKE

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.sig/nodeCategorizes an issue or PR as relevant to SIG Node.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions