how can i get the "container_fs_io_current"? why is always 0

[TwitchChen]

when i check the CAvisor, the "fs_**_io" is not 0,see the

picture:

but，when it export to the prometheus/metrics ,it become 0:
like this

and the prometheus Graph get 0
why?
@ZhenyangZhao @fabxc

[matthiasr]

As your screenshot shows, it is already zero in the metrics that cAdvisor
exports to Prometheus, so the query result is correct. If cAdvisor does not
plumb this metric through properly, that is probably a bug in cAdvisor. Try
upgrading to the latest version of that to see if this is fixed in the
meantime, and if not, please file an issue there.

Matthias Rampke
Engineer

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany | +49 173
6395215

Managing Director: Alexander Ljung | Incorporated in England & Wales with
Company No. 6343600 | Local Branch Office | AG Charlottenburg | HRB 110657B

[ZhenyangZhao]

@matthiasr
Hi matthiasr. I have the same problem like @Chentongxuedefanhua .
I had checked cadvisor by cadvisor's api "http://192.168.252.198:9104/api/v1.2/containers/docker/00774698bf2e1c098c91ddaa5a20425954ce8007c2c6762b762f2b737098f40a" the disk io is correct. see pic 1
"docker stats 00774698bf2e1c098c91ddaa5a20425954ce8007c2c6762b762f2b737098f40a" is correct also. see pic 2.

which metric is corresponding "docker stats diskio"?

pic 1: cadvisor

pic 2: docker stats

[jimmidyson]

@ZhenyangZhao Prometheus does not use the cadvisor API, it uses the metrics that Prometheus exports at http:///metrics. The problem looks like a bug in cadvisor not properly exporting the disk stats correctly for Prometheus to scrape. Can you share the content of http:///metrics so we can confirm that please? Once we confirm that you will need to raise an issue at https://github.com/google/cadvisor/issues as this looks like a bug in cadvisor, not prometheus.

[ZhenyangZhao]

@jimmidyson
Thanks for your reply.
This is my cadvisor metrics content:

# HELP container_fs_io_current Number of I/Os currently in progress
# TYPE container_fs_io_current gauge
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/99b12bd1cb1c561f03c3966ff359ba63724c09da0c2a473b6d0e55e3e94e7908",image="alpine:3.2",name="sharp_kilby"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/5b329d53c595fc50a025ea2069eed5754ccd342671f5630a6c6436f6e285e730",image="rancher/agent:v1.0.2",io_rancher_container_system="rancher-agent",name="rancher-agent"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/94e473a37f73471b457e0bd9557b6c37e8bed20a2d98ea2f41949689136ffc4a",image="rancher/agent-instance:v0.8.1",io_rancher_container_agent_id="28",io_rancher_container_ip="10.42.76.190/16",io_rancher_container_name="Network Agent",io_rancher_container_system="NetworkAgent",io_rancher_container_uuid="14b2a783-094b-4db9-8dc5-5e8d2f45d778",name="14b2a783-094b-4db9-8dc5-5e8d2f45d778"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4e9094ef694cdb12182d8e6d815a0c7587e4a64a74691f6cec68c32f04a4725f",image="infinityworks/graf-db:08",io_rancher_container_ip="10.42.82.242/16",io_rancher_container_name="hor-monitor_grafana_graf-db_1",io_rancher_container_uuid="5f3a200f-6eaf-4913-a73c-b34703f47c73",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/grafana/graf-db",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_deployment_unit="c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_hash="07503d4e296376ac5a731cc61a6ebb1486e9d5ba",io_rancher_service_launch_config="graf-db",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/grafana/graf-db",name="r-hor-monitor_grafana_graf-db_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/678dc00d22343d0b8618bf58ea38b119b29ee12117ca270eb28fecbf9d33554b",image="tutum/redis:latest",io_rancher_container_ip="10.42.54.49/16",io_rancher_container_name="hor-monitor_metric-websocket_redis_1",io_rancher_container_uuid="be7e2896-0a94-4c87-8dd2-e49125aa78c9",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/metric-websocket/redis",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_deployment_unit="73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_hash="894dcaba8d6ad168528d49cfe22c0c0183da84dc",io_rancher_service_launch_config="redis",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/metric-websocket/redis",name="r-hor-monitor_metric-websocket_redis_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/7980e0e1cacbe23415d70a40c675661e4a765aca3ec9926952d4d51ca681956a",image="neunnhor/prom-conf:1.1",io_rancher_container_ip="10.42.3.132/16",io_rancher_container_name="hor-monitor_prometheus_prom-conf_1",io_rancher_container_uuid="2c3db5bb-281d-47ad-946b-c0101d6a3a79",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus/prom-conf",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="97bc52293d0102e2664740ec9cf55defb1305fba",io_rancher_service_launch_config="prom-conf",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus/prom-conf",name="r-hor-monitor_prometheus_prom-conf_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/00774698bf2e1c098c91ddaa5a20425954ce8007c2c6762b762f2b737098f40a",image="neunnhor/grafana:3.0.1",io_rancher_container_ip="10.42.138.103/16",io_rancher_container_name="hor-monitor_grafana_1",io_rancher_container_uuid="08bc0e20-142b-4a92-a674-c5715643fefc",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/grafana",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_deployment_unit="c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_hash="31252fce9c21cd6ee1f93e230aa4f3c2c68e468f",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="graf-db,grafana-influxdb-config",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/grafana",name="r-hor-monitor_grafana_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4c88b80650f5e4419dd5d79be5b73b4533f535da99fb9e42379077849da98738",image="prom/prometheus:v1.0.1",io_rancher_container_ip="10.42.215.152/16",io_rancher_container_name="hor-monitor_prometheus_1",io_rancher_container_uuid="6a619344-8537-4a36-a7af-fd2d2772da04",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="41c09b3708af0b551c177cf96c05aa0c66885ffd",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="prom-conf, alertmanager",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus",name="r-hor-monitor_prometheus_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/68fda8a0de2ccc29e9be91bb355efc44a836d5ce28a8d52ef332a6dd6cf02221",image="redis:3.0.7",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.48.198/16",io_rancher_container_name="hor-logging_redis_1",io_rancher_container_uuid="bee8076e-1a22-40cd-97b8-a21671c6de8c",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/redis",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="70fe8341-2809-49be-a5da-5fb29ebf68ee",io_rancher_service_hash="9b8321df07e04872d9c5078861e667ddb29fe2ce",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/redis",name="r-hor-logging_redis_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/82d5045946ec20106f889fa923e3b75beec493aa877b4a4f1baff0c9ac306e1f",image="infinityworks/ranch-eye:03",io_rancher_container_dns="true",io_rancher_container_name="hor-monitor_ranch-eye_1",io_rancher_container_uuid="e39242bc-203c-4192-b0e9-cde0a034f2a3",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/ranch-eye",io_rancher_scheduler_global="true",io_rancher_service_deployment_unit="d814911f-906b-426a-90b7-aa1186c8d61e",io_rancher_service_hash="35db067bbba5c8fc7adcaac90c89e9b34c0c1945",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_service_requested_host_id="3",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/ranch-eye",name="r-hor-monitor_ranch-eye_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/a9b4f755c673402b5f2e6ae78764ad164ad377133bda88725c6d50996b408d75",image="neunnhor/metric-websocket:1.0",io_rancher_container_ip="10.42.147.35/16",io_rancher_container_name="hor-monitor_metric-websocket_1",io_rancher_container_uuid="63123521-5058-431e-afc2-5cef3e4e3b7a",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/metric-websocket",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_deployment_unit="73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_hash="02818f2aeaa78b8315ce9c19a4fc0926c348be2a",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="redis",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/metric-websocket",name="r-hor-monitor_metric-websocket_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/b8cfbe2ec39f8545f92caf7d74902a525493fd363719f1026b25f0d289f5eb2d",image="prom/node-exporter:latest",io_rancher_container_dns="true",io_rancher_container_name="hor-monitor_node-exporter_1",io_rancher_container_uuid="810f803e-af92-41e6-b52d-934d8e2ebe6a",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/node-exporter",io_rancher_scheduler_global="true",io_rancher_service_deployment_unit="1a021a8c-38b8-47a7-a6e1-83fc44b62edc",io_rancher_service_hash="e8e5253197824df6bb78a8fbf1f0db066e3faf2b",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_service_requested_host_id="3",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/node-exporter",name="r-hor-monitor_node-exporter_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/f3a5dbd2361aec1deb880025b4a0087851e36a4cef6d050c8dda53ae21c3e321",image="prom/alertmanager:v0.3.0",io_rancher_container_ip="10.42.139.171/16",io_rancher_container_name="hor-monitor_prometheus_alertmanager_1",io_rancher_container_pull_image="always",io_rancher_container_uuid="962595a9-7278-48a4-8560-e34d1f787ab3",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus/alertmanager",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="e2a7a13615add33f09cdec41a9f11f4397e4728e",io_rancher_service_launch_config="alertmanager",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus/alertmanager",name="r-hor-monitor_prometheus_alertmanager_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4d57515c4136ec04bd2b2625f123254ed77932b9c0ec9625211c3802bd0a6871",image="infinityworks/prometheus-rancher-exporter:04",io_rancher_container_agent_id="436",io_rancher_container_agent_role="environment",io_rancher_container_create_agent="true",io_rancher_container_ip="10.42.155.118/16",io_rancher_container_name="hor-monitor_prometheus-rancher-exporter_1",io_rancher_container_uuid="a78d6f7b-1c16-4e79-9784-41076aa11b34",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus-rancher-exporter",io_rancher_service_deployment_unit="1fe28a51-c256-4a69-8e9e-b0ec0b1cfeaf",io_rancher_service_hash="8ea6199ac59637d3126f7b777415d9c4bfda27fe",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus-rancher-exporter",name="r-hor-monitor_prometheus-rancher-exporter_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",elasticsearch_master_config_version="0",id="/docker/fe24a3e7b801416bbbb8e0ed348f535dd40ad78cc5889e775ed21ffdefcede23",image="elasticsearch:2.2.1",io_rancher_container_hostname_override="container_name",io_rancher_container_name="hor-logging_elasticsearch-masters_elasticsearch-base-master_1",io_rancher_container_uuid="8b8ed87b-5f72-454c-ab7c-1a69ec09e52e",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-masters/elasticsearch-base-master",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_service_hash="449f21275160b0b6381c92d36a501cc9ff422aff",io_rancher_service_launch_config="elasticsearch-base-master",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-masters/elasticsearch-base-master",name="r-hor-logging_elasticsearch-masters_elasticsearch-base-master_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/690c946eec029e7e397c769c5f55ac2c3a98f4b16b26f6433bac0bc3d02a3626",image="rancher/elasticsearch-conf:v0.5.0",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.169.182/16",io_rancher_container_name="hor-logging_elasticsearch-masters_1",io_rancher_container_uuid="fb0b4a13-2d7e-4e6e-be6d-9a1614532b1c",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-masters",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_service_hash="bbe92686780ea3f4b96cab579dce4eca1ed93913",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="elasticsearch-base-master,elasticsearch-datavolume-masters",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-masters",name="r-hor-logging_elasticsearch-masters_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",elasticsearch_datanode_config_version="0",id="/docker/9335ed73bf09a92108ba10ca89dff9241be82bea5562022a7195bf2c4abfe508",image="elasticsearch:2.2.1",io_rancher_container_hostname_override="container_name",io_rancher_container_name="hor-logging_elasticsearch-datanodes_elasticsearch-base-datanode_2",io_rancher_container_uuid="fce699f7-7650-4932-a5d5-d7ee75df4025",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-datanodes/elasticsearch-base-datanode",io_rancher_scheduler_affinity_container_label_ne="io.rancher.stack_service.name=${stack_name}/${service_name}",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_service_hash="467860c29f2af09c027c177e2f60579eb60f7de7",io_rancher_service_launch_config="elasticsearch-base-datanode",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-datanodes/elasticsearch-base-datanode",name="r-hor-logging_elasticsearch-datanodes_elasticsearch-base-datanode_2"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/42f0db49c3da0201a439aa890638b10895f837830e1b503b287918e9d1089b71",image="rancher/agent-instance:v0.8.3",io_rancher_container_agent_id="407",io_rancher_container_agent_role="LoadBalancerAgent",io_rancher_container_agent_uri_prefix="delegate",io_rancher_container_create_agent="true",io_rancher_container_ip="10.42.178.11/16",io_rancher_container_name="kibana_kibana-vip_1",io_rancher_container_system="LoadBalancerAgent",io_rancher_container_uuid="78917e86-e429-411b-9c81-62625401ed68",io_rancher_project_name="kibana",io_rancher_project_service_name="kibana/kibana-vip",io_rancher_service_deployment_unit="e2fe8d2c-4b89-476d-b8ca-4246b02c11a5",io_rancher_service_hash="f4a884754b0e81520c2e0df3390dd8d0e9f09bd7",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="kibana",io_rancher_stack_service_name="kibana/kibana-vip",name="r-kibana_kibana-vip_1"} 0
container_fs_io_current{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/b3673928768ced0ce9d7ca45a3bdba227f56f2e96847175a09952f75a3394446",image="rancher/elasticsearch-conf:v0.5.0",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.41.88/16",io_rancher_container_name="hor-logging_elasticsearch-datanodes_2",io_rancher_container_uuid="55554ab6-cc46-4c46-9667-9feb72379237",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-datanodes",io_rancher_scheduler_affinity_container_label_ne="io.rancher.stack_service.name=${stack_name}/${service_name}",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_service_hash="5cea5681cab8eaf833beddc37abbb07243e74ae8",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="elasticsearch-base-datanode,elasticsearch-datavolume-datanode",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-datanodes",name="55554ab6-cc46-4c46-9667-9feb72379237"} 0
# HELP container_fs_io_time_seconds_total Cumulative count of seconds spent doing I/Os
# TYPE container_fs_io_time_seconds_total counter
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/99b12bd1cb1c561f03c3966ff359ba63724c09da0c2a473b6d0e55e3e94e7908",image="alpine:3.2",name="sharp_kilby"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/5b329d53c595fc50a025ea2069eed5754ccd342671f5630a6c6436f6e285e730",image="rancher/agent:v1.0.2",io_rancher_container_system="rancher-agent",name="rancher-agent"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/94e473a37f73471b457e0bd9557b6c37e8bed20a2d98ea2f41949689136ffc4a",image="rancher/agent-instance:v0.8.1",io_rancher_container_agent_id="28",io_rancher_container_ip="10.42.76.190/16",io_rancher_container_name="Network Agent",io_rancher_container_system="NetworkAgent",io_rancher_container_uuid="14b2a783-094b-4db9-8dc5-5e8d2f45d778",name="14b2a783-094b-4db9-8dc5-5e8d2f45d778"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4e9094ef694cdb12182d8e6d815a0c7587e4a64a74691f6cec68c32f04a4725f",image="infinityworks/graf-db:08",io_rancher_container_ip="10.42.82.242/16",io_rancher_container_name="hor-monitor_grafana_graf-db_1",io_rancher_container_uuid="5f3a200f-6eaf-4913-a73c-b34703f47c73",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/grafana/graf-db",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_deployment_unit="c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_hash="07503d4e296376ac5a731cc61a6ebb1486e9d5ba",io_rancher_service_launch_config="graf-db",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/grafana/graf-db",name="r-hor-monitor_grafana_graf-db_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/678dc00d22343d0b8618bf58ea38b119b29ee12117ca270eb28fecbf9d33554b",image="tutum/redis:latest",io_rancher_container_ip="10.42.54.49/16",io_rancher_container_name="hor-monitor_metric-websocket_redis_1",io_rancher_container_uuid="be7e2896-0a94-4c87-8dd2-e49125aa78c9",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/metric-websocket/redis",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_deployment_unit="73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_hash="894dcaba8d6ad168528d49cfe22c0c0183da84dc",io_rancher_service_launch_config="redis",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/metric-websocket/redis",name="r-hor-monitor_metric-websocket_redis_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/7980e0e1cacbe23415d70a40c675661e4a765aca3ec9926952d4d51ca681956a",image="neunnhor/prom-conf:1.1",io_rancher_container_ip="10.42.3.132/16",io_rancher_container_name="hor-monitor_prometheus_prom-conf_1",io_rancher_container_uuid="2c3db5bb-281d-47ad-946b-c0101d6a3a79",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus/prom-conf",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="97bc52293d0102e2664740ec9cf55defb1305fba",io_rancher_service_launch_config="prom-conf",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus/prom-conf",name="r-hor-monitor_prometheus_prom-conf_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/00774698bf2e1c098c91ddaa5a20425954ce8007c2c6762b762f2b737098f40a",image="neunnhor/grafana:3.0.1",io_rancher_container_ip="10.42.138.103/16",io_rancher_container_name="hor-monitor_grafana_1",io_rancher_container_uuid="08bc0e20-142b-4a92-a674-c5715643fefc",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/grafana",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_deployment_unit="c55bd1f8-a95c-404c-bc4e-c018a56e9d32",io_rancher_service_hash="31252fce9c21cd6ee1f93e230aa4f3c2c68e468f",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="graf-db,grafana-influxdb-config",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/grafana",name="r-hor-monitor_grafana_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4c88b80650f5e4419dd5d79be5b73b4533f535da99fb9e42379077849da98738",image="prom/prometheus:v1.0.1",io_rancher_container_ip="10.42.215.152/16",io_rancher_container_name="hor-monitor_prometheus_1",io_rancher_container_uuid="6a619344-8537-4a36-a7af-fd2d2772da04",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="41c09b3708af0b551c177cf96c05aa0c66885ffd",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="prom-conf, alertmanager",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus",name="r-hor-monitor_prometheus_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/68fda8a0de2ccc29e9be91bb355efc44a836d5ce28a8d52ef332a6dd6cf02221",image="redis:3.0.7",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.48.198/16",io_rancher_container_name="hor-logging_redis_1",io_rancher_container_uuid="bee8076e-1a22-40cd-97b8-a21671c6de8c",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/redis",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="70fe8341-2809-49be-a5da-5fb29ebf68ee",io_rancher_service_hash="9b8321df07e04872d9c5078861e667ddb29fe2ce",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/redis",name="r-hor-logging_redis_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/82d5045946ec20106f889fa923e3b75beec493aa877b4a4f1baff0c9ac306e1f",image="infinityworks/ranch-eye:03",io_rancher_container_dns="true",io_rancher_container_name="hor-monitor_ranch-eye_1",io_rancher_container_uuid="e39242bc-203c-4192-b0e9-cde0a034f2a3",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/ranch-eye",io_rancher_scheduler_global="true",io_rancher_service_deployment_unit="d814911f-906b-426a-90b7-aa1186c8d61e",io_rancher_service_hash="35db067bbba5c8fc7adcaac90c89e9b34c0c1945",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_service_requested_host_id="3",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/ranch-eye",name="r-hor-monitor_ranch-eye_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/a9b4f755c673402b5f2e6ae78764ad164ad377133bda88725c6d50996b408d75",image="neunnhor/metric-websocket:1.0",io_rancher_container_ip="10.42.147.35/16",io_rancher_container_name="hor-monitor_metric-websocket_1",io_rancher_container_uuid="63123521-5058-431e-afc2-5cef3e4e3b7a",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/metric-websocket",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_deployment_unit="73f1e86c-b34c-465c-b758-ae523f83846b",io_rancher_service_hash="02818f2aeaa78b8315ce9c19a4fc0926c348be2a",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="redis",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/metric-websocket",name="r-hor-monitor_metric-websocket_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/b8cfbe2ec39f8545f92caf7d74902a525493fd363719f1026b25f0d289f5eb2d",image="prom/node-exporter:latest",io_rancher_container_dns="true",io_rancher_container_name="hor-monitor_node-exporter_1",io_rancher_container_uuid="810f803e-af92-41e6-b52d-934d8e2ebe6a",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/node-exporter",io_rancher_scheduler_global="true",io_rancher_service_deployment_unit="1a021a8c-38b8-47a7-a6e1-83fc44b62edc",io_rancher_service_hash="e8e5253197824df6bb78a8fbf1f0db066e3faf2b",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_service_requested_host_id="3",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/node-exporter",name="r-hor-monitor_node-exporter_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/f3a5dbd2361aec1deb880025b4a0087851e36a4cef6d050c8dda53ae21c3e321",image="prom/alertmanager:v0.3.0",io_rancher_container_ip="10.42.139.171/16",io_rancher_container_name="hor-monitor_prometheus_alertmanager_1",io_rancher_container_pull_image="always",io_rancher_container_uuid="962595a9-7278-48a4-8560-e34d1f787ab3",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus/alertmanager",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_deployment_unit="c1c3f624-9fa4-4ff6-b6fc-bc258e300833",io_rancher_service_hash="e2a7a13615add33f09cdec41a9f11f4397e4728e",io_rancher_service_launch_config="alertmanager",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus/alertmanager",name="r-hor-monitor_prometheus_alertmanager_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/4d57515c4136ec04bd2b2625f123254ed77932b9c0ec9625211c3802bd0a6871",image="infinityworks/prometheus-rancher-exporter:04",io_rancher_container_agent_id="436",io_rancher_container_agent_role="environment",io_rancher_container_create_agent="true",io_rancher_container_ip="10.42.155.118/16",io_rancher_container_name="hor-monitor_prometheus-rancher-exporter_1",io_rancher_container_uuid="a78d6f7b-1c16-4e79-9784-41076aa11b34",io_rancher_project_name="hor-monitor",io_rancher_project_service_name="hor-monitor/prometheus-rancher-exporter",io_rancher_service_deployment_unit="1fe28a51-c256-4a69-8e9e-b0ec0b1cfeaf",io_rancher_service_hash="8ea6199ac59637d3126f7b777415d9c4bfda27fe",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="hor-monitor",io_rancher_stack_service_name="hor-monitor/prometheus-rancher-exporter",name="r-hor-monitor_prometheus-rancher-exporter_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",elasticsearch_master_config_version="0",id="/docker/fe24a3e7b801416bbbb8e0ed348f535dd40ad78cc5889e775ed21ffdefcede23",image="elasticsearch:2.2.1",io_rancher_container_hostname_override="container_name",io_rancher_container_name="hor-logging_elasticsearch-masters_elasticsearch-base-master_1",io_rancher_container_uuid="8b8ed87b-5f72-454c-ab7c-1a69ec09e52e",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-masters/elasticsearch-base-master",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_service_hash="449f21275160b0b6381c92d36a501cc9ff422aff",io_rancher_service_launch_config="elasticsearch-base-master",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-masters/elasticsearch-base-master",name="r-hor-logging_elasticsearch-masters_elasticsearch-base-master_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/690c946eec029e7e397c769c5f55ac2c3a98f4b16b26f6433bac0bc3d02a3626",image="rancher/elasticsearch-conf:v0.5.0",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.169.182/16",io_rancher_container_name="hor-logging_elasticsearch-masters_1",io_rancher_container_uuid="fb0b4a13-2d7e-4e6e-be6d-9a1614532b1c",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-masters",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="884c32d0-c5ba-49b4-a1e3-bcb0d5ea0db2",io_rancher_service_hash="bbe92686780ea3f4b96cab579dce4eca1ed93913",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="elasticsearch-base-master,elasticsearch-datavolume-masters",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-masters",name="r-hor-logging_elasticsearch-masters_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",elasticsearch_datanode_config_version="0",id="/docker/9335ed73bf09a92108ba10ca89dff9241be82bea5562022a7195bf2c4abfe508",image="elasticsearch:2.2.1",io_rancher_container_hostname_override="container_name",io_rancher_container_name="hor-logging_elasticsearch-datanodes_elasticsearch-base-datanode_2",io_rancher_container_uuid="fce699f7-7650-4932-a5d5-d7ee75df4025",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-datanodes/elasticsearch-base-datanode",io_rancher_scheduler_affinity_container_label_ne="io.rancher.stack_service.name=${stack_name}/${service_name}",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_service_hash="467860c29f2af09c027c177e2f60579eb60f7de7",io_rancher_service_launch_config="elasticsearch-base-datanode",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-datanodes/elasticsearch-base-datanode",name="r-hor-logging_elasticsearch-datanodes_elasticsearch-base-datanode_2"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/42f0db49c3da0201a439aa890638b10895f837830e1b503b287918e9d1089b71",image="rancher/agent-instance:v0.8.3",io_rancher_container_agent_id="407",io_rancher_container_agent_role="LoadBalancerAgent",io_rancher_container_agent_uri_prefix="delegate",io_rancher_container_create_agent="true",io_rancher_container_ip="10.42.178.11/16",io_rancher_container_name="kibana_kibana-vip_1",io_rancher_container_system="LoadBalancerAgent",io_rancher_container_uuid="78917e86-e429-411b-9c81-62625401ed68",io_rancher_project_name="kibana",io_rancher_project_service_name="kibana/kibana-vip",io_rancher_service_deployment_unit="e2fe8d2c-4b89-476d-b8ca-4246b02c11a5",io_rancher_service_hash="f4a884754b0e81520c2e0df3390dd8d0e9f09bd7",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_stack_name="kibana",io_rancher_stack_service_name="kibana/kibana-vip",name="r-kibana_kibana-vip_1"} 0
container_fs_io_time_seconds_total{device="/dev/disk/by-label/cloudimg-rootfs",id="/docker/b3673928768ced0ce9d7ca45a3bdba227f56f2e96847175a09952f75a3394446",image="rancher/elasticsearch-conf:v0.5.0",io_rancher_container_hostname_override="container_name",io_rancher_container_ip="10.42.41.88/16",io_rancher_container_name="hor-logging_elasticsearch-datanodes_2",io_rancher_container_uuid="55554ab6-cc46-4c46-9667-9feb72379237",io_rancher_project_name="hor-logging",io_rancher_project_service_name="hor-logging/elasticsearch-datanodes",io_rancher_scheduler_affinity_container_label_ne="io.rancher.stack_service.name=${stack_name}/${service_name}",io_rancher_scheduler_affinity_container_label_soft="io.rancher.service.deployment.unit=00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_scheduler_affinity_host_label="com.neunn.host.service.logging=true",io_rancher_service_deployment_unit="00286b61-2ffb-4692-b864-10ccb7f84679",io_rancher_service_hash="5cea5681cab8eaf833beddc37abbb07243e74ae8",io_rancher_service_launch_config="io.rancher.service.primary.launch.config",io_rancher_sidekicks="elasticsearch-base-datanode,elasticsearch-datavolume-datanode",io_rancher_stack_name="hor-logging",io_rancher_stack_service_name="hor-logging/elasticsearch-datanodes",name="55554ab6-cc46-4c46-9667-9feb72379237"} 0

[fabxc]

This indeed seems to be a cAdvisor issue. I opened google/cadvisor#1412 and am closing here.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.