Missing 'path' key for '/' mount with [[inputs.disk]] in Docker

[n3v3rf411]

Bug report

Monitoring Docker Engine host within a Docker container doesn't have the 'path' key for '/' mount. I was following the FAQ here: https://github.com/influxdata/telegraf/blob/1.5.1/docs/FAQ.md

Relevant telegraf.conf:

[global_tags]
  dc = "denver-1"

[agent]
  interval = "10s"

# OUTPUTS
[[outputs.file]]
  files = ["stdout", "/tmp/metrics.out"]
   data_format = "json"

# INPUTS
[[inputs.cpu]]
  percpu = true
  totalcpu = false
  # filter all fields beginning with 'time_'
  fielddrop = ["time_*"]

[[inputs.system]]

[[inputs.disk]]
  # *** uncommenting mount_points below shows no disk metrics because there is no path for the / mount in the output ***
  #mount_points = ["/"]
  ignore_fs = ["tmpfs", "devtmpfs", "overlay"]

System info:

Using telegraf:1.5.1-alpine and tested on telegraf:1.5.1 as well on Amazon AMI EC2 instance. Docker version 17.09.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3.

Steps to reproduce:

1. Create telegraf.conf with the contents above.
2. Execute docker run --rm --name telegraf -v $(pwd)/telegraf.conf:/etc/telegraf/telegraf.conf:ro -v /:/hostfs:ro -v /etc:/hostfs/etc:ro -v /proc:/hostfs/proc:ro -v /sys:/hostfs/sys:ro -v /var/run/utmp:/var/run/utmp:ro -e HOST_ETC=/hostfs/etc -e HOST_PROC=/hostfs/proc -e HOST_SYS=/hostfs/sys -e HOST_MOUNT_PREFIX=/hostfs telegraf:1.5.1-alpine

Expected behavior:

{"fields":{"free":104028594176,"inodes_free":6504742,"inodes_total":6553600,"inodes_used":48858,"total":105553080320,"used":1421832192,"used_percent":1.3483418142266226},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"411ba9d18ae8","mode":"ro","path":"/"},"timestamp":1516959270}

Actual behavior:

Missing path:

{"fields":{"free":104028594176,"inodes_free":6504742,"inodes_total":6553600,"inodes_used":48858,"total":105553080320,"used":1421832192,"used_percent":1.3483418142266226},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"411ba9d18ae8","mode":"ro"},"timestamp":1516959270}

Additional info:

2018/01/29 01:30:42 I! Using config file: /etc/telegraf/telegraf.conf
2018-01-29T01:30:42Z I! Starting Telegraf v1.5.1
2018-01-29T01:30:42Z I! Loaded outputs: file
2018-01-29T01:30:42Z I! Loaded inputs: inputs.cpu inputs.system inputs.disk
2018-01-29T01:30:42Z I! Tags enabled: dc=denver-1 host=76c9728997be
2018-01-29T01:30:42Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"76c9728997be", Flush Interval:10s
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/var/lib/docker/plugins"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/var/lib/docker/overlay2"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/etc"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/resolv.conf"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/hostname"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/hosts"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/var/run/utmp"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/etc/telegraf/telegraf.conf"},"timestamp":1517189450}
{"fields":{"load1":0.02,"load15":0,"load5":0.02,"n_cpus":1,"n_users":1},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189450}
{"fields":{"uptime":1177},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189450}
{"fields":{"uptime_format":" 0:19"},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189450}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/var/lib/docker/plugins"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/var/lib/docker/overlay2"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/etc"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/resolv.conf"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/hostname"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"rw","path":"/etc/hosts"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/var/run/utmp"},"timestamp":1517189460}
{"fields":{"free":103941640192,"inodes_free":6500473,"inodes_total":6553600,"inodes_used":53127,"total":105553080320,"used":1508786176,"used_percent":1.4308013992609674},"name":"disk","tags":{"dc":"denver-1","device":"xvda1","fstype":"ext4","host":"76c9728997be","mode":"ro","path":"/etc/telegraf/telegraf.conf"},"timestamp":1517189460}
{"fields":{"load1":0.01,"load15":0,"load5":0.02,"n_cpus":1,"n_users":1},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189460}
{"fields":{"uptime":1187},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189460}
{"fields":{"uptime_format":" 0:19"},"name":"system","tags":{"dc":"denver-1","host":"76c9728997be"},"timestamp":1517189460}
{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.20477137177011,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.39761431411530185,"usage_user":0.39761431411530185},"name":"cpu","tags":{"cpu":"cpu0","dc":"denver-1","host":"76c9728997be"},"timestamp":1517189460}
^C2018-01-29T01:31:03Z I! Hang on, flushing any cached metrics before shutdown

[danielnelson]

Can you without HOST_MOUNT_PREFIX set and paste the output? You may wish to run with these options added on the end so that it will exit after one collection: --input-filter disk --test

[n3v3rf411]

Without HOST_MOUNT_PREFIX:

# docker run --rm --name telegraf     -v /:/hostfs:ro -v /etc:/hostfs/etc:ro -v /proc:/hostfs/proc:ro -v /sys:/hostfs/sys:ro -v /var/run/utmp:/var/run/utmp:ro -e HOST_ETC=/hostfs/etc -e HOST_PROC=/hostfs/proc -e HOST_SYS=/hostfs/sys telegraf:1.5.1-alpine --input-filter disk --test
2018/01/30 09:06:36 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.disk, Collection 1
> disk,path=/,device=overlay,fstype=overlay,mode=rw,host=663d27b3f73f total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i 1517303196000000000
> disk,path=/hostfs,device=xvda1,fstype=ext4,mode=ro,host=663d27b3f73f total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i 1517303196000000000
> disk,path=/hostfs/var/lib/docker/plugins,device=xvda1,fstype=ext4,mode=rw,host=663d27b3f73f inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426 1517303196000000000
> disk,host=663d27b3f73f,path=/hostfs/var/lib/docker/overlay2,device=xvda1,fstype=ext4,mode=rw inodes_used=53131i,total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i 1517303196000000000
> disk,path=/hostfs/var/lib/docker/overlay2/c6ac6a3cd3eafa07840b43cad7168660687cc57c43d734f13c593c6ade5a5874/merged,device=overlay,fstype=overlay,mode=rw,host=663d27b3f73f total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i 1517303196000000000
> disk,path=/hostfs/var/lib/docker/overlay2/c6ac6a3cd3eafa07840b43cad7168660687cc57c43d734f13c593c6ade5a5874/merged,device=overlay,fstype=overlay,mode=rw,host=663d27b3f73f free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i 1517303196000000000
> disk,path=/hostfs/etc,device=xvda1,fstype=ext4,mode=ro,host=663d27b3f73f inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426 1517303196000000000
> disk,mode=rw,host=663d27b3f73f,path=/etc/resolv.conf,device=xvda1,fstype=ext4 inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426 1517303196000000000
> disk,device=xvda1,fstype=ext4,mode=rw,host=663d27b3f73f,path=/etc/hostname used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i,free=103940083712i 1517303196000000000
> disk,path=/etc/hosts,device=xvda1,fstype=ext4,mode=rw,host=663d27b3f73f total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426,inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i 1517303196000000000
> disk,device=xvda1,fstype=ext4,mode=ro,host=663d27b3f73f,path=/var/run/utmp inodes_total=6553600i,inodes_free=6500469i,inodes_used=53131i,total=105553080320i,free=103940083712i,used=1510342656i,used_percent=1.4322774293289426 1517303196000000000

With HOST_MOUNT_PREFIX:

]# docker run --rm --name telegraf     -v /:/hostfs:ro -v /etc:/hostfs/etc:ro -v /proc:/hostfs/proc:ro -v /sys:/hostfs/sys:ro -v /var/run/utmp:/var/run/utmp:ro -e HOST_ETC=/hostfs/etc -e HOST_PROC=/hostfs/proc -e HOST_SYS=/hostfs/sys -e HOST_MOUNT_PREFIX=/hostfs telegraf:1.5.1-alpine --input-filter disk --test
2018/01/30 09:07:00 I! Using config file: /etc/telegraf/telegraf.conf
* Plugin: inputs.disk, Collection 1
> disk,path=/,device=overlay,fstype=overlay,mode=rw,host=370d8c6e5cf0 used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i,total=105553080320i,free=103940079616i,used=1510346752i 1517303220000000000
> disk,fstype=ext4,mode=ro,host=370d8c6e5cf0,device=xvda1 total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i 1517303220000000000
> disk,fstype=ext4,mode=rw,host=370d8c6e5cf0,path=/var/lib/docker/plugins,device=xvda1 total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i 1517303220000000000
> disk,path=/var/lib/docker/overlay2,device=xvda1,fstype=ext4,mode=rw,host=370d8c6e5cf0 total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i 1517303220000000000
> disk,path=/var/lib/docker/overlay2/283d6d3d95dab8f9f19bbb6380232a7eb015bb0d26da1f73d52ded9a42bac94a/merged,device=overlay,fstype=overlay,mode=rw,host=370d8c6e5cf0 inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i,total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952 1517303220000000000
> disk,device=overlay,fstype=overlay,mode=rw,host=370d8c6e5cf0,path=/var/lib/docker/overlay2/283d6d3d95dab8f9f19bbb6380232a7eb015bb0d26da1f73d52ded9a42bac94a/merged used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i,total=105553080320i,free=103940079616i,used=1510346752i 1517303220000000000
> disk,fstype=ext4,mode=ro,host=370d8c6e5cf0,path=/etc,device=xvda1 free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i,total=105553080320i 1517303220000000000
> disk,path=/etc/resolv.conf,device=xvda1,fstype=ext4,mode=rw,host=370d8c6e5cf0 total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i 1517303220000000000
> disk,host=370d8c6e5cf0,path=/etc/hostname,device=xvda1,fstype=ext4,mode=rw total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i 1517303220000000000
> disk,path=/etc/hosts,device=xvda1,fstype=ext4,mode=rw,host=370d8c6e5cf0 inodes_free=6500470i,inodes_used=53130i,total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952,inodes_total=6553600i 1517303220000000000
> disk,path=/var/run/utmp,device=xvda1,fstype=ext4,mode=ro,host=370d8c6e5cf0 inodes_total=6553600i,inodes_free=6500470i,inodes_used=53130i,total=105553080320i,free=103940079616i,used=1510346752i,used_percent=1.4322813136185952 1517303220000000000

I think left trimming paths starting with HOST_MOUNT_PREFIX causes the root path /hostfs to become nothing.