cAdvisor logs errors when a container it monitors is removed

[AaronKalair]

Hi,

cAdvisor version - v0.25.0-17543be
On AWS AMI - amzn-ami-2017.03.f-amazon-ecs-optimized (ami-9eb4b1e5)
Using the devicemapper storage driver.

We're seeing our cAdvisor logs full of:

E1016 09:16:47.508913       1 fs.go:333] Stat fs failed. Error: no such file or directory
E1016 09:16:48.487228       1 fs.go:333] Stat fs failed. Error: no such file or directory

Which appears to be coming from here - https://github.com/google/cadvisor/blob/v0.25.0/fs/fs.go#L333

This starts happening after containers have been removed.

To replicate:

1. Start any random docker container
2. Start cAdvisor
3. Tail the cAdvisor containers logs
4. Note there are no error messages
5. Docker stop the container
6. Docker rm the container
7. You'll now see the above error printed every second in the cAdvisor logs

This will continue until the cAdvisor container is restarted

Is this the intended behaviour?

Thanks!

[janderson2]

Still seeing this behaviour with version 0.28.3 (1e567c2) while running the binary natively on an EC2 instance running ubuntu 16.04 and using aufs storage driver .

[dashpole]

The reason for this is that cAdvisor attempts to collect metrics on all filesystems on the host when it starts up. We should lower the verbosity of the log message or ignore devicemapper filesystems.

[tiagoad]

I'm still getting my logs filled with this message every second. Any fix possible while the image is not updated?

[rhuddleston]

#1847 any plans on a new release soon?

[amihura]

+++

[nightah]

@dashpole while it doesn't look like #1847 is necessarily a fix for this, is it likely there might be a change to lower the log verbosity of said error in the instance of devicemapper/zfs filesystems?

At this point, even if we run cadvisor with --v=0 option, the error still appears. You can imagine if containers are being removed frequently, without cadvisor being restarted it can result in hundreds of lines of error logs per second.

[joshperry]

FYI, having this problem with the btrfs driver as well.

[eklitzke]

In addition to lowering the log verbosity, I suggest adding an option like prometheus-node-exporter's --collector.filesystem.ignored-fs-types which would tell cadvisor to ignore the listed filesystem types. This could be used to explicitly ignore aufs etc.