Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restrict Fstype for Disk Fill Alert #77

Closed
xixipangma opened this issue Jan 24, 2020 · 3 comments
Closed

Restrict Fstype for Disk Fill Alert #77

xixipangma opened this issue Jan 24, 2020 · 3 comments

Comments

@xixipangma
Copy link

xixipangma commented Jan 24, 2020

After adding Alert 2.7, I noticed that i'm getting disk fill alerts for /run/user/1113 mountpoint, since the rule will apply to all fstype labels of tmpfs as well. Perhaps we should set rule to restrict it to ext4 and xfs etc or may be add a not equal to condition to rule out tmpfs ?

predict_linear(node_filesystem_free_bytes{fstype=~"ext4|xfs"}[1h], 4 * 3600) < 0

@samber
Copy link
Owner

samber commented Jan 24, 2020

Should be better if we exclude this specific fstype:

predict_linear(node_filesystem_free_bytes{fstype!~"tmpfs"}[1h], 4 * 3600) < 0

@paulfantom
Copy link

That's unfortunate. tmpfs is frequently used for directories which should be pruned on node reboot, but filling them up during operations can cause problems with system. A good example is a website which allows uploading files. By default webserver will use a temporary directory for storing uploaded chunks, which then will be copied as a file to a target directory. So if you fill up temporary dir, you should have alert as this won't allow for future uploads and impact site. Another example would be to have /var/log as tmpfs when you have a centralized log solution.
Those are the reasons why in official node_exporter mixins fstype is a configurable option and why hardcoded exclusion of tmpfs shouldn't be implemented.

@paulfantom
Copy link

paulfantom commented Jan 25, 2020

@xixipangma why not exclude that mountpoint from node_exporter with collector.filesystem.ignored-fs-types or collector.filesystem.ignored-mount-points?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants