Node filesystem alerting is not aligned with kubelet eviction thresholds #3474
Assignees
Labels
kind:bug
Something isn't working
topic:monitoring
Everything related to monitoring of services in a running cluster
Comments
gdemonet
added
kind:bug
alexandre-allard
added a commit
that referenced
this issue
Aug 4, 2021
We lower the thresholds for the following alerts NodeFilesystemAlmostOutOfSpace: - warning from 5% to 20% - critical from 3% to 12% NodeFilesystemAlmostOutOfFiles: - warning from 5% to 15% - critical from 3% to 8% Otherwise we don't receive alert before kubelet starts evicting pods when disk is full as its threshold is set to 10% of available disk space and 5% of inodes free. Refs: #3474
alexandre-allard
added a commit
that referenced
this issue
Aug 5, 2021
``` ./tools/rule_extractor/rule_extractor.py \ -i <control-plane-ip> -p 8443 -t rules ``` Refs: #3474
|
Fixed by #3479 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component: kubernetes, alerting
What happened:
When trying to raise
NodeFilesystemAlmostOutOfSpacealerts manually (by creating arbitrary large files in the root FS), we realized kubelet triggered the eviction process long before we reached the 5% threshold.What was expected:
To get an alert much before kubelet starts evicting pods.
Resolution proposal:
Lower the alert thresholds according to kubelet defaults (which we don't want to change for now):
NodeFilesystemAlmostOutOfSpace(kubelet usesnodefs.available<10%):5%20% space left3%12% space leftNodeFilesystemAlmostOutOfFiles(kubelet usesnodefs.inodesFree<5%):5%15% inodes left3%8% inodes leftThe text was updated successfully, but these errors were encountered: