Harden disk collector: exclude network filesystems, bound statfs, df-compatible used_percent#40
Merged
Merged
Conversation
Fixes two robustness findings from the 2026-07-13 project review: Issue #26 — a hung network filesystem froze the entire metric loop. statfs(2) on an unreachable NFS/CIFS server blocks (indefinitely for hard mounts), and fastTick runs all collectors sequentially, so one dead NAS mount silently froze every metric. Network filesystem types (nfs, nfs4, cifs, smb3, fuse.sshfs, 9p, ...) are now excluded by default — remote capacity belongs to the remote system's monitoring — and as defense in depth each statfs call runs behind a 2s timeout. A timed-out mountpoint is skipped for a 1-minute cool-down so abandoned goroutines (statfs cannot be canceled) do not pile up one per tick. Issue #27 — used_percent was computed as used/total from Bfree, which counts root-reserved blocks (5% on ext4) as free: the dashboard showed ~95% while df already reported 100% and services failed to write, under the default disk_crit_percent threshold. The percentage now uses df's formula, used / (used + Bavail). total_bytes and used_bytes keep their raw meaning; only the percentage basis changes, so no API version bump. Duplicate mountpoints in /proc/mounts (overmounts, bind-mount setups) are also deduplicated, keeping the last entry — the filesystem actually visible at the path — so history ring buffers no longer receive two points per tick for the same mountpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the two disk-collector robustness findings from the 2026-07-13 project review:
#26 — a hung network filesystem blocked the entire metric loop.
statfs(2)on an unreachable NFS/CIFS server blocks (indefinitely for hard mounts), andfastTickruns all collectors sequentially, so a single dead NAS mount froze every metric while the dashboard silently served a stale snapshot. Two complementary changes:nfs,nfs4,cifs,smbfs,smb3,9p,fuse.sshfs,davfs,afs,glusterfs,ceph,curlftpfs) are now indefaultExcludedFSTypes— local storage is what the dashboard is about; remote capacity belongs to the remote system's monitoring.statfscall now runs in a goroutine behind a 2 s timeout (statfsWithTimeout). Sincestatfscannot be canceled, a timed-out goroutine lingers until the syscall returns; to bound the pile-up, a timed-out mountpoint enters a 1-minute cool-down (badUntil) and is skipped instead of being retried every tick.#27 — misleading disk-usage math and duplicate mountpoints.
used_percentnow usesdf's formulaused / (used + Bavail)instead ofused / totalfromBfree.Bfreecounts root-reserved blocks (typically 5 % on ext4) as free, so a full disk showed ~95 % — below the defaultdisk_crit_percent: 95— whiledfreported 100 % and services were already failing to write.total_bytes/used_byteskeep their raw meaning; only the percentage basis changes (value semantics, no JSON shape change, so no API version bump). Documented indocs/API.md./proc/mounts(overmounts, some bind-mount setups) are deduplicated keeping the last entry — the filesystem actually visible at that path — preserving first-seen order. This also stopsfastTickfrom writing twoHistoryPoints with the same timestamp into one ring-buffer key per tick.Note one intentional behavior change encoded in the updated
TestDiskCollector_Collect: the existing fixture ends with anoverlayovermount of/, so after keep-last dedup/resolves to the overlay entry and is excluded (previously it was reported asext4with values thatstatfs("/")actually took from the overlay).Related Issue
Closes #26
Closes #27
Checklist
go test ./...passes locally) — includinggo test ./... -race; new tests cover network-FS exclusion, the df percentage formula (Bfree > Bavail), zero-denominator guard, mountpoint dedup (later entry wins, one stat per unique mountpoint), and a hungstatfs(returns within the timeout, healthy mounts still reported, exactly one attempt per cool-down window)go vet ./...andgolangci-lint runare clean (0 issues)docs/API.md) — added notes on df-compatibleused_percentsemantics, dedup, and network-FS exclusion underdisks/api/v1/...response shapes, or a new API version was introduced instead — JSON shape is unchanged; only the percentage's value basis changes to matchdfNot verifiable without hardware: behavior against a real hard-mounted NFS share on a Pi (the hang was reproduced with an injected blocking
statfsFunc, per the acceptance criteria in #26).