Skip to content

feat(diskspace.plugin): smart ZFS dataset deduplication#21643

Merged
ilyam8 merged 15 commits intonetdata:masterfrom
ilyam8:feat/diskspace-zfs-dataset-deduplication
Jan 27, 2026
Merged

feat(diskspace.plugin): smart ZFS dataset deduplication#21643
ilyam8 merged 15 commits intonetdata:masterfrom
ilyam8:feat/diskspace-zfs-dataset-deduplication

Conversation

@ilyam8
Copy link
Copy Markdown
Member

@ilyam8 ilyam8 commented Jan 26, 2026

Summary

Fixes: #21632

  • Replace unconditional ZFS dataset exclusion with capacity-based heuristic
  • Keep datasets with quotas (they have different capacity than pool)
  • Keep all datasets when pool itself is not mounted
  • Cache exclusion decisions to minimize statvfs() syscalls
  • Add configuration options for manual control

Problem

Previously, diskspace.plugin excluded all ZFS datasets unconditionally, monitoring only pools. This caused issues:

  1. Datasets with quotas have genuinely different capacity and should be monitored separately
  2. When only datasets are mounted (no pool), nothing was monitored

Solution

Scenario Old Behavior New Behavior
Pool tank (1TB) Keep Keep
Dataset tank/home (no quota, shows 1TB) Exclude Exclude
Dataset tank/vm (quota 200GB) Exclude Keep
Only datasets mounted (no pool) Exclude all Keep all
Inside LXC container Exclude Keep all

Performance:

To avoid calling statvfs() on every dataset every second:

  • Pool capacities are cached and refreshed every 5 minutes
  • Dataset exclusion decisions are cached for 5 minutes
  • Excluded datasets skip statvfs() entirely using cached decision

For systems with hundreds of ZFS datasets, this reduces syscalls from hundreds per second to just a few.

Visual Flow

                         ┌─────────────────────┐
                         │   ZFS Mount Point   │
                         └──────────┬──────────┘
                                    │
                        ┌───────────▼───────────┐
                        │  Cached as excluded?  │──Yes─→ SKIP (no statvfs)
                        └───────────┬───────────┘
                                    │No
                                    │ [statvfs() called]
                        ┌───────────▼───────────┐
                        │   Is ZFS filesystem?  │──No──→ KEEP
                        └───────────┬───────────┘
                                    │Yes
                        ┌───────────▼───────────┐
                        │   Is root mount (/)?  │──Yes─→ KEEP
                        └───────────┬───────────┘
                                    │No
                        ┌───────────▼───────────┐
                        │ Inside LXC container? │──Yes─→ KEEP
                        └───────────┬───────────┘
                                    │No
                        ┌───────────▼───────────┐
                        │  Is it a pool (no /)? │──Yes─→ KEEP
                        └───────────┬───────────┘
                                    │No (it's a dataset)
                        ┌───────────▼───────────┐
                        │  Heuristic enabled?   │──No──→ Pattern Match
                        └───────────┬───────────┘
                                    │Yes
                        ┌───────────▼───────────┐
                        │ Pool in cache & fresh?│──No──→ KEEP (pool not mounted)
                        └───────────┬───────────┘
                                    │Yes
                        ┌───────────▼───────────┐
                        │ Capacity < pool cap?  │──Yes─→ KEEP (has quota)
                        └───────────┬───────────┘
                                    │No
                                    ▼
                              EXCLUDE & CACHE
                         (mirrors pool capacity)

Configuration

New options in [plugin:proc:diskspace]:

# Smart heuristic (default) - exclude datasets mirroring pool capacity
zfs datasets heuristic = yes

# Manual pattern exclusion (when heuristic disabled)
zfs datasets heuristic = no
exclude zfs datasets on paths = tank/docker/* tank/cache/*
Test Plan

Tested on a system with ZFS pool rpool and datasets rpool/data, rpool/ROOT (no quotas)

Test Case Configuration Result
Default (heuristic enabled) zfs datasets heuristic = yes ✅ Datasets excluded, only pool monitored
Heuristic disabled, default pattern zfs datasets heuristic = no ✅ All datasets collected (default !* excludes nothing)
Heuristic disabled, custom pattern exclude zfs datasets on paths = rpool/data ✅ Only matched dataset excluded

Not directly tested (no quota datasets available):

  • Dataset with quota → should be kept based on capacity comparison logic (verified via debug logs showing capacity comparison works correctly)

Verified via debug logs:

  • Pool capacity correctly cached
  • Dataset capacity comparison works (capacity >= pool capacity → exclude)
  • Cache hit/miss behavior correct
  • 5-minute recheck interval works
Additional Information
For users: How does this change affect me?

Summary by cubic

Smart ZFS dataset deduplication in diskspace.plugin using a capacity-based heuristic that keeps datasets with quotas, excludes those mirroring their pool, and caches pool capacities and dataset exclusion decisions with a 5-minute recheck to reduce statvfs calls. Exclusion runs on both fast and slow paths, with defensive fixes to prevent a NULL mount_source crash and a dictionary race.

  • New Features

    • Excludes ZFS datasets that match pool capacity; keeps datasets with quotas.
    • Keeps all datasets when the pool is not mounted; never excludes root; keeps all mounts inside LXC.
    • Configuration in [plugin:proc:diskspace]:
      • zfs datasets heuristic (default: yes)
      • exclude zfs datasets on paths (used when heuristic is disabled)
  • Refactors

    • Exported is_lxcfs_proc_mounted() from proc.plugin for use by other collectors.

Written for commit f851055. Summary will update on new commits.

Replace unconditional ZFS dataset exclusion with a capacity-based
heuristic that:

- Excludes datasets that mirror pool capacity (no quota set)
- Keeps datasets with quotas (different capacity than pool)
- Keeps all datasets when pool itself is not mounted
- Keeps all mounts inside LXC containers
- Never excludes root filesystem

New configuration options in [plugin:proc:diskspace]:
- "zfs datasets heuristic" (default: yes) - enable smart heuristic
- "exclude zfs datasets on paths" - manual pattern exclusion when
  heuristic is disabled

Also exports is_lxcfs_proc_mounted() from proc.plugin for use by
other collectors that need LXC container detection.
Avoid calling statvfs() for ZFS mounts on every collection cycle.
Pool capacities and exclusion decisions are now cached and only
rechecked every 5 minutes (ZFS_DATASET_RECHECK_SECONDS).

This eliminates the performance overhead of the two-pass heuristic
while still adapting to quota changes within a reasonable timeframe.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements smart ZFS dataset deduplication in diskspace.plugin to avoid duplicate metrics from datasets that mirror their parent pool's capacity. The solution uses a two-pass capacity-based heuristic to identify and exclude datasets without quotas, while keeping datasets with quotas and handling edge cases like LXC containers and unmounted pools.

Changes:

  • Exported is_lxcfs_proc_mounted() function from proc.plugin for shared use across collectors
  • Implemented two-pass ZFS dataset filtering: Pass 1 collects pool capacities, Pass 2 excludes datasets matching pool capacity
  • Added configuration options for manual control via zfs datasets heuristic and exclude zfs datasets on paths

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File Description
src/collectors/proc.plugin/plugin_proc.h Added function declaration for is_lxcfs_proc_mounted() to make it available to other collectors
src/collectors/proc.plugin/plugin_proc.c Changed is_lxcfs_proc_mounted() from static to exported function
src/collectors/diskspace.plugin/plugin_diskspace.c Core implementation: added ZFS pool tracking, capacity-based heuristic, LXC detection caching, and configuration options
Comments suppressed due to low confidence (1)

src/collectors/proc.plugin/plugin_proc.c:125

  • The condition if (unlikely(!ff)) on line 119 will always be true since ff is initialized to NULL on line 117. This means the function opens /proc/self/mounts on every call but never closes it properly in the success path after the first open, leading to a file descriptor leak.

The logic should be restructured to avoid the unnecessary check. The variable initialization and file opening should be unconditional, or the check should be removed entirely since ff is always NULL at that point.

    if (unlikely(!ff)) {
        char filename[FILENAME_MAX + 1];
        snprintfz(filename, FILENAME_MAX, "/proc/self/mounts");
        ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT);
        if (unlikely(!ff))
            return false;
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files

Confidence score: 4/5

  • This PR appears safe to merge overall; the only reported issue is medium severity and limited to ZFS dataset handling.
  • Most notable risk: in src/collectors/diskspace.plugin/plugin_diskspace.c, slow-path collection bypasses the new ZFS dataset exclusion, so marked-slow datasets can still be collected and duplicates may persist.
  • Pay close attention to src/collectors/diskspace.plugin/plugin_diskspace.c - ensure the ZFS exclusion heuristic applies consistently in both fast and slow paths.
Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/collectors/diskspace.plugin/plugin_diskspace.c">

<violation number="1" location="src/collectors/diskspace.plugin/plugin_diskspace.c:637">
P2: The new ZFS dataset exclusion is only applied in the fast path; slow-path collection bypasses it. ZFS datasets marked as slow will still be collected, so duplicates can persist despite the heuristic. Apply the same exclusion logic in the slow path (which may require carrying mount_source into basic_mountinfo) to keep behavior consistent.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Main as Diskspace Plugin
    participant ZFS as NEW: ZFS Heuristic
    participant State as Pool Info Dict
    participant Kernel as OS / Kernel

    Note over Main,Kernel: Initialization
    Main->>Main: Load Config (zfs heuristic = yes)
    Main->>State: Initialize Dictionary

    Note over Main,Kernel: Collection Cycle (Start)

    rect rgb(23, 37, 84)
        Note over Main,Kernel: Pass 1: Build Pool Capacity Cache
        Main->>ZFS: zfs_collect_pool_capacities()
        ZFS->>Kernel: is_lxcfs_proc_mounted() (Cache result)
        
        loop Every Mountpoint
            ZFS->>ZFS: Is ZFS Pool? (No '/' in source)
            opt Is Pool
                ZFS->>Kernel: statvfs(mount_path)
                ZFS->>State: Store max_capacity & seen_flag
            end
        end
    end

    rect rgb(30, 41, 59)
        Note over Main,Kernel: Pass 2: Filter & Collect Stats
        
        loop Every Mountpoint
            Main->>Kernel: statvfs(mount_path)
            Main->>ZFS: should_exclude_zfs_mount(mount, stats)
            
            alt CHANGED: Exclusion Logic
                ZFS->>ZFS: Check: Root? LXC? Not ZFS?
                
                alt Is Dataset (Candidate for exclusion)
                    ZFS->>State: Get parent Pool info
                    
                    alt Pool not mounted
                        ZFS-->>Main: Keep (Safety fallback)
                    else Dataset Capacity < Pool Capacity
                        ZFS-->>Main: Keep (Has Quota)
                    else Capacity == Pool Capacity
                        Note right of ZFS: Duplicate metric detected
                        ZFS-->>Main: NEW: EXCLUDE
                    end
                else Is Pool / Root / Other
                    ZFS-->>Main: Keep
                end
            end

            opt If Kept (Not Excluded)
                Main->>Main: Collect metrics & update charts
            end
        end
    end
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

@ilyam8 ilyam8 requested a review from Copilot January 26, 2026 15:46
@ilyam8
Copy link
Copy Markdown
Member Author

ilyam8 commented Jan 26, 2026

@cubic-dev-ai review this PR

@cubic-dev-ai
Copy link
Copy Markdown
Contributor

cubic-dev-ai bot commented Jan 26, 2026

@cubic-dev-ai review this PR

@ilyam8 I have started the AI code review. It will take a few minutes to complete.

Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant Plugin as Diskspace Plugin
    participant Proc as Proc Helper
    participant Cache as ZFS Pool Cache
    participant VFS as Kernel VFS
    participant Core as Netdata Core

    Note over Plugin,Cache: NEW: Phase 1 - Heuristic Prep (Every 5 mins)

    Plugin->>Proc: CHANGED: is_lxcfs_proc_mounted()
    Proc-->>Plugin: inside_lxc (bool)

    opt Cache Expired (300s)
        Plugin->>Cache: Flush Dictionary
        loop Filter for ZFS Pools
            Plugin->>VFS: Parse mount_source
            alt Is ZFS Pool (no '/')
                Plugin->>VFS: statvfs(pool_path)
                VFS-->>Plugin: pool_capacity
                Plugin->>Cache: Store max_capacity & mark_seen
            end
        end
    end

    Note over Plugin,Core: Phase 2 - Collection Cycle

    loop Every Mount Point
        Plugin->>VFS: statvfs(mount_path)
        VFS-->>Plugin: mount_stats

        alt NEW: ZFS Filtering Logic
            Plugin->>Cache: Get pool info (max_capacity, seen)
            
            alt Inside LXC OR Root OR Pool OR Heuristic=No
                Plugin->>Plugin: Decision: KEEP
            else Pool itself not mounted
                Plugin->>Plugin: Decision: KEEP (Safety fallback)
            else Dataset Size < Pool Max Size
                Plugin->>Plugin: Decision: KEEP (Has Quota)
            else Dataset Size ≈ Pool Max Size
                Plugin->>Plugin: Decision: EXCLUDE (Duplicate)
            end
        end

        opt Decision == KEEP
            Plugin->>Core: Update Charts & Metrics
        end
    end
Loading

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/collectors/proc.plugin/plugin_proc.c:125

  • The is_lxcfs_proc_mounted function has a resource leak. The procfile *ff is initialized as NULL and checked with if (unlikely(!ff)) which will always be true on first call. However, the variable is local (not static), so the file handle is never reused and is leaked on subsequent calls. The variable ff should be declared as static procfile *ff = NULL; to properly cache the file handle across calls, or the function should be redesigned to properly manage the resource lifecycle.
bool is_lxcfs_proc_mounted(void) {
    procfile *ff = NULL;

    if (unlikely(!ff)) {
        char filename[FILENAME_MAX + 1];
        snprintfz(filename, FILENAME_MAX, "/proc/self/mounts");
        ff = procfile_open(filename, " \t", PROCFILE_FLAG_DEFAULT);
        if (unlikely(!ff))
            return false;
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Fix LXC detection to run unconditionally (once at startup)
- Use "!*" as default pattern (explicitly exclude nothing)
- Create dictionary/pattern only when needed for the active mode
- Clarify tolerance comparison comments
- Add mount_source to basic_mountinfo struct
- Apply ZFS exclusion check to slow path (prevents duplicates)
@ilyam8 ilyam8 force-pushed the feat/diskspace-zfs-dataset-deduplication branch from f7f415b to 4d9bda4 Compare January 26, 2026 15:55
Remove pointless conditional check (local variable was always NULL)
and simplify control flow with single exit point for procfile_close().
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ensive checks

- Handle NULL mount_source in basic_mountinfo_create_and_copy() to prevent
  strdupz(NULL) crash when mountinfo parsing encounters errors
- Add defensive NULL check after dictionary_set() in zfs_collect_pool_capacities()
…ookup

Use dictionary_get_and_acquire_item() to hold a reference while reading
pool info values. This prevents a race where the main thread could flush
the dictionary (freeing values) while the slow worker thread is reading
from a pointer returned by dictionary_get().
@ilyam8 ilyam8 requested a review from Copilot January 26, 2026 16:15
@ilyam8 ilyam8 marked this pull request as ready for review January 26, 2026 16:15
@ilyam8 ilyam8 requested a review from thiagoftsm as a code owner January 26, 2026 16:15
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ison

The previous logic tracked max_capacity across all ZFS mounts, but in ZFS
statvfs returns total = used + available. This means datasets with data
show larger "total" than empty datasets or the pool itself.

For example:
- rpool (pool): 1.7T total (empty)
- rpool/ROOT/pve-1: 1.9T total (222G used + 1.7T available)

Using max (1.9T) made empty datasets (1.7T) look like they had quotas.

Fix: Track pool_capacity from the pool mount only, not datasets. Datasets
with capacity >= pool capacity have no quota and should be excluded.
Datasets with capacity < pool capacity have quotas and should be kept.
Move zfs_exclude_datasets_pattern initialization alongside other exclusion
patterns in do_disk_space_stats(). Create it unconditionally so the config
option always appears in netdata.conf, allowing users to customize it even
when starting with heuristic enabled and later switching to pattern mode.

Renamed option to "exclude space metrics on zfs datasets" for consistency
with other exclusion options.
Bug: ZFS dataset exclusion worked for the first 5 minutes, then stopped.
Datasets that should be excluded suddenly started being collected.

Root cause: dictionary_flush() cleared all pool info entries every 5 minutes.
If subsequent statvfs() calls failed (e.g., mount temporarily busy), the
dictionary remained empty and should_exclude_zfs() couldn't find pool info,
so it returned false (keep all datasets).

Fix: Remove dictionary_flush() and update entries in place. The dictionary_set()
overwrites existing entries when successful, but if statvfs fails, the previous
valid pool info is preserved. This ensures stable exclusion behavior.

Additional simplification: Moved statvfs() call before dictionary operations -
only add/update pool info when we successfully get capacity data.
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/collectors/diskspace.plugin/plugin_diskspace.c">

<violation number="1" location="src/collectors/diskspace.plugin/plugin_diskspace.c:388">
P2: Pool mount presence is never reset, so after a pool unmounts its datasets can still be excluded based on stale `pool_mount_seen`/capacity data. Clear or reset pool entries before rebuilding to ensure unmounted pools are treated as not mounted.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Combine pool capacity and dataset exclusion into single cache dictionary.
Previously we only cached pool capacities and checked every dataset on
every collection cycle. Now we also cache the exclusion decision per
dataset, avoiding statvfs() syscalls for excluded datasets.

Changes:
- Replace zfs_pool_info_dict with unified zfs_cache
- Add zfs_dataset_cached_excluded() check before statvfs()
- Per-entry timestamps instead of global zfs_last_checked
- Add debug logs (to be removed after testing)

Performance: For systems with hundreds of ZFS datasets, this reduces
statvfs() calls from hundreds per second to just a few (only pools
and datasets needing recheck).
@ilyam8 ilyam8 marked this pull request as draft January 26, 2026 17:33
Copy link
Copy Markdown
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/collectors/diskspace.plugin/plugin_diskspace.c">

<violation number="1" location="src/collectors/diskspace.plugin/plugin_diskspace.c:509">
P2: Pool capacity cache entries are used without checking staleness. If a pool was previously mounted and then unmounted, its cached entry persists and datasets keep being excluded forever, even though the design calls for keeping datasets when the pool is not mounted. Add a `last_checked` freshness check before using `pool_entry->pool_capacity`.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

If a ZFS pool was previously mounted and then unmounted, its cache entry
persisted indefinitely. Datasets would continue to be excluded based on
stale pool capacity, even though the design requires keeping all datasets
when the pool is not mounted (so at least one mount reports pool capacity).

Fix: Add freshness check before using pool_entry->pool_capacity. If the
pool cache entry is older than ZFS_DATASET_RECHECK_SECONDS, treat the pool
as unmounted and keep the dataset.
Using exactly ZFS_DATASET_RECHECK_SECONDS could cause race conditions
where an entry is considered stale right as it's being refreshed.

Using 2x interval means the entry must have missed at least one full
refresh cycle before being considered stale, which reliably indicates
the pool is no longer mounted.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ilyam8 ilyam8 merged commit 0f7ccc4 into netdata:master Jan 27, 2026
153 checks passed
@ilyam8 ilyam8 deleted the feat/diskspace-zfs-dataset-deduplication branch January 27, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: 2.8.5 broke zfs mount disk_space./ no more available

4 participants