[Bug]: 2.8.5 broke zfs mount disk_space./ no more available

[BenasPaulikas]

Bug description

[Bug]: 2.8.5 broke zfs mount disk_space./ no more available

Expected behavior

disk_space./ to work

Steps to reproduce

zfs lxc + netdata. netdata wont report disk_space./ anymore in 2.8.5 in 2.8.4 works good

Installation method

kickstart.sh

System info

.

Netdata build info

Packaging:
    Netdata Version ____________________________________________ : v2.8.5
    Installation Type __________________________________________ : binpkg-deb
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ : ubuntu 24.04
    Configure Options __________________________________________ : cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_STANDARD=11 -DCMAKE_CXX_STANDARD=14 -DBUILD_SHARED_LIBS=OFF -DCMAKE_C_FLAGS=' -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_CXX_FLAGS='  -fexceptions -fno-omit-frame-pointer -funwind-tables -fasynchronous-unwind-tables' -DCMAKE_COMPILE_DEFINITIONS='_GNU_SOURCE' -DCMAKE_EXE_LINKER_FLAGS=' -fexceptions -rdynamic' -DCMAKE_SHARED_LINKER_FLAGS=''
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /usr/share/netdata/web
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 6.17.4-1-pve
    Operating System ___________________________________________ : unknown
    Operating System ID ________________________________________ : unknown
    Operating System ID Like ___________________________________ : unknown
    Operating System Version ___________________________________ : unknown
    Operating System Version ID ________________________________ : 24.04
    Detection __________________________________________________ : unknown
Hardware:
    CPU Cores __________________________________________________ : 16
    CPU Frequency ______________________________________________ : 5582000000
    RAM Bytes __________________________________________________ : 31457280000
    Disk Capacity ______________________________________________ : 107374313472
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : none
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : lxc
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : Ubuntu
    Container Operating System ID ______________________________ : ubuntu
    Container Operating System ID Like _________________________ : debian
    Container Operating System Version _________________________ : 24.04.3 LTS (Noble Numbat)
    Container Operating System Version ID ______________________ : 24.04
    Container Operating System Detection _______________________ : /etc/os-release
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
    Memory Allocator ___________________________________________ : system
Database Engines:
    dbengine (compression) _____________________________________ : YES (zstd lz4)
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : NO
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
    libmnl (library for working with netfilter) ________________ : YES
    stacktraces (library for getting stack traces) _____________ : libbacktrace (mmap, threads, data)
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    windows (monitor Windows systems) __________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    network-viewer (monitor TCP/UDP IPv4/6 sockets) ____________ : YES
    systemd-journal (monitor journal logs) _____________________ : YES
    windows-events (monitor Windows events) ____________________ : NO
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : YES
    Xen VBD Error Tracking _____________________________________ : NO
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : NO
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO
Runtime Information:
    Profile ____________________________________________________ : standalone
    Stream Parent (accept data from Children) __________________ : NO
    Stream Child (send data to a Parent) _______________________ : NO
    Total System Memory ________________________________________ : 31457280000
    Available System Memory ____________________________________ : 14510402560

Additional info

No response

[ilyam8]

Hi. See issue (#21518) and PR (#21532). What is broken? We excluded datasets and kept pools.

[BenasPaulikas]

@ilyam8 thank you for such insanely fast reply.
my lxc container isn't aware about pools I believe. It's only aware of this:

df -h /
Filesystem                  Size  Used Avail Use% Mounted on
data/zd1/subvol-100-disk-0  200G   21G  180G  11% /

my lxc doesn't have any zfs controls

[julianneswinoga]

Hey seems I was affected by this too, I really liked that I could monitor my ZFS datasets with netdata. It gave really fine-graned info, monitoring the entire pool capacity isn't as helpful for me. I see #21518 was the impetus for the removal, but can we not simply change the alerting behavior rather than removing the underlying data?

[ilyam8]

@julianneswinoga, we were acting on information that

All ZFS datasets within a pool share the pool's total free space. This is a core design principle of ZFS—datasets are thinly provisioned and can grow dynamically as needed, drawing from the common pool of available storage.

Since this is the default, it makes sense to exclude these not only from alerting but also from metric collection. We don’t collect meaningless metrics, and this avoids consuming dgengine space quota.

[BenasPaulikas]

Imagine host zfs pool has 100GB of space.
lxc has 50 GB assigned.

Previously lxc netdata had disk_space./ which said 50gb total, 20gb used, 30gb available. Very convenient.

Now if I would query this data from host I would see space free 100-20=80GB. Host's netdata doesn't provide data/zd1/subvol-100-disk-0 information. Only pool information. (note that data/zd1 can have less space assigned than 100% of the pool)

Also let's keep in mind that LXC is usually isolated from host. So it would be strange asking for host's netdata of free space of LXC.

Unlike host, lxc doesn't even know how much free space zfs pool has. Basically lxc netdata doesn't have a way to know free space :)

Please correct me if I'm wrong somewhere.
Thank you :)

[julianneswinoga]

@ilyam8 that is partially correct. To add onto what @BenasPaulikas is saying; ZFS datasets can have different apparent sizes to their parent pool, called a quota or reservation: https://docs.oracle.com/cd/E23823_01/html/819-5461/gazvb.html

For a real-world example (output slightly trimmed for privacy):

# sudo zfs list -r
NAME                          USED  AVAIL  REFER  MOUNTPOINT
mediaPool                  2.24T  3.07T   132K  /mediaPool
mediaPool/backups           105G   919G   105G  /mediaPool/backups
mediaPool/homeassistant    13.1M  9.99G  13.1M  /mediaPool/homeassistant
mediaPool/misc              417G  83.2G   417G  /mediaPool/misc
mediaPool/movies           1.61T   404G  1.61T  /mediaPool/movies
rpool                       191G  34.1G   112K  /rpool
rpool/ROOT                 98.7G  34.1G    96K  /rpool/ROOT
rpool/data                 74.2G  34.1G   104K  /rpool/data
rpool/grafana              1.27G   743M  1.27G  /rpool/grafana
rpool/plex                 12.4G  7.63G  8.84G  /rpool/plex
rpool/tautulli              194M  1.81G   125M  /rpool/tautulli
tankMediaPool              9.37T  5.04T   149K  /tankMediaPool
tankMediaPool/TV           6.81T  1.19T  6.81T  /tankMediaPool/TV
# sudo zfs get -r quota tankMediaPool mediaPool | grep -vE 'autosnap|pve'
NAME                       PROPERTY  VALUE  SOURCE
mediaPool                  quota     none   default
mediaPool/backups          quota     1T     local
mediaPool/homeassistant    quota     10G    local
mediaPool/misc             quota     500G   local
mediaPool/movies           quota     2T     local
rpool                      quota     none   default
rpool/ROOT                 quota     none   default
rpool/data                 quota     none   default
rpool/grafana              quota     2G     local
rpool/plex                 quota     20G    local
rpool/tautulli             quota     2G     local
tankMediaPool              quota     none   default
tankMediaPool/TV           quota     8T     local
# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                           32G     0   32G   0% /dev
tmpfs                         6.3G  8.3M  6.3G   1% /run
tmpfs                          32G   29M   32G   1% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                          32G     0   32G   0% /tmp
tmpfs                         1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
rpool                          35G  128K   35G   1% /rpool
rpool/plex                     17G  8.9G  7.7G  54% /rpool/plex
rpool/tautulli                2.0G  125M  1.9G   7% /rpool/tautulli
rpool/ROOT                     35G  128K   35G   1% /rpool/ROOT
rpool/data                     35G  128K   35G   1% /rpool/data
mediaPool                     3.1T  256K  3.1T   1% /mediaPool
mediaPool/homeassistant        10G   14M   10G   1% /mediaPool/homeassistant
tankMediaPool                 5.1T  256K  5.1T   1% /tankMediaPool
mediaPool/misc                500G  417G   84G  84% /mediaPool/misc
tankMediaPool/TV              8.0T  6.9T  1.2T  86% /tankMediaPool/TV
mediaPool/backups             1.0T  105G  920G  11% /mediaPool/backups
mediaPool/movies              2.0T  1.7T  404G  81% /mediaPool/movies
/dev/fuse                     128M   20K  128M   1% /etc/pve
rpool/grafana                 2.0G  1.3G  744M  64% /rpool/grafana

I use quotas to ensure that I get a "soft warning" when a dataset fills up. Any "out of space" errors are limited to that dataset and do not affect any other services. Since the quota/reservation is virtual I can easily increase the size of the dataset, and check to see if the whole pool is close to the storage capacity. If I only get warnings on when a pool is nearing maximum capacity that is a much more involved process since unless you physically add more storage lots of services will break very fast and it is hard to resolve.

[ilyam8]

I understood that you can modify the default behavior with quotas and reservations. Since the default is

datasets within a pool share the pool's total free space.
can grow dynamically as needed

I plan to keep the current logic with the following fixes:

• If not inside LXC → exclude datasets by default, using the default pattern * (configurable via netdata.conf, can change to !* to include all datasets)
• If inside LXC → do not exclude datasets, regardless of pattern.

[ilyam8]

I tried to analyze the problem with AI, and an alternative is:

• Pass 1: Scan all ZFS mounts, remember the largest capacity seen for each pool name. This is the pool's capacity.
• Pass 2: For each ZFS mount:
  • If container mode → keep
  • If root filesystem → keep
  • If capacity < pool's max → keep (has quota, independent)
  • If capacity == pool's max → keep only ONE as the pool representative, exclude the rest as duplicates

[julianneswinoga]

As long as there is an option to override the behaviour I'm a happy camper 🙂