Pulse Host Agent over-reports usable storage on ZFS (TrueNAS) due to dataset AVAIL summation

[ak-tr]

Describe the bug

Pulse host agent reports incorrect usable storage size on ZFS systems (TrueNAS SCALE).
Instead of reporting the pool’s actual usable capacity, it sums the AVAIL value for every ZFS dataset, resulting in massively inflated storage numbers.

Example:
A TrueNAS system with a single ~5.05 TB usable ZFS pool is reported in Pulse as ~46 TB usable because there are multiple datasets, each showing the same 5.05 TB AVAIL value. The agent treats them as independent filesystems and adds them together.

This leads to misleading storage metrics, incorrect % usage, and incorrect alerting.

To Reproduce

1. Install the Pulse host agent on a TrueNAS SCALE host using the install script.
2. Go to Reporting → Hosts → Storage in Pulse.
3. Observe that the “usable” capacity is multiplied by the number of ZFS datasets instead of matching the ZFS pool’s actual capacity.

Expected behavior

Pulse should detect ZFS pools and report pool-level capacity, not sum the “available” values for each dataset.
All ZFS datasets share the same underlying pool storage, so their AVAIL values represent the same free space and should not be aggregated.

Expected behaviour:

• Report each ZFS pool once.
• Do not treat child datasets as separate storage volumes.
• Show accurate usable storage values that match zpool list or the TrueNAS UI.

Environment

• Pulse Version: Latest (installed via install-docker-agent.sh as of Feb 2025)
• Installation Type: Manual host agent on TrueNAS SCALE (not Docker)

Additional context

Below is the textual output of zfs list (dataset names anonymized).
Each dataset reports the same 5.05 TB AVAIL value, even though the pool has only 5.05 TB total usable capacity:

poolname                             5.05T
poolname/.system                     5.05T
poolname/.system/configs-*           5.05T
poolname/.system/cores               5.05T
poolname/.system/nfs                 5.05T
poolname/.system/samba4              5.05T
poolname/.system/vm                  5.05T
poolname/user1                       5.05T
poolname/user2                       5.05T

[LordShadowen]

@ak-tr Can I ask about how you installed the host agent on TrueNAS? Did you just run the script? Does that work?
My understanding was that Truenas doesn't allow changes to the system (or at least, doesn't like them and they would get overwritten on the next update....)
I'm also interested in trying to monitor TrueNAS with Pulse, but was curious about your experience before starting to break things :)

[ak-tr]

@ak-tr Can I ask about how you installed the host agent on TrueNAS? Did you just run the script? Does that work?
My understanding was that Truenas doesn't allow changes to the system (or at least, doesn't like them and they would get overwritten on the next update....)
I'm also interested in trying to monitor TrueNAS with Pulse, but was curious about your experience before starting to break things :)

Yes I installed directly using the script. But, you're right. I rebooted my TrueNAS instance and it is no longer reported in Pulse.

I think I could make a boot script that re enables the service but it would have to recreate the user for the agent and a bunch of other stuff.

Maybe a VM inside of TrueNAS to run the agent might be more robust. Not sure if the overhead makes it's worth the hassle now though.

[rcourtman]

TrueNAS SCALE uses an immutable root, so running the host-agent install script only works until the next reboot/update—systemd unit, user, and files are wiped. Pulse doesn’t currently support a persistent host-agent install on SCALE. The reliable option is to run the agent in a small VM/container on that node (or another host) and point it at the SCALE IP, instead of reinstalling on every boot. If native SCALE support is needed, it would require a dedicated install path that survives upgrades.

[LordShadowen]

Thanks both - that tracks with my understanding of how TrueNAS works,

But I don't understand what you mean by this part: "The reliable option is to run the agent in a small VM/container on that node (or another host) and point it at the SCALE IP".
Running a VM or a container in TrueNAS would not be problem, but they would be isolated from the host (by design), so how could it collect metrics? What do you mean by pointing it at Truenas IP?

Very very specifically, I'm quite interested in monitoring the HDD temperatures in TrueNAS. How would the agent running on a VM (or container) would have access to that info?

[ak-tr]

I would also be interested in the answer to this.

[rcourtman]

Quick update: the ZFS double-counting fix is already in v4.32.0+ (commit f4fd492). On installation, pulse-host-agent has to run on the system it is monitoring; it reads local gopsutil data and sensors -j output, so a VM or container instance would only report that guest and cannot see the SCALE host’s ZFS pools or hardware sensors. TrueNAS SCALE’s immutable root wipes the service on reboot, and there is no supported persistent host-agent install there today; the only current workaround would be reinstalling after each boot or building a SCALE-specific integration that stores the unit on a persistent dataset. Also note that the agent currently reports CPU/NVMe/GPU temperatures via lm-sensors and does not surface SATA HDD temperatures yet.

[ak-tr]

I installed using --platform truenas and when I run systemctl status pulse-host-agent, I get 203/EXEC error. Do you know why this might happen? TrueNAS is running in a VM on a Proxmox instance if that helps.

[ak-tr]

Ah, I see it's because exec is off for where the executable is stored.

Is it safe to move the pulse host agent files elsewhere. Will it work as expected on reboot?

[rcourtman]

203/EXEC means systemd couldn’t execute . That usually happens when the download isn’t an executable for this host or \ is on a dataset mounted . I’ve updated the installer to fail fast when the target path is noexec or the download isn’t executable. To narrow yours: please share \ and , plus /data/pulse-host-agent/pulse-host-agent: cannot open `/data/pulse-host-agent/pulse-host-agent' (No such file or directory). If \ (or ) shows , flip exec on that dataset or reinstall to an exec-capable path and the service should start.