Kernel panic in zfs_findernotify_callback → zfs_getattr_znode_unlocked under sustained vnode churn (race on freed znode); single-byte binary workaround documented

[adriant0ie]

Disclaimer: Bug analysis and patch development in collaboration with Claude (Anthropic). The actual binary surgery, monitoring, and verification under load was done by myself during a live storage migration; no warranty is implied.

Environment

•	macOS: 10.10.5 Yosemite (Mac mini Server, mid-2011, 16 GB RAM)
•	openzfsonosx version: 2.4.1rc4 (build string: zfs-macOS-2.4.1rc4-0-ga9fc067-dirty)
•	ZFS pool version: 5000
•	Boot args: -v keepsyms=1 kext-dev-mode=1
•	Storage: 2× Pegasus DAS over Thunderbolt 1, ZFS pools on SSDs (6-disk RAIDZ1) and HDDs (5-disk RAIDZ1)
•	Workload at time of panic: bulk migration via tar -cpf - . | tar -xpf - pipelines, sustained ~80–400 MB/s writes, creating thousands of vnodes/sec on the destination dataset

Symptom

Reproducible kernel panic under sustained vnode-creation workload (file extraction of many small files via tar). Latency to panic varied from a few seconds to a few minutes, but inevitable on any large-file-count phase.

Symbolicated panic stack from /Library/Logs/DiagnosticReports/Kernel_2026-05-20-122729_mini.panic (attached in full):

panic(cpu 4 caller 0xffffff8019cb3df8): Kernel trap at 0x000007f9b....
CR2: 0x000007f9b....  (userspace address → stale/freed pointer dereference)

Backtrace:
  zfs_getattr_znode_unlocked + 0x4b7
  zfs_vnop_getattr
  vnode_getattr
  zfs_findernotify_callback
  vfs_iterate
  zfs_findernotify_thread
  call_continuation

The CR2 register containing a userspace-looking address while inside kernel-mode code is the diagnostic — it's the value of a pointer being dereferenced that should have been a kernel-space znode struct but had been freed and the memory reused or never reinitialized.

Root cause analysis

zfs_findernotify_thread periodically calls vfs_iterate(rootmount, zfs_findernotify_callback, ...) to walk vnodes and check Finder info. The callback eventually dereferences the znode via vnode_getattr → zfs_vnop_getattr → zfs_getattr_znode_unlocked.
Under sustained vnode churn (thousands of creates/sec during bulk import), there appears to be a race window where the vnode pointer handed to the callback by vfs_iterate can be freed or recycled before the callback's read path completes. The crash signature (zfs_getattr_znode_unlocked+0x4b7, fault on what looks like userspace memory) is consistent with a use-after-free of the znode struct: the dereference happens, the memory has been returned to the slab and possibly reused for unrelated allocations, and the resulting read produces garbage that the code can't survive.
The function name (zfs_getattr_znode_unlocked) is also suggestive — the "unlocked" suffix implies the caller is responsible for ensuring lifetime, and vfs_iterate may not be holding the right reference in this code path.

Reproducer

On openzfsonosx 2.4.1rc4 install on Yosemite, with a ZFS pool present:

# Create a dataset with lz4 compression
sudo zfs create -o compression=lz4 -o recordsize=128K pool/repro

# Generate sustained small-file write churn
sudo bash -c '
  cd /pool/repro
  for i in $(seq 1 50000); do
    echo "$i $(date +%s%N)" > "f_$i.txt"
  done
'

In our environment this reliably panicked within seconds. Throughput at time of panic was typically 50–80 MB/s on SSD; faster pools may panic sooner.

A more diagnostic monitor while reproducing:

# Watch memory_throttle_count rate — climbs sharply just before panic
sysctl kstat.zfs.misc.arcstats.memory_throttle_count

Workaround applied (production, verified stable)

In-place binary patch of /Library/Extensions/zfs.kext/Contents/MacOS/zfs, replacing the first byte of _zfs_findernotify_callback (0x55, push %rbp) with 0xC3 (ret). This causes the callback to return immediately on every invocation. The thread continues to run normally and vfs_iterate still walks the vnode list, but the racy dereference is never reached.

# Locate offset
nm /Library/Extensions/zfs.kext/Contents/MacOS/zfs | grep findernotify_callback
# 000000000018f1f0 t _zfs_findernotify_callback

# Apply patch (single byte at offset 0x18f1f0 = 1634800 decimal)
printf '\xc3' | sudo dd of=/Library/Extensions/zfs.kext/Contents/MacOS/zfs \
                          bs=1 seek=1634800 count=1 conv=notrunc

# Force prelinked-kernel cache rebuild (essential, otherwise stale cache loads)
sudo touch /Library/Extensions/zfs.kext
sudo kextcache -invalidate /
sudo rm -f /System/Library/PrelinkedKernels/prelinkedkernel
sudo kextcache -system-prelinked-kernel
sudo kextcache -u /

After the patch, in our environment: 12+ TiB transferred across a 14-hour migration window across both SSD and HDD pools with zero panics. Pre-patch: panics within minutes of any bulk-write workload.

Important detour we hit

Initial attempt also patched _zfs_findernotify_thread itself (0x55 → 0xC3 at offset 0x1871a0). This broke kext load:

SPL: spl_set_thread_importance error: ret 4

ret 4 = KERN_INVALID_ARGUMENT. Sequence:
1 SPL creates the findernotify thread
2 Patched thread function returns immediately, thread terminates
3 SPL calls spl_set_thread_importance() / thread_policy_set() on the now-dead thread's port
4 Mach call returns KERN_INVALID_ARGUMENT
5 SPL doesn't handle this gracefully → panic

The lesson: the thread must continue to exist; only its work needs to be neutralized. Patching the callback (which the thread calls in a loop via vfs_iterate) achieves this — the thread runs, but its loop body is a no-op.

This may also be useful information for proper-fix authors: any future patch that wants to disable findernotify cleanly should probably also gate the thread's creation in SPL setup, not patch the thread function itself.

Limitations of this workaround

•	Findernotify is disabled. Finder integration is degraded on desktop installs (stale file views, lagging Get Info, etc.). No impact on server / headless / migration use.
•	Per-binary patch. Offsets are specific to this build; any kext update requires re-patching against the new binary.
•	Requires kext-dev-mode=1. The patch invalidates the kext signature; signed-only systems would reject the modified kext.
•	Subject to cache invalidation issues. The prelinked kernel cache can serve a stale unpatched copy even after the on-disk binary is patched; the full cache invalidation sequence above is necessary.
•	Not a fix. The underlying race condition remains in the source.

Hashes of the unpatched and patched binaries from our system, for anyone wanting to verify their own binary against ours:

Original: d2f9e80bf1b3c90d62da187a970954254eab4ed8e50c9a7988f8a602ebb9bf4f
Patched: dda1f72ef843250ff8b9fd42e5d8bb745118840ece9204c9bb26a1e95eda5194

These are for the 2.4.1rc4-0-ga9fc067-dirty build distributed for Yosemite. Other builds will differ.

Kernel_2026-05-20-122729_mini.panic.txt

[lundman]

OK Claude said:

Bug 1 — zfs_findernotify_callback (the main reported race):

• Moved vfs_fsprivate(mp) to after the vfs_name check — no functional change, but makes the guard order clearer
• Removed (mp != zfsvfs->z_vfs): this dereference happened before zfs_enter held the teardown lock, creating a window where zfsvfs_free() could race and free the struct before
  the field was read. The vfs_name check already ensures we're on a ZFS mount; zfs_enter is the correct guard for concurrent unmount
• The zfs_enter call is now the sole protection, which is correct: if the FS is torn down, it returns EIO under the teardown lock

Bug 2 — zfs_findernotify_refresh:

• Added if (zfsvfs == NULL) return; — if vfs_setfsprivate(NULL) was already called during unmount, vfs_fsprivate returns NULL and the previous code would unconditionally
  dereference it (NULL deref, or UAF in the tighter race window)

:)