Mirror pool won't import — reproducible zio_err=0x34 on root objset during log_replay, all standard recovery flags fail (TrueNAS SCALE, ZFS 2.3.4)

[DamalZoha]

Hello everyone. I have some issues with my system and with the help of Claude AI, I tried to fix it. but failed. This post was made with AI, claude had all my reports and contexts.

Hoping someone recognizes this signature. I have a 2-disk mirror pool that went offline unexpectedly and now fails to import with every standard recovery method, despite strong evidence the pool and data are structurally intact.

System: ZimaBlade, TrueNAS SCALE 25.10.5 (reverted back to this version after briefly running a newer update — the pool broke shortly after that update, within about a day)

Pool: 2x 4TB WD Red Plus (WDC WD40EFPX) in a single mirror vdev, ashift=12, ZFS pool version 5000 / filesystem version 5

Symptom: Pool shows offline in the UI. zpool import (no args) correctly detects the pool as ONLINE with both mirror members ONLINE. But actually importing it — by name, with -f, -F, -FX, -o readonly=on, in every combination, with zil_replay_disable=1 and zfs_recover=1 set — fails every time with either:

cannot import 'Nirala_Main_Pool': one or more devices is currently unavailable

or, with readonly/-F variants:

cannot import 'Nirala_Main_Pool': insufficient replicas
Destroy and re-create the pool from a backup source.

What we've ruled out:

• Drive health: SMART overall-health PASSED on both disks, 0 reallocated sectors, 0 pending sectors, 0 offline uncorrectable, 0 UDMA CRC errors on both. Clean extended self-test history.
• Cabling/power: dmesg shows zero ata reset/link/error events during any import attempt.
• Label/uberblock corruption: zdb -l on both disks shows all 4 labels intact and identical (txg=4145589 at first check, later txg=4147295), matching guid_sum across both disks.
• Hostid mismatch: found and ruled out as sole cause (pool hostid didn't match system hostid, likely from an unclean shutdown, but forcing past this alone didn't resolve the import).
• Filesystem/read-write mode: confirmed block devices are fully read-write (/sys/block/sdX/ro = 0, blockdev --getro = 0).

The actual reproducible signature, from zpool events -v after every failed attempt:

ereport.fs.zfs.data
  zio_err = 0x34
  zio_objset = 0x0
  zio_object = 0x125
  zio_level = 0x0
  zio_blkid = 0x0
  zio_priority = 0x0 [SYNC_READ]
followed immediately by:
ereport.fs.zfs.log_replay

This exact pair repeats identically across every import attempt, regardless of flags used. zio_err=0x34 (52, ENOMSG) on object 0x125 in objset 0x0 (the MOS/root objset) during log replay, every time.

Most interesting data point: zdb -e -p /dev/disk/by-partuuid Nirala_Main_Pool completes successfully:

spa_misc.c:430:spa_load_note(): spa_load(Nirala_Main_Pool, config trusted): LOADED

So zdb can fully load the pool's trusted config from disk, which suggests the underlying data and most metadata are intact — the failure appears isolated to processing one specific intent log record during the import path itself.

Suspected trigger: Two days before discovery, I ran zpool import -f once, unsuccessfully, and I'm not fully certain if it was interrupted (system was later found in a bad state, and I subsequently reverted TrueNAS to the prior version). My best guess is this left a partially-written/corrupted ZIL record referencing object 0x125 that log replay can't process, and which none of the standard bypass mechanisms are routing around.

Question: Is there a way to force ZFS to skip/discard this specific corrupted log record (object 0x125 in the MOS) during import rather than aborting entirely? Is zfs_recover=1 expected to cover this class of MOS-level replay error, or is there a lower-level tool/patch needed here? Happy to provide full zdb -e -bcsvL output or anything else that would help diagnose.

[0xshubhs]

Decode the error code first, it changes the picture.

include/os/linux/spl/sys/errno.h:41:

#define ECKSUM  EBADE

EBADE is 52, which is 0x34. So zio_err=0x34 is ECKSUM, a checksum failure. Not an I/O error, not a missing device.

That explains the rest:

"insufficient replicas" and "devices unavailable" are what ZFS says when it can't get a valid copy. Both mirror members are there and returning data, the data just fails its checksum.

Clean SMART is expected. The drives are returning exactly what was written. What was written is bad.

-F/-FX failing means the corruption isn't only in recent txgs, the rewind candidates fail the same check.

The part that matters: a checksum failure on the root objset on both mirror halves. Independent disks don't corrupt identically. That's above the block layer, so either the version you briefly ran or memory. ZimaBlade has no ECC, and bad RAM does exactly this: checksum computed over already-corrupt data, then written correctly to both members. A mirror can't protect you when both copies are consistently wrong.

Run memtest86+ overnight before anything else. If the RAM is bad, every further import attempt is building on corrupt arithmetic.

To look without touching the pool:

zdb -e -ul /dev/disk/by-id/<member>

That gives txgs and timestamps. If any predate the upgrade, zpool import -o readonly=on -f -X -T <txg> targets one directly instead of letting -F pick. Do it on ddrescue images, not the originals, if the data matters.

Straight answer: ECKSUM on the root objset with no valid copy either side is often terminal, and -FX already failing suggests there's no consistent txg to land on. Image the disks and test the RAM before spending more attempts.

[0xshubhs]

Found it. It is spa_load_verify, but not the part I pointed you at.

spa.c:2831, at the top of the function, before anything to do with spa_load_verify_metadata:

	dsl_pool_config_enter(spa->spa_dsl_pool, FTAG);
	error = dmu_objset_find_dp(spa->spa_dsl_pool,
	    spa->spa_dsl_pool->dp_root_dir_obj, verify_dataset_name_len, NULL,
	    DS_FIND_CHILDREN);
	dsl_pool_config_exit(spa->spa_dsl_pool, FTAG);
	if (error != 0)
		return (error);

DS_FIND_CHILDREN means every dsl_dir in the pool. For each one dmu_objset_find_dp_impl calls dsl_dataset_hold_obj, and that does this at dsl_dataset.c:686:

		if (err == 0 && !ds->ds_is_snapshot) {
			err = dsl_prop_get_int_ds(ds,
			    zfs_prop_to_name(ZFS_PROP_REFRESERVATION),
			    &ds->ds_reserved);

which ends up in dsl_prop_get_dd doing zap_lookup(mos, dsl_dir_phys(dd)->dd_props_zapobj, ...). That is your read of object 293. The callback is only checking dataset name lengths, so the pool is being refused over a property lookup on a dataset nobody asked to open.

The spa_load_verify_metadata gate is 17 lines further down at :2848 and only wraps traverse_pool. Setting it to 0 skipped the scan and left the dataset walk running, which is why nothing changed. My mistake, I pointed you at the wrong half of that function.

-N cannot help either. It is ZFS_IMPORT_ONLY and only affects mounting, which happens after the import has already succeeded.

Two things in your own output that matter. The second ereport is now ereport.fs.zfs.zpool, not log_replay. That is the generic load failure, and it means readonly did work exactly as predicted and spa_ld_verify_logs is out of the picture. The failure is now spa_ld_verify_pool_data at spa.c:5508, which returns spa_vdev_err(rvd, VDEV_AUX_CORRUPT_DATA, ECKSUM). libzfs maps ECKSUM to EZFS_CKSUM, and EZFS_CKSUM is literally the string "insufficient replicas" (libzfs_util.c:176). The "destroy and re-create" line is the no_info fallback at the bottom of zpool_explain_recover. So that message was never about your devices.

You asked how to see which function. It is already being logged:

echo 0 > /proc/spl/kstat/zfs/dbgmsg
zpool import -o readonly=on -N -f -R /mnt/recover Nirala_Main_Pool
cat /proc/spl/kstat/zfs/dbgmsg

spa_load_failed() and spa_load_note() (spa_misc.c:407 and :421) call zfs_dbgmsg unconditionally, and zfs_dbgmsg_enable defaults to on (zfs_debug.c:53). You get one line per load stage and FAILED: spa_load_verify failed [error=52] at the end. Same format as the LOADED line you already got out of zdb.

Now the part you should know about, because it changes what we thought we knew. Your zdb -dddd ... 0 293 did not show that object 293 reads fine. dump_zap_stats at zdb.c:983:

	error = zap_get_stats(os, object, &zs);
	if (error)
		return;

and the dump loop immediately after it:

	for (zap_cursor_init(&zc, os, object);
	    zap_cursor_retrieve(&zc, attrp) == 0;
	    zap_cursor_advance(&zc)) {

Both give up silently on a read error. Everything you reported back (type, maxblkid, fill, dsize) comes from the dnode, which lives in a different block. dnode maxblkid is printed at zdb.c:3956 and the ZAP viewer is called at :3968. You saw the last line before the read and then no entries. That was a failed read presented as a clean one.

That also answers why zdb loads the pool when the kernel will not. zdb.c:9235:

	int rewind = ZPOOL_NEVER_REWIND;

It goes into the import config at :9644, and spa_load_verify bails on it in its first statement, spa.c:2826:

	if (policy.zlp_rewind & ZPOOL_NEVER_REWIND ||
	    policy.zlp_maxmeta == UINT64_MAX)
		return (0);

zdb never runs the dataset walk at all. That is the entire difference between the two. And you cannot get there from zpool import: -F and -X set ZPOOL_DO_REWIND and ZPOOL_EXTREME_REWIND, not NEVER, and nothing in cmd/ or lib/ ever sets ZPOOL_LOAD_META_THRESH, so the maxmeta door is unreachable too. There is no module parameter for this and no way to exclude a single object.

So stop importing. zdb has a tool for exactly this situation, -B:

Generate a backup stream, similar to zfs send, but for the numeric objset ID, and without opening the dataset. This can be useful in recovery scenarios if dataset metadata has become corrupted but the dataset itself is readable.

Get the objset IDs:

zdb -e -p /dev/disk/by-partuuid -d Nirala_Main_Pool

then one dataset at a time:

zdb -e -p /dev/disk/by-partuuid -B Nirala_Main_Pool/<objsetid> > /mnt/elsewhere/archived.zsend

It refuses to write to a terminal, so redirect it. zstream dump the file to sanity check it, then zfs recv -F newpool/Archived < archived.zsend.

It still does dsl_dataset_hold_obj on the one objset you name, so it will fail on 389 and 400. Those are samba4 and netdata. Archived, Photos and Documents have their own props objects and nothing in your traversal touched them.

[DamalZoha]

@0xshubhs

Tried the zdb -B extraction on a small test dataset before committing to the full pool. Documents (objset 66928, live dataset, 814M, 918 objects, no encryption markers in zdb -d output):

$ whoami
truenas_admin
$ id
uid=950(truenas_admin) gid=950(truenas_admin) groups=950(truenas_admin),544(builtin_administrators)

$ sudo chown truenas_admin:truenas_admin /mnt/usb_test

$ sudo zdb -e -p /dev/disk/by-partuuid -B Nirala_Main_Pool/66928 > /mnt/usb_test/documents_test.zsend
dump_backup: dmu_send_obj: Permission denied
EXIT: 0

$ ls -la /mnt/usb_test/documents_test.zsend
-rw-r--r-- 1 truenas_admin truenas_admin 0 Aug 26 15:01 /mnt/usb_test/documents_test.zsend

File came out at 0 bytes — total failure, not partial. Exit code was 0 despite the error, so the error print isn't treated as fatal by zdb itself, just silently produces nothing.

Ran sudo dmesg -T | tail -30 right after — nothing relevant, no AppArmor denials, no other system-level access errors logged around that timestamp. TrueNAS SCALE does run AppArmor profiles generally (visible at boot), but nothing fired for this specific operation.

Also confirmed via zdb -e -p /dev/disk/by-partuuid -d Nirala_Main_Pool/Archived/Documents that this dataset shows no encryption in its output, in case that was relevant.

Given the sudo/root context, this doesn't look like a Linux filesystem permissions issue — seems like something inside dmu_send_obj itself is refusing. Is there a known reason zdb -B would hit a permission check here even running as root? Missing pool-level property, some interaction with the dataset being under an encrypted parent (Archived's parent tree), or something else in that code path?

[DamalZoha]

@0xshubhs

After posting the previous post; while waiting, tested -B on a few more datasets to see if the failure was specific to Documents:

$ sudo zdb -e -p /dev/disk/by-partuuid -B Nirala_Main_Pool/35233 > /mnt/usb_test/calibre_test.zsend
dump_backup: dmu_send_obj: Permission denied
EXIT: 0
(0 bytes)

$ sudo zdb -e -p /dev/disk/by-partuuid -B Nirala_Main_Pool/410 > /mnt/usb_test/damals_test.zsend
dump_backup: dmu_send_obj: Permission denied
EXIT: 0
(0 bytes)

$ sudo zdb -e -p /dev/disk/by-partuuid -B Nirala_Main_Pool/9884 > /mnt/usb_test/archived_test.zsend
dump_backup: dmu_send_obj: Permission denied
EXIT: 0
(0 bytes)

Tested: Calibre (35233, unencrypted, top-level under Archived), Damals_closet (410, encrypted), and Archived itself (9884, the parent dataset). All three fail identically to Documents — same error, same 0-byte result, same exit code 0. So this doesn't appear tied to encryption or to a dataset's position in the hierarchy — looks like -B is failing universally on this pool/system.

[DamalZoha]

@0xshubhs

also, Confirmed zdb/ZFS module version: 2.3.4-1 (from /sys/module/zfs/version; this build doesn't support --version or -V for a version string, and dpkg -l | grep zfsutils returned nothing, so that's the only version info I have).

Tried -G to dump the debug buffer alongside -B, hoping it would show the source of the permission denial:

$ sudo zdb -e -p /dev/disk/by-partuuid -G -B Nirala_Main_Pool/66928 > /mnt/usb_test/documents_test2.zsend
dump_backup: dmu_send_obj: Permission denied
ZFS_DBGMSG(zdb) START:
[... full pool load trace, ending in ...]
spa_misc.c:430:spa_load_note(): spa_load(Nirala_Main_Pool, config trusted): LOADED
spa.c:9320:spa_async_request(): spa=Nirala_Main_Pool async request task=32
ZFS_DBGMSG(zdb) END
EXIT: 0

Pool load trace is identical to previous successful LOADED runs — nothing unusual there. But the debug buffer contains nothing about the dmu_send_obj: Permission denied error itself — that message isn't going through zfs_dbgmsg at all, so whatever's failing seems to be a simpler/lower-level check inside the send-stream code path, not something logged via the pool-load tracing.

Also tested across multiple datasets to rule out it being specific to one: Documents (66928, unencrypted), Calibre (35233, unencrypted), Damals_closet (410, encrypted), and Archived itself (9884, parent dataset) — all four fail identically, same error text, same 0-byte output, same exit code 0. So this doesn't appear tied to encryption or dataset hierarchy position — looks like -B is failing universally on this pool under this zdb build/environment.

Running as truenas_admin via sudo throughout (confirmed uid=950, groups include builtin_administrators). Any idea what permission check dmu_send_obj might be hitting here that wouldn't be satisfied even by root, or a way to get more detail on that specific failure (short of instrumenting the binary)?

And, Tried -O and -r as a possible alternate path, since they're a different code path than -B:

$ sudo zdb -e -p /dev/disk/by-partuuid -O Nirala_Main_Pool/Archived/Documents /
failed to lookup dataset=Nirala_Main_Pool/Archived/Documents path=/: Input/output error

$ sudo zdb -e -p /dev/disk/by-partuuid -r Nirala_Main_Pool/Archived/Documents / /mnt/usb_test/root_test
failed to lookup dataset=Nirala_Main_Pool/Archived/Documents path=/: Input/output error
zdb: internal error: Input/output error

Different error than -B gives (Input/output error vs Permission denied) — so these two code paths seem to be failing for different reasons, even though both end up blocked.

Separately: tried dropping into a true root shell (sudo -i) rather than prefixing each command with sudo, to rule out anything sudo-context-specific, and re-ran the -B test. The SSH session itself dropped/reconnected partway through (no output captured for that specific run — file documents_test3.zsend exists but is 0 bytes, consistent with every other -B attempt). Checked journalctl -k for the relevant window afterward — nothing logged, no kernel errors, no crash signatures. Pool drives (sda/sdb) confirmed completely unchanged before and after. Given the clean kernel log, this looks like an unrelated SSH/session hiccup rather than anything caused by zdb itself, but flagging it in case it's meaningful.

At this point I've tested -B, -O, and -r across multiple datasets (encrypted and not, parent and child) with consistent failures on each. Wanted to check in before trying anything further — is there a next diagnostic step you'd suggest, or should I hold here?

[DamalZoha]

@0xshubhs

Hi — no pressure at all given how much time you've already put into this (genuinely appreciate the source-level detail, it's taught me a lot about how import verification actually works). Just checking in in case this thread slipped by, since it's been quiet for a few days.

Current state, for reference: still stuck on the zdb -B → dmu_send_obj: Permission denied issue, consistent across every dataset I've tried (encrypted, unencrypted, parent, child), 0-byte output, exit code 0, nothing relevant in the zfs_dbgmsg buffer even with -G. Also tried -O/-r as an alternative — different error there (Input/output error on path lookup), which might be a separate issue.

No rush — the data isn't time-sensitive, and I'm holding off on touching the pool further until I hear back. Whenever you have a chance, would appreciate any pointers on that permission check.