Persistent zpool status errors after deleting corrupted files (links=0, no namespace entry, scrubs/clear ineffective)

[cronald-sys]

• OpenZFS version: zfs-2.3.4-1
• TrueNAS SCALE 25.10.4
• RAIDZ2 pool of 8 disks
• No dedup
• xattrs = sa

A few weeks ago my pool experienced corruption during a hardware issue (later traced to a temporary connectivity problem with my HBA). The affected files were successfully identified by zpool status -v.

Those files were subsequently deleted because they were media files that could simply be restored from another source. At the time they were deleted there were no snapshots containing those files. Since then, the pool has resilvered successfully, multiple successful scrubs have completed, zpool scrub -e completed, many zpool clear operations have been performed . All devices report 0 read/write/checksum errors, and the pool is otherwise operating normally. However, zpool status -v continues to report the deleted files as permanent errors, leaving the pool in an unhealthy state.

zpool status -v still reports:

MegaPool-v2/MegaPool/TV_Shows:<0xc74>
MegaPool-v2/MegaPool/TV_Shows:<0xc76>
MegaPool-v2/MegaPool/TV_Shows:<0x1282>
MegaPool-v2/MegaPool/TV_Shows:<0xde6>

A normal scrub reports "scrub repaired 0B". An error scrub reports: "scrubbed 43 error blocks", but the four entries remain.

Using zdb, each affected object still exists. each reports "type: ZFS plain file
links: 0". Despite this, the dnodes remain allocated and zbd -ddddd can still fully walk the objects' metadata and block tree. The parent directories still exist and are healthy. The parent ZAPs no longer reference these object IDs. For example, Season 6, episode x = object 3187, episode y = object 3183. The orphaned objects are not present in the directory listing. All four affected objects exhibit the same behavior. The objects are no longer reachable through the filesystem namespace, for example, "find /mnt/MegaPool-v2/MegaPool/TV_Shows -xdev -inum 3188" returns no results. The objects still have valid metadata (uid, size, timestamps, etc) and zbd -ddddd can still walk the object.

The affected objects were deleted before any snapshots existed that referenced them. Current snapshots were created weeks later, so snapshots should not be retaining these objects.

Is this an expected state for orphaned objects after corruption, or does this appear to be an edge case in object cleanup? Is there a supported mechanism to complete cleanup of these objects?

[tonyhutter]

My guess is that the internal error logs need to be cleared. I don't know if a scrub will actually check to see if those error log entries are still valid, but in an ideal world it would.

Does zpool clear MegaPool-v2 clear those error log entries (the corrupted file lines)?