How to debug fs-verity corruption errors?

[marcoh00]

Hi,

I had a look at my system logs today and noticed that something seems to be not quite right with some of the fs-verity information on my /sysroot filesystem:

sudo journalctl -b | grep fs-verity
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=5984256, level=-1, want_hash=sha512:9d8200fef4a47da5d1ac8564b3ee166a77a323ae9161b61eb5601847670b88aad43887e339f9fc433e4b3a5a7db5aee071352d6c83553db61c96c8ab4674cd94, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=6377472, level=-1, want_hash=sha512:4a111154abdb6accae2b48f6e767c31b2e5da6ba3abe6f02f091f29f4bd13df5f2a20a24e5bec438fc90efde6facbed96631c172c8a90f2c7862352a97877fc7, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=6508544, level=-1, want_hash=sha512:48f103c1c27de29cd2e4166126d74c027465d85d0a426a2b26acdbb1689d4bd71aa0988f56333b6032692cf42a71d885052082f403e943ee887c997c0e5614e9, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=6897664, level=-1, want_hash=sha512:37b2b7bd7bb09a6f43cd095db34afab92d19cfc99a781ec717dd4d355ea305e4982dd60ee670b087a6a8a0aaa5d1c7686925115160e4f084081d5170f8e41482, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=7557120, level=-1, want_hash=sha512:84145e6adb70332f17369e41885d5a9e6788513db2ae7fb0405e7875ae4bf5f9d8e46b37a21ffbe5b7666fc6b8380828e898fac7ec59506ba51815d0d9e13323, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=7946240, level=-1, want_hash=sha512:9b13e7d0903d73ed62d6acf20216bfb8114afeafd7288f2b1344f9b5eaefa9f7674c97bd8475733fda9e2ce1b12ff4c3979438dcb9a49a3784d08c6e9cffd097, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 23391777): FILE CORRUPTED! pos=7815168, level=-1, want_hash=sha512:a95bf21a43dbbfcfa7e38fe0ce41bfbd3abccbbe8ca8e04556734d2135fddffafd75ed64f50d8ecfb4847a2d772aaf63196bbda822314fcbb8c34705db78f561, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 16147539): FILE CORRUPTED! pos=2228224, level=-1, want_hash=sha512:43e3994577ff5f3a125cd4e10c6b1ce21ce17617adef7ee971e3201a6f2d76b0212790470ead010b536d55cc569b048e3146efd9898625870eccc1dde6448cb5, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 16147539): FILE CORRUPTED! pos=2752512, level=-1, want_hash=sha512:b353f96c40eaf31ae4ccedf91147c56e2eec0d684718212a0e5cb5acc1a7f832eb9af5312bfef5943c6cd495ef5c3b3d6d9bdde2c6dfe73f02046b43fdb28f87, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:19 archlinux kernel: fs-verity (nvme1n1p3, inode 16147546): FILE CORRUPTED! pos=2621440, level=-1, want_hash=sha512:375115cb424855c66a0669aec56b7292ce5b04a8542cae6ec7468ce2be65928a0f66bb82ec04d35bb478c97db0406b76b6f9fb899e5295238418e743912a50c0, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:32 archlinux kernel: fs-verity (nvme1n1p3, inode 24282025): FILE CORRUPTED! pos=11927552, level=-1, want_hash=sha512:22370f4edc3c8e69f97de5b6e42c8cf755a3429328d2f3e9142bd9e294b3b5fc0531a4c012a633bc8b51b78ab629b1679c65948dd89c63e42c64821d7981f312, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:32 archlinux kernel: fs-verity (nvme1n1p3, inode 24282025): FILE CORRUPTED! pos=8519680, level=-1, want_hash=sha512:0fffc40fc9e0d1cd498756d220db8b17265c0327032b36b63b31732a457148806a701d9bbf8ce9fd9a03a8b2d1e6963b5b09175ea919a4aacf05b2b1ceb94c4e, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
Sep 01 13:01:32 archlinux kernel: fs-verity (nvme1n1p3, inode 24282025): FILE CORRUPTED! pos=11665408, level=-1, want_hash=sha512:ede2607ab80c42c05dd46dc857479c3b08465e375856dc8724df2ced43846b3400939f660a0c584d5f6da9275e499c0e8cc1278992ba4c134927c05c18737853, real_hash=sha512:2d23913d3759ef01704a86b4bee3ac8a29002313ecc98a7424425a78170f219577822fd77e4ae96313547696ad7d5949b58e12d5063ef2ee063b595740a3a12d
+ many more

This is an Arch Linux setup with kernel 7.2.2, native composefs with UKI booted via systemd-boot. It doesn't seem to effect the running system as far as I can see (yet?). /sysroot is on a btrfs filesystem and btrfs scrub doesn't show any errors. Neither do the logs close to the fs-verity corruption warnings, so the FS seems to be fine.

Any idea on how to debug which files are affected and what could be the root cause here?

Thanks!

[meyshad]

There is a useful clue in the repeated real_hash: 2d23913d...a3a12d is the SHA-512 digest of exactly 4096 zero bytes. Together with level=-1, the kernel is saying that several file-data blocks were read as zero pages where the fs-verity tree expected non-zero contents. That is more suggestive of a hole/zero-page being returned somewhere in the composefs/Btrfs/kernel path than independent random corruption in all those files.

A clean Btrfs scrub does not contradict this. Scrub checks the stored blocks against Btrfs' own checksums; fs-verity checks file contents against the Merkle root recorded when verity was enabled. The filesystem can therefore be internally checksum-consistent while fs-verity still rejects the file.

I would first map every reported inode before changing anything:

findmnt -T /sysroot -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo find /sysroot -xdev \( -inum 23391777 -o -inum 16147539 -o -inum 16147546 -o -inum 24282025 \) -print
sudo btrfs inspect-internal inode-resolve 23391777 /sysroot

With composefs the visible path may be an overlay while the inode belongs to an OSTree object, so also check /sysroot/ostree/repo/objects. For each resolved file, retrieve the enforced digest and then perform a full read; fsverity measure is constant-time and does not itself read every data block.

sudo fsverity measure /path/to/file
sudo dd if=/path/to/file of=/dev/null bs=1M status=progress
sudo ostree fsck --repo=/sysroot/ostree/repo

The full read should fail with EIO at an affected range. I would keep the inode-to-path results and compare a boot with the previous kernel/deployment before repairing or re-pulling anything. If the zero-page pattern occurs only on 7.2.2, it is strong evidence for a regression above the storage layer. The kernel's description of read verification and FS_IOC_MEASURE_VERITY is in the fs-verity documentation.

[marcoh00]

Hi,

thanks for your answer, I appreciate you trying to help.

The full read should fail with EIO at an affected range.

What's interesting is that it does not actually fail. Only the same kernel logs as above, but neither cat (one of them is a plaintext file), nor dd actually complain when reading the file. fsverity measure agrees with composefs/the directory structure.

it is strong evidence for a regression above the storage layer

Yes, that's what I assumed and that's basically why I'm asking here instead of $CHATBOT. I lack the intuition which layer exactly might be affected here that would show this pattern of fs-verity errors.