some copied files are corrupted (chunks replaced by zeros)

[terinjokes]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	(rolling)
Kernel Version	6.5.11
Architecture	amd64
OpenZFS Version	2.2.0
Reference	https://bugs.gentoo.org/917224

Describe the problem you're observing

When installing the Go compiler with Portage, many of the internal compiler commands have been corrupted by having most of the files replaced by zeros.

$  file /usr/lib/go/pkg/tool/linux_amd64/* | grep data
/usr/lib/go/pkg/tool/linux_amd64/asm:       data
/usr/lib/go/pkg/tool/linux_amd64/cgo:       data
/usr/lib/go/pkg/tool/linux_amd64/compile:   data
/usr/lib/go/pkg/tool/linux_amd64/covdata:   ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=xHCzRQtrkEP-Bbxql0SF/zxsofCJFlBoPlUclgwBG/TrsgK6SKiY4q6TIhyBjU/UwcISvZgqfQaEf3Kr_Tq, not stripped
/usr/lib/go/pkg/tool/linux_amd64/cover:     data
/usr/lib/go/pkg/tool/linux_amd64/link:      data
/usr/lib/go/pkg/tool/linux_amd64/vet:       data

$ hexdump /usr/lib/go/pkg/tool/linux_amd64/compile
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000fa0 0000 0000 0000 0000 0000 0000 5a41 3447
0000fb0 336a 3933 5a49 4f2d 6641 6342 7a6d 3646
0000fc0 582f 5930 5a4d 6761 5659 6f34 6d39 4130
0000fd0 4957 6555 2f67 686d 6a63 6675 5976 4e6a
0000fe0 346c 3070 5157 494e 5f41 5a2f 336d 6342
0000ff0 4e6d 4a4f 306c 4277 4a72 774d 4d41 006c
0001000 0000 0000 0000 0000 0000 0000 0000 0000
*
0ac9280 5a41 3447 336a 3933 5a49 4f2d 6641 6342
0ac9290 7a6d 3646 582f 5930 5a4d 6761 5659 6f34
0ac92a0 6d39 4130 4957 6555 2f67 686d 6a63 6675
0ac92b0 5976 4e6a 346c 3070 5157 494e 5f41 5a2f
0ac92c0 336d 6342 4e6d 4a4f 306c 4277 4a72 774d
0ac92d0 4d41 006c 0000 0000 0000 0000 0000 0000
0ac92e0 0000 0000 0000 0000 0000 0000 0000 0000
*
1139380 0000 0000 0000 0000 0000
1139389

I'm able to reproduce on two separate machines running 6.5.11 and ZFS 2.2.0.

ZFS does not see any errors with the pool.

$ zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0

errors: No known data errors

$ zpool status -t
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0  (100% trimmed, completed at Tue 31 Oct 2023 11:15:47 PM GMT)

errors: No known data errors

Describe how to reproduce the problem

1. On a system running ZFS 2.2.0, upgrade pools to enable the block cloning feature.
2. emerge -1 dev-lang/go, where Portage's TMPDIR is on ZFS.
3. After a successful install of Go, the files in /usr/lib/go/pkg/tool/linux_amd64/compile are corrupted.

I was able to reproduce with and without Portage's "native-extensions" feature. I was unable to reproduce after changing Portage's TMPDIR to another filesystem (such as tmpfs).

Include any warning/errors/backtraces from the system logs

[thulle]

I'm also hit by this bug, happened after upgrade to ZFS 2.2.0. Mounting tmpfs on portage TMPDIR (build directory for go) solves the issue. Currently on kernel 6.5.9, originally happened on 6.5.7.

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

[terinjokes]

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption. My understanding is 8.32 predates the switch to automatically using reflink?

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

This is the Go BuildID. Compare the output of a non-corrupted build:

$ file /usr/lib/go/pkg/tool/linux_amd64/compile
/usr/lib/go/pkg/tool/linux_amd64/compile: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=AZG4j339IZ-OAfBcmzF6/X0YMZagYV4o9m0AWIUeg/mhcjufvYjNl4p0WQNIA_/Zm3BcmNOJl0wBrJMwAMl, not stripped

With the data remaining in the file:

$ hexdump -C /var/tmp/portage/dev-lang/go-1.21.4/image/usr/lib/go/pkg/tool/linux_amd64/compile
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000fa0  00 00 00 00 00 00 00 00  00 00 00 00 41 5a 47 34  |............AZG4|
00000fb0  6a 33 33 39 49 5a 2d 4f  41 66 42 63 6d 7a 46 36  |j339IZ-OAfBcmzF6|
00000fc0  2f 58 30 59 4d 5a 61 67  59 56 34 6f 39 6d 30 41  |/X0YMZagYV4o9m0A|
00000fd0  57 49 55 65 67 2f 6d 68  63 6a 75 66 76 59 6a 4e  |WIUeg/mhcjufvYjN|
00000fe0  6c 34 70 30 57 51 4e 49  41 5f 2f 5a 6d 33 42 63  |l4p0WQNIA_/Zm3Bc|
00000ff0  6d 4e 4f 4a 6c 30 77 42  72 4a 4d 77 41 4d 6c 00  |mNOJl0wBrJMwAMl.|
00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00ac9280  41 5a 47 34 6a 33 33 39  49 5a 2d 4f 41 66 42 63  |AZG4j339IZ-OAfBc|
00ac9290  6d 7a 46 36 2f 58 30 59  4d 5a 61 67 59 56 34 6f  |mzF6/X0YMZagYV4o|
00ac92a0  39 6d 30 41 57 49 55 65  67 2f 6d 68 63 6a 75 66  |9m0AWIUeg/mhcjuf|
00ac92b0  76 59 6a 4e 6c 34 70 30  57 51 4e 49 41 5f 2f 5a  |vYjNl4p0WQNIA_/Z|
00ac92c0  6d 33 42 63 6d 4e 4f 4a  6c 30 77 42 72 4a 4d 77  |m3BcmNOJl0wBrJMw|
00ac92d0  41 4d 6c 00 00 00 00 00  00 00 00 00 00 00 00 00  |AMl.............|
00ac92e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01139380  00 00 00 00 00 00 00 00  00                       |.........|
01139389

[RichardBelzer]

Had the same issue here... Upgraded from 2.1.13 to 2.2.0 last month right after it came out. I'm on Ubuntu and built the ZFS package from the 2.2.0 tag.

I am discovering random files that were being stored after the 2.2.0 upgrade have suffered silent corruption and either have repetitious data like @thulle reported or have large blocks of contiguous zeroes instead of the data that they should be filled with. I don't use or build go like the other posters.

zpool scrub reports no errors, zpool status reports no errors. Let me know if there is any other information needed.

[thulle]

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption.

This seems to solve the corruption issue on my end too.

[thesamesam]

#11900 (comment) onwards is highly relevant. See also #14753 and #11900 as a whole.

It would be interesting to know if people here have all upgraded their pool for the new block cloning feature or if there's a mix here.

[thulle]

feature@block_cloning active here

[RichardBelzer]

My understanding is that block cloning is enabled by default when upgrading to 2.2.0, but my not be getting used unless an application like coreutils is actually leveraging it.

Can people post the result of this:

zpool get all tank | grep bclone

(where tank is the name of their pool with the corruption)

[thulle]

kc3000    bcloneused                     442M                           -
kc3000    bclonesaved                    1.42G                          -
kc3000    bcloneratio                    4.30x                          -

[RichardBelzer]

My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.

Any non-zero number for either field and you may OR may not have silent corruption.

[rincebrain]

That is likely but not really safe to say yet, we don't have enough data.

[terinjokes]

but my not be getting used unless an application like coreutils is actually leveraging it.

Is it accurate to say it's being used when the FICLONE ioctl is used, and possibly when copy_source_range?

[rincebrain]

I wouldn't even feel comfortable saying that yet, I agree it's likely, but I'd really like more certainty before declaring "you're safe if not this".

[RichardBelzer]

Agreed, we don't know what we don't know, we need to:

1. Get a machine / test case to repro it most of the time (Ideally 100%). I think we're here already here with newer coreutils and building go.
2. Determine what we think is the bug and produce a patch
3. Patch the source, rerun the test(s), verify that this now happens 0% of the time

I don't know if there's a more robust way of figuring this out outside of that. And of course I'm handwaving 2 which could be a heavy lift.

[rincebrain]

I'd assume, not having looked, that it's something like copy_file_range isn't dirtying things so the thing that triggers a force txg sync on SEEK_DATA/SEEK_HOLE with a dirty thing isn't firing.

But as I'm somewhat occupied recovering from being badly ill at home, my time to test and debug this is rather finite, so I wouldn't assume I'll be doing that in a timely fashion.

[robn]

I agree with @rincebrain on the rough theory. I did look, and landed on code I've looked at before while trying to get my head around corner cases in block cloning. I have been suspicious of it for a while now, and remain so.

When we start to clone a block, we indicate intent to modify the target dbuf by calling dmu_buf_will_clone, which is a special case that unwinds any changes on the dbuf, and then sets it to DB_NOFILL.

In dmu.c:

void
dmu_buf_will_clone(dmu_buf_t *db_fake, dmu_tx_t *tx)
{
	dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;

	/*
	 * Block cloning: We are going to clone into this block, so undirty
	 * modifications done to this block so far in this txg. This includes
	 * writes and clones into this block.
	 */
	mutex_enter(&db->db_mtx);
	VERIFY(!dbuf_undirty(db, tx));
	ASSERT0P(dbuf_find_dirty_eq(db, tx->tx_txg));
	if (db->db_buf != NULL) {
		arc_buf_destroy(db->db_buf, db);
		db->db_buf = NULL;
	}
	mutex_exit(&db->db_mtx);

	dmu_buf_will_not_fill(db_fake, tx);
}

void
dmu_buf_will_not_fill(dmu_buf_t *db_fake, dmu_tx_t *tx)
{
	dmu_buf_impl_t *db = (dmu_buf_impl_t *)db_fake;

	mutex_enter(&db->db_mtx);
	db->db_state = DB_NOFILL;
	DTRACE_SET_STATE(db, "allocating NOFILL buffer");
	mutex_exit(&db->db_mtx);

	dbuf_noread(db);
	(void) dbuf_dirty(db, tx);
}

It seems there's a window there where the lock is down and the dbuf is not-dirty. That may be a place where a second thread can add a change to that block, which then gets trampled.

Unfortunately dbuf_dirty and dbuf_undirty are extremely complex, and the gap is tiny, and without a tight test case its gonna take me a lot of time to track this down, which I don't have at the moment¹. So this is an unhelpful comment but maybe it gives someone else an idea of where to look.

Footnotes

1. but you can buy some of my time 😉 ↩

[grahamperrin]

#15526 (comment)

My understanding is that block cloning is enabled by default when upgrading to 2.2.0, …

With FreeBSD 14.0-RELEASE, there's complementary use of the vfs.zfs.bclone_enabled kernel state, 0 (zero):

root@mowa219-gjp4-freebsd-14-zfs-vm:~ # sysctl -d vfs.zfs.bclone_enabled
vfs.zfs.bclone_enabled: Enable block cloning
root@mowa219-gjp4-freebsd-14-zfs-vm:~ # sysctl vfs.zfs.bclone_enabled
vfs.zfs.bclone_enabled: 0
root@mowa219-gjp4-freebsd-14-zfs-vm:~ # freebsd-version -kru ; uname -aKU
14.0-RELEASE
14.0-RELEASE
14.0-RELEASE
FreeBSD mowa219-gjp4-freebsd-14-zfs-vm 14.0-RELEASE FreeBSD 14.0-RELEASE releng/14.0-n265380-f9716eee8ab GENERIC amd64 1400097 1400097
root@mowa219-gjp4-freebsd-14-zfs-vm:~ # pkg upgrade -r FreeBSD-base
Updating FreeBSD-base repository catalogue...
FreeBSD-base repository is up to date.
All repositories are up to date.
Checking for upgrades (0 candidates): 100%
Processing candidates (0 candidates): 100%
Checking integrity... done (0 conflicting)
Your packages are up to date.
root@mowa219-gjp4-freebsd-14-zfs-vm:~ # exit
logout
grahamperrin@mowa219-gjp4-freebsd-14-zfs-vm:~ % pkg -vv | grep -e url -e enabled
    url             : "pkg+https://pkg.freebsd.org/FreeBSD:14:amd64/latest",
    enabled         : yes,
    url             : "pkg+https://pkg.freebsd.org/FreeBSD:14:amd64/base_release_0",
    enabled         : yes,
grahamperrin@mowa219-gjp4-freebsd-14-zfs-vm:~ % 

sysctl(8)

Whilst 14.0 is not yet announced, I can't imagine a change to the value at this time.

[rincebrain]

Time to upstream that and add Linux support, with how broken this is.

[KungFuJesus]

I am curious if it is broken in the same way on FreeBSD. Has anyone managed to reproduce this with intentional reflink copies? From what I recall the implementer's main platform was FreeBSD.

Also some of these stack traces are blown assertions in the ZIL. Are we certain all of these are in regard to the BRT feature? I know a lot of the more recent optimizations with regard to ZIL messed with lock granularity, do we know that these things haven't happened with BRT not being leveraged?

[rincebrain]

Here is probably not the right place to debate whether the other bugs around the ZIL are or are not from BRT. At least in 15529, I only tagged the ones where it was explicitly the case that block cloning was happening.

I'm also not sure what you mean by "intentional reflink support" here - in both FreeBSD and Linux's case, they're calling a copy_file_range analogue that is going to reflink if it can and do a boring copy otherwise, nobody here is explicitly trying FICLONE, which has last I checked no analogue on FreeBSD at all.

[terinjokes]

The cp from coreutils-9.3 is using FICLONE, and falling back to copy_file_range if it fails.

[KungFuJesus]

By intentional I mean doing cp with a reflink argument, assuming the BSD variant of cp has it. I'd expect that by default it doesn't try to use it automatically like coreutils' --reflink=auto behavior. When something like make is copying things around with quickly generated build artifacts, I expect you're dealing with code that is likely to contend race conditions more than a file that's been stable on disk for a few seconds and is being copied once with the BRT backed clone feature.

[rincebrain]

My understanding is that cp on FreeBSD just invokes copy_file_range in every case, and as I said, has no FICLONE analogue.

[KungFuJesus]

Ah gotcha, so the benefits of the reduced syscall are there without requiring repeated userspace / kernelspace round trips, but the BRT based shortcut and space savings are prohibited behind that sysctl. Makes sense. I am curious if FICLONE is the issue here and copy_file_range worked just fine?

[rincebrain]

Since the code, if I'm reading it right, for FICLONE just invokes the same backend function, it shouldn't be FICLONE specifically. I would guess differing semantics about how easy races are to trigger between the two platforms' memory management, but I really don't know.

[tonyhutter]

Pinging @pjd on block cloning.

I'm seeing if I can make a non-gentoo reproducer.

[thulle]

Since I have 1.42GB of possibly affected files I thought I'd check for stretches of zeroes in the other files to see if anything other than compiling go might have triggered this.
The block cloning commit only seem to change zdb code regarding the ZIL, so I just want to verify that we have no way of dumping the BRT to check which files contain cloned blocks, right?

Currently dumping a list of all file blocks to check for multiple references to the same block.

[rincebrain]

I don't think zdb knows how to dump it as it is now, no. I don't think it'd be hard to teach it, though.

[robn]

Its not hard to dump, but also it doesn't really show what you want. Each entry is just the offset within a vdev (half a DVA), and the number of references to it. It doesn't know what a file is, nor even really what a block pointer is.

Probably the dumbest version is to extend the in-memory mini-BRT used in zdb -b to note the object in which cloned blocked were seen, and then dump those out at the end. It does mean a full scan though (about the same heft as a scrub).

Hmm, now I think about it, it might not help anyway. If the problem is that there was a race, and we applied a change in the wrong order, such that zeroes got written, then the clone itself was never performed - we did a real write, just with zero content. So there's no clone to look for.

[numinit]

correct, it will generally only replicate with a coreutils that is trying the SEEK_HOLE optimization.

[ipkpjersi]

I'm using coreutils 8.32, so basically any distro that isn't using coreutils 9.x is completely unaffected by this issue?

[classabbyamp]

not completely, but it takes away many possible points of creating corruption (this also assumes the distro didn't backport the SEEK_HOLE changes, like EL9 did, iirc)

[Ukko-Ylijumala]

I'm using coreutils 8.32, so basically any distro that isn't using coreutils 9.x is completely unaffected by this issue?

To add to @classabbyamp answer, any software (besides coreutils) which tries to find holes in files it is both reading and writing at the same time is potentially affected. However, the potential timing window is so small and the access pattern so specific that the likelihood of actually tripping on this bug is rather small.

[ipkpjersi]

In that case, would zfs send/recv be affected and would rsync be affected? Or would that only really be the case if coreutils 9.x is present?

I did find some ppas I could look into like ppa:arter97/zfs and ppa:patrickdk/zfs and ppa:ofthesun/zfs at least so I can probably patch my Ubuntu systems, but I have a feeling patching my Proxmox systems may require me learning how to build zfs from source, although I did find kneutron/ansitest has a couple zfs build scripts so maybe that will help me learn building zfs from source.

Thanks for all the additional information, it's quite interesting learning about this bug even though it's been patched already.

Edit: Sounds like there's a newer Proxmox with this bug already fixed/backported so that's good at least.

Edit 2: I have gone with ppa:arter97/zfs and upgraded my Ubuntu systems to zfs 2.2.2 and upgraded my Proxmox systems to Proxmox 8.1 which supposedly includes the zfs fix backported to 2.2.0, so I suppose all is well for me now.

[alpha754293]

I have been testing this, using this command: parallel --lb --halt-on-error now,fail=1 ./zhammer.sh /data 10000000 16k 10000 ::: $(seq $(nproc))

I haven't been able to replicate this issue using zhammer on a PC running Ubuntu 22.04 with kernel 6.2.0-37-generic, zfs-2.1.6-0york1~22.04, and zfs-kmod-2.1.9-2ubuntu1.1.

I also haven't been able to replicate it using zhammer on another PC running Proxmox pve-manager/7.4-17/513c62be (running kernel: 5.15.131-1-pve) with zfs-2.1.11-pve1 and zfs-kmod-2.1.11-pve1.

I thought that this issue definitely affected version 2.1.11 and above, and possibly even earlier versions (like the 2.1.6/2.1.9 I am running on my other PC) - or am I mistaken and this is not the case?

Same thing here.

I am using Proxmox 7.4-3 with coreutils version 8.32-4+b1 and zfsutils-linux: 2.1.11-pve1 and zfs-2.1.11-pve1 and zfs-kmod-2.1.11-pve1 and I've tested 640 million files (64 threads * 10 million files/thread) and haven't been able to replicate the issue on my end neither.

When I check my Solaris 10 1/13 system, GNU coreutils is NOT installed by default.

As far as I can so, it looks like (until I can find information otherwise, or someone else can educate me) that the core software libraries used for system administration is installed under the Solaris package SUNWadmc, so I am guessing that that's where the tool cp for Solaris comes through (rather than through GNU coreutils).

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

Thanks.

[0x5c]

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

[robszy]

Anyone can answer my question in comment: #15526 (comment) ?

[alpha754293]

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

Thank you, @0x5c.

Unfortunately, since Oracle bought Sun Microsystems, Solaris has been closed source since 2010, so it would be impossible to tell from the outside.

[0x5c]

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

Thank you, @0x5c.

Unfortunately, since Oracle bought Sun Microsystems, Solaris has been closed source since 2010, so it would be impossible to tell from the outside.

If the bug is present in that version of ZFS, then it should theoretically be possible to trigger with anything that seeks to the next hole in a file, regardless of what that program may be. If coreutils 9.x can build for solaris, then that could be a way to try to reproduce the bug. Even a minimal program that just does the equivalent to coreutils cp would be enough.

[robszy]

I found out why second copy from my comment #15526 (comment) is done by seeking hole/data:

ZFS reports number of blocks that is less than size of file thats why second copy is sparse but it shouldn't. Of course it whouldn't be good for reproducer but the real question:

Do we have another bug in zfs in reporting number of blocks ?

BEcause of that cp doesn't to copy_file_Range as it should and perform faster :)
For me it should report real number of blocks.

[RichardBelzer]

We're still hitting this bug in 2.2.3 and it hasn't been fully fixed yet. I found this issue here, more people are still experiencing it: #15933

Just wanted to keep people aware and to keep checking your data for silent corruption.