some copied files are corrupted (chunks replaced by zeros)

[terinjokes]

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	(rolling)
Kernel Version	6.5.11
Architecture	amd64
OpenZFS Version	2.2.0
Reference	https://bugs.gentoo.org/917224

Describe the problem you're observing

When installing the Go compiler with Portage, many of the internal compiler commands have been corrupted by having most of the files replaced by zeros.

$  file /usr/lib/go/pkg/tool/linux_amd64/* | grep data
/usr/lib/go/pkg/tool/linux_amd64/asm:       data
/usr/lib/go/pkg/tool/linux_amd64/cgo:       data
/usr/lib/go/pkg/tool/linux_amd64/compile:   data
/usr/lib/go/pkg/tool/linux_amd64/covdata:   ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=xHCzRQtrkEP-Bbxql0SF/zxsofCJFlBoPlUclgwBG/TrsgK6SKiY4q6TIhyBjU/UwcISvZgqfQaEf3Kr_Tq, not stripped
/usr/lib/go/pkg/tool/linux_amd64/cover:     data
/usr/lib/go/pkg/tool/linux_amd64/link:      data
/usr/lib/go/pkg/tool/linux_amd64/vet:       data

$ hexdump /usr/lib/go/pkg/tool/linux_amd64/compile
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000fa0 0000 0000 0000 0000 0000 0000 5a41 3447
0000fb0 336a 3933 5a49 4f2d 6641 6342 7a6d 3646
0000fc0 582f 5930 5a4d 6761 5659 6f34 6d39 4130
0000fd0 4957 6555 2f67 686d 6a63 6675 5976 4e6a
0000fe0 346c 3070 5157 494e 5f41 5a2f 336d 6342
0000ff0 4e6d 4a4f 306c 4277 4a72 774d 4d41 006c
0001000 0000 0000 0000 0000 0000 0000 0000 0000
*
0ac9280 5a41 3447 336a 3933 5a49 4f2d 6641 6342
0ac9290 7a6d 3646 582f 5930 5a4d 6761 5659 6f34
0ac92a0 6d39 4130 4957 6555 2f67 686d 6a63 6675
0ac92b0 5976 4e6a 346c 3070 5157 494e 5f41 5a2f
0ac92c0 336d 6342 4e6d 4a4f 306c 4277 4a72 774d
0ac92d0 4d41 006c 0000 0000 0000 0000 0000 0000
0ac92e0 0000 0000 0000 0000 0000 0000 0000 0000
*
1139380 0000 0000 0000 0000 0000
1139389

I'm able to reproduce on two separate machines running 6.5.11 and ZFS 2.2.0.

ZFS does not see any errors with the pool.

$ zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0

errors: No known data errors

$ zpool status -t
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0  (100% trimmed, completed at Tue 31 Oct 2023 11:15:47 PM GMT)

errors: No known data errors

Describe how to reproduce the problem

1. On a system running ZFS 2.2.0, upgrade pools to enable the block cloning feature.
2. emerge -1 dev-lang/go, where Portage's TMPDIR is on ZFS.
3. After a successful install of Go, the files in /usr/lib/go/pkg/tool/linux_amd64/compile are corrupted.

I was able to reproduce with and without Portage's "native-extensions" feature. I was unable to reproduce after changing Portage's TMPDIR to another filesystem (such as tmpfs).

Include any warning/errors/backtraces from the system logs

[thulle]

I'm also hit by this bug, happened after upgrade to ZFS 2.2.0. Mounting tmpfs on portage TMPDIR (build directory for go) solves the issue. Currently on kernel 6.5.9, originally happened on 6.5.7.

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

[terinjokes]

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption. My understanding is 8.32 predates the switch to automatically using reflink?

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

This is the Go BuildID. Compare the output of a non-corrupted build:

$ file /usr/lib/go/pkg/tool/linux_amd64/compile
/usr/lib/go/pkg/tool/linux_amd64/compile: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=AZG4j339IZ-OAfBcmzF6/X0YMZagYV4o9m0AWIUeg/mhcjufvYjNl4p0WQNIA_/Zm3BcmNOJl0wBrJMwAMl, not stripped

With the data remaining in the file:

$ hexdump -C /var/tmp/portage/dev-lang/go-1.21.4/image/usr/lib/go/pkg/tool/linux_amd64/compile
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000fa0  00 00 00 00 00 00 00 00  00 00 00 00 41 5a 47 34  |............AZG4|
00000fb0  6a 33 33 39 49 5a 2d 4f  41 66 42 63 6d 7a 46 36  |j339IZ-OAfBcmzF6|
00000fc0  2f 58 30 59 4d 5a 61 67  59 56 34 6f 39 6d 30 41  |/X0YMZagYV4o9m0A|
00000fd0  57 49 55 65 67 2f 6d 68  63 6a 75 66 76 59 6a 4e  |WIUeg/mhcjufvYjN|
00000fe0  6c 34 70 30 57 51 4e 49  41 5f 2f 5a 6d 33 42 63  |l4p0WQNIA_/Zm3Bc|
00000ff0  6d 4e 4f 4a 6c 30 77 42  72 4a 4d 77 41 4d 6c 00  |mNOJl0wBrJMwAMl.|
00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00ac9280  41 5a 47 34 6a 33 33 39  49 5a 2d 4f 41 66 42 63  |AZG4j339IZ-OAfBc|
00ac9290  6d 7a 46 36 2f 58 30 59  4d 5a 61 67 59 56 34 6f  |mzF6/X0YMZagYV4o|
00ac92a0  39 6d 30 41 57 49 55 65  67 2f 6d 68 63 6a 75 66  |9m0AWIUeg/mhcjuf|
00ac92b0  76 59 6a 4e 6c 34 70 30  57 51 4e 49 41 5f 2f 5a  |vYjNl4p0WQNIA_/Z|
00ac92c0  6d 33 42 63 6d 4e 4f 4a  6c 30 77 42 72 4a 4d 77  |m3BcmNOJl0wBrJMw|
00ac92d0  41 4d 6c 00 00 00 00 00  00 00 00 00 00 00 00 00  |AMl.............|
00ac92e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01139380  00 00 00 00 00 00 00 00  00                       |.........|
01139389

[RichardBelzer]

Had the same issue here... Upgraded from 2.1.13 to 2.2.0 last month right after it came out. I'm on Ubuntu and built the ZFS package from the 2.2.0 tag.

I am discovering random files that were being stored after the 2.2.0 upgrade have suffered silent corruption and either have repetitious data like @thulle reported or have large blocks of contiguous zeroes instead of the data that they should be filled with. I don't use or build go like the other posters.

zpool scrub reports no errors, zpool status reports no errors. Let me know if there is any other information needed.

[thulle]

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption.

This seems to solve the corruption issue on my end too.

[thesamesam]

#11900 (comment) onwards is highly relevant. See also #14753 and #11900 as a whole.

It would be interesting to know if people here have all upgraded their pool for the new block cloning feature or if there's a mix here.

[thulle]

feature@block_cloning active here

[RichardBelzer]

My understanding is that block cloning is enabled by default when upgrading to 2.2.0, but my not be getting used unless an application like coreutils is actually leveraging it.

Can people post the result of this:

zpool get all tank | grep bclone

(where tank is the name of their pool with the corruption)

[thulle]

kc3000    bcloneused                     442M                           -
kc3000    bclonesaved                    1.42G                          -
kc3000    bcloneratio                    4.30x                          -

[RichardBelzer]

My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.

Any non-zero number for either field and you may OR may not have silent corruption.

[rincebrain]

That is likely but not really safe to say yet, we don't have enough data.

[terinjokes]

but my not be getting used unless an application like coreutils is actually leveraging it.

Is it accurate to say it's being used when the FICLONE ioctl is used, and possibly when copy_source_range?

[rincebrain]

I wouldn't even feel comfortable saying that yet, I agree it's likely, but I'd really like more certainty before declaring "you're safe if not this".

[RichardBelzer]

Agreed, we don't know what we don't know, we need to:

1. Get a machine / test case to repro it most of the time (Ideally 100%). I think we're here already here with newer coreutils and building go.
2. Determine what we think is the bug and produce a patch
3. Patch the source, rerun the test(s), verify that this now happens 0% of the time

I don't know if there's a more robust way of figuring this out outside of that. And of course I'm handwaving 2 which could be a heavy lift.

[rincebrain]

I'd assume, not having looked, that it's something like copy_file_range isn't dirtying things so the thing that triggers a force txg sync on SEEK_DATA/SEEK_HOLE with a dirty thing isn't firing.

But as I'm somewhat occupied recovering from being badly ill at home, my time to test and debug this is rather finite, so I wouldn't assume I'll be doing that in a timely fashion.