some copied files are corrupted (chunks replaced by zeros)

System information

Type	Version/Name
Distribution Name	Gentoo
Distribution Version	(rolling)
Kernel Version	6.5.11
Architecture	amd64
OpenZFS Version	2.2.0
Reference	https://bugs.gentoo.org/917224

Describe the problem you're observing

When installing the Go compiler with Portage, many of the internal compiler commands have been corrupted by having most of the files replaced by zeros.

$  file /usr/lib/go/pkg/tool/linux_amd64/* | grep data
/usr/lib/go/pkg/tool/linux_amd64/asm:       data
/usr/lib/go/pkg/tool/linux_amd64/cgo:       data
/usr/lib/go/pkg/tool/linux_amd64/compile:   data
/usr/lib/go/pkg/tool/linux_amd64/covdata:   ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=xHCzRQtrkEP-Bbxql0SF/zxsofCJFlBoPlUclgwBG/TrsgK6SKiY4q6TIhyBjU/UwcISvZgqfQaEf3Kr_Tq, not stripped
/usr/lib/go/pkg/tool/linux_amd64/cover:     data
/usr/lib/go/pkg/tool/linux_amd64/link:      data
/usr/lib/go/pkg/tool/linux_amd64/vet:       data

$ hexdump /usr/lib/go/pkg/tool/linux_amd64/compile
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000fa0 0000 0000 0000 0000 0000 0000 5a41 3447
0000fb0 336a 3933 5a49 4f2d 6641 6342 7a6d 3646
0000fc0 582f 5930 5a4d 6761 5659 6f34 6d39 4130
0000fd0 4957 6555 2f67 686d 6a63 6675 5976 4e6a
0000fe0 346c 3070 5157 494e 5f41 5a2f 336d 6342
0000ff0 4e6d 4a4f 306c 4277 4a72 774d 4d41 006c
0001000 0000 0000 0000 0000 0000 0000 0000 0000
*
0ac9280 5a41 3447 336a 3933 5a49 4f2d 6641 6342
0ac9290 7a6d 3646 582f 5930 5a4d 6761 5659 6f34
0ac92a0 6d39 4130 4957 6555 2f67 686d 6a63 6675
0ac92b0 5976 4e6a 346c 3070 5157 494e 5f41 5a2f
0ac92c0 336d 6342 4e6d 4a4f 306c 4277 4a72 774d
0ac92d0 4d41 006c 0000 0000 0000 0000 0000 0000
0ac92e0 0000 0000 0000 0000 0000 0000 0000 0000
*
1139380 0000 0000 0000 0000 0000
1139389

I'm able to reproduce on two separate machines running 6.5.11 and ZFS 2.2.0.

ZFS does not see any errors with the pool.

$ zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0

errors: No known data errors

$ zpool status -t
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:07:24 with 0 errors on Wed Nov  1 00:06:45 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        zroot                                         ONLINE       0     0     0
          nvme-WDS100T1X0E-XXXXXX_XXXXXXXXXXXX-part2  ONLINE       0     0     0  (100% trimmed, completed at Tue 31 Oct 2023 11:15:47 PM GMT)

errors: No known data errors

Describe how to reproduce the problem

On a system running ZFS 2.2.0, upgrade pools to enable the block cloning feature.
emerge -1 dev-lang/go, where Portage's TMPDIR is on ZFS.
After a successful install of Go, the files in /usr/lib/go/pkg/tool/linux_amd64/compile are corrupted.

I was able to reproduce with and without Portage's "native-extensions" feature. I was unable to reproduce after changing Portage's TMPDIR to another filesystem (such as tmpfs).

Include any warning/errors/backtraces from the system logs

I'm also hit by this bug, happened after upgrade to ZFS 2.2.0. Mounting tmpfs on portage TMPDIR (build directory for go) solves the issue. Currently on kernel 6.5.9, originally happened on 6.5.7.

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption. My understanding is 8.32 predates the switch to automatically using reflink?

No idea if it's worth noting, but the remaining data in the files seems to be a repetition of the same base64 encoded data. Decoding it gave me nothing comprehensible.

This is the Go BuildID. Compare the output of a non-corrupted build:

$ file /usr/lib/go/pkg/tool/linux_amd64/compile
/usr/lib/go/pkg/tool/linux_amd64/compile: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=AZG4j339IZ-OAfBcmzF6/X0YMZagYV4o9m0AWIUeg/mhcjufvYjNl4p0WQNIA_/Zm3BcmNOJl0wBrJMwAMl, not stripped

With the data remaining in the file:

$ hexdump -C /var/tmp/portage/dev-lang/go-1.21.4/image/usr/lib/go/pkg/tool/linux_amd64/compile
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000fa0  00 00 00 00 00 00 00 00  00 00 00 00 41 5a 47 34  |............AZG4|
00000fb0  6a 33 33 39 49 5a 2d 4f  41 66 42 63 6d 7a 46 36  |j339IZ-OAfBcmzF6|
00000fc0  2f 58 30 59 4d 5a 61 67  59 56 34 6f 39 6d 30 41  |/X0YMZagYV4o9m0A|
00000fd0  57 49 55 65 67 2f 6d 68  63 6a 75 66 76 59 6a 4e  |WIUeg/mhcjufvYjN|
00000fe0  6c 34 70 30 57 51 4e 49  41 5f 2f 5a 6d 33 42 63  |l4p0WQNIA_/Zm3Bc|
00000ff0  6d 4e 4f 4a 6c 30 77 42  72 4a 4d 77 41 4d 6c 00  |mNOJl0wBrJMwAMl.|
00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00ac9280  41 5a 47 34 6a 33 33 39  49 5a 2d 4f 41 66 42 63  |AZG4j339IZ-OAfBc|
00ac9290  6d 7a 46 36 2f 58 30 59  4d 5a 61 67 59 56 34 6f  |mzF6/X0YMZagYV4o|
00ac92a0  39 6d 30 41 57 49 55 65  67 2f 6d 68 63 6a 75 66  |9m0AWIUeg/mhcjuf|
00ac92b0  76 59 6a 4e 6c 34 70 30  57 51 4e 49 41 5f 2f 5a  |vYjNl4p0WQNIA_/Z|
00ac92c0  6d 33 42 63 6d 4e 4f 4a  6c 30 77 42 72 4a 4d 77  |m3BcmNOJl0wBrJMw|
00ac92d0  41 4d 6c 00 00 00 00 00  00 00 00 00 00 00 00 00  |AMl.............|
00ac92e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
01139380  00 00 00 00 00 00 00 00  00                       |.........|
01139389

@thulle

Had the same issue here... Upgraded from 2.1.13 to 2.2.0 last month right after it came out. I'm on Ubuntu and built the ZFS package from the 2.2.0 tag.

I am discovering random files that were being stored after the 2.2.0 upgrade have suffered silent corruption and either have repetitious data like @thulle reported or have large blocks of contiguous zeroes instead of the data that they should be filled with. I don't use or build go like the other posters.

zpool scrub reports no errors, zpool status reports no errors. Let me know if there is any other information needed.

After downgrading coreutils from 9.3 to 8.32, I am no longer able to reproduce this corruption.

This seems to solve the corruption issue on my end too.

#11900 (comment) onwards is highly relevant. See also #14753 and #11900 as a whole.

It would be interesting to know if people here have all upgraded their pool for the new block cloning feature or if there's a mix here.

feature@block_cloning active here

My understanding is that block cloning is enabled by default when upgrading to 2.2.0, but my not be getting used unless an application like coreutils is actually leveraging it.

Can people post the result of this:

zpool get all tank | grep bclone

(where tank is the name of their pool with the corruption)

kc3000    bcloneused                     442M                           -
kc3000    bclonesaved                    1.42G                          -
kc3000    bcloneratio                    4.30x                          -

My understanding is this: If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption.

Any non-zero number for either field and you may OR may not have silent corruption.

That is likely but not really safe to say yet, we don't have enough data.

but my not be getting used unless an application like coreutils is actually leveraging it.

Is it accurate to say it's being used when the FICLONE ioctl is used, and possibly when copy_source_range?

I wouldn't even feel comfortable saying that yet, I agree it's likely, but I'd really like more certainty before declaring "you're safe if not this".

Agreed, we don't know what we don't know, we need to:

Get a machine / test case to repro it most of the time (Ideally 100%). I think we're here already here with newer coreutils and building go.
Determine what we think is the bug and produce a patch
Patch the source, rerun the test(s), verify that this now happens 0% of the time

I don't know if there's a more robust way of figuring this out outside of that. And of course I'm handwaving 2 which could be a heavy lift.

I'd assume, not having looked, that it's something like copy_file_range isn't dirtying things so the thing that triggers a force txg sync on SEEK_DATA/SEEK_HOLE with a dirty thing isn't firing.

But as I'm somewhat occupied recovering from being badly ill at home, my time to test and debug this is rather finite, so I wouldn't assume I'll be doing that in a timely fashion.

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

Anyone can answer my question in comment: #15526 (comment) ?

@0x5c

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

Thank you, @0x5c.

Unfortunately, since Oracle bought Sun Microsystems, Solaris has been closed source since 2010, so it would be impossible to tell from the outside.

@0x5c

This should suggest that the probability of there being an issue with Solaris ZFS for this specific issue, should be relatively rather low, given that as far as I can tell, it doesn't use the same method to copy files as GNU coreutils cp does -- but again, I can be wrong.

The bug could be hit by anything that tries to find holes in a file that is also being written to, which includes but is not limited to coreutils 9.x. Anything that seeks to the next hole in a file, while it is being written to, during a very specific moment of the write operation, could end up incorrectly reading zeroes.

Thank you, @0x5c.

Unfortunately, since Oracle bought Sun Microsystems, Solaris has been closed source since 2010, so it would be impossible to tell from the outside.

If the bug is present in that version of ZFS, then it should theoretically be possible to trigger with anything that seeks to the next hole in a file, regardless of what that program may be. If coreutils 9.x can build for solaris, then that could be a way to try to reproduce the bug. Even a minimal program that just does the equivalent to coreutils cp would be enough.

ZTS: Add dirty dnode stress test

dnode_is_dirty: check dnode and its data for dirtiness

dmu_buf_will_clone: fix race in transition back to NOFILL

I found out why second copy from my comment #15526 (comment) is done by seeking hole/data:

ZFS reports number of blocks that is less than size of file thats why second copy is sparse but it shouldn't. Of course it whouldn't be good for reproducer but the real question:

Do we have another bug in zfs in reporting number of blocks ?

BEcause of that cp doesn't to copy_file_Range as it should and perform faster :)
For me it should report real number of blocks.

workstation: set zfs_dmu_offset_next_sync=0 to avoid openzfs/zfs#15526

ZTS: Add dirty dnode stress test

We're still hitting this bug in 2.2.3 and it hasn't been fully fixed yet. I found this issue here, more people are still experiencing it: #15933

Just wanted to keep people aware and to keep checking your data for silent corruption.

ZTS: Add dirty dnode stress test

dnode_is_dirty: use dn_dirty_txg to check dirtiness

Navigation Menu

Repository navigation

some copied files are corrupted (chunks replaced by zeros) #15526

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

350 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

some copied files are corrupted (chunks replaced by zeros) #15526

Description

System information

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

Activity

thulle commented on Nov 14, 2023

terinjokes commented on Nov 14, 2023

RichardBelzer commented on Nov 14, 2023

thulle commented on Nov 14, 2023

thesamesam commented on Nov 14, 2023

thulle commented on Nov 14, 2023

RichardBelzer commented on Nov 14, 2023

thulle commented on Nov 14, 2023

RichardBelzer commented on Nov 14, 2023

rincebrain commented on Nov 14, 2023

terinjokes commented on Nov 14, 2023

rincebrain commented on Nov 14, 2023

RichardBelzer commented on Nov 14, 2023

rincebrain commented on Nov 14, 2023

350 remaining items

0x5c commented on Dec 5, 2023

robszy commented on Dec 5, 2023

alpha754293 commented on Dec 5, 2023

0x5c commented on Dec 6, 2023

robszy commented on Dec 13, 2023

RichardBelzer commented on Mar 13, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions