silent corruption gives input/output error but cannot be detected with scrub, experienced on 0.7.5 and 0.8.3 versions #10697
Comments
|
Thanks for finding the other issues! Hmmm, interesting, issue #10019 shows almost the exact hardware, I'm using a Xeon E3-1226 v3 and also uses avx2 when I check with The other issues involved are with send/recv, but this is not the case. The original zpool created an input/output error, which is just propagated to the new zpools. It could not detect there is a fault when sending the zfs dataset, which is what I experience when I can just read everything with the zdb commands. I have Is there a way to delete a used zfs blocks directly with some command? Although I agree this could be very dangerous. Indeed recreate the dataset and all the snapshots would be a possibility since it only involves the snapshots of a couple of months ago. So, this is, as you suggest, the only way to get rid of this input/output error? Thanks again for any update. |
|
If it's a any help , I got the same issue on a old Intel i5-5200U using zfs 0.8.3-1ubuntu12.3 on my laptop. The only solution to fix my problem , was delete all the snapshots newer than a certain date , and zpool scrub , zpool clear a couple of times. No warning in dmesg. What frighten me most is zfs happily continues taking snapshot without any warnings. |
|
I am getting the same problem; not sure who can help. I have many files that I can't see (cat, less, ... reports: Input/output error); ls is reporting the file howver. I replaced a disk recently, not sure if I should try a resilver or if it was the reason of the problem. I checked old snapshots, and the problem is also there. cp returns: Bad Address, and dmesg reports the error (nothing for cat, ....): Pool is healthy, scrub did not report any issue. I opened a new issue as it relates also to the new open ZFS 2.0: #11443 (comment) |
|
@shuther what did you do with systemrescue to confirm the issue? I'm not clear on whether you have reproduced the same issue using systemrescue or have used it to confirm that a different version of ZFS is unaffected. |
|
I can confirm that cat reports also input/output error in systemrescue. And nothing shows up in dmesg. My gut feeling is that the data on disk is good (I am not very familiar with zdb but I was able to extract a piece of the file - if needed I could confirm it matches what I have in a backup), but the problem is connected with the metadata in a way that scrub is not reporting an error, but something else fails. |
|
At least you could detect the corruption with zfs send/recv, but for me this was silent also! I managed to copy the corrupted files from the person who still had it in his local directory. I cannot delete the corrupted files on any of the copy zpools and I just renamed them to a hidden directory. So, this is really bad for zfs it's "rock solid" reputation! |
|
If there is a pool which passes zfs scrub and gives io error on read without any dmesg traces. I wonder if it would be nice to identify the code path of such an io error with a stack. I would imagine there should not too many places in the code which produce this error and somebody could instrument them to dump a stack trace in dmesg for debugging this issue. This could have greatly speed up the investigation of this scary bug. IMHO |
|
This should be relatively easy to investigate, since you can recreate it across send/receive. Can you make a clone and remove everything except the problematic directory from it, and then send this? And make the stream available somewhere? Or if this makes the problem go away, at least you can then start tracking down what it is that makes it go away. |
|
That is possible. I will have a look, but it will take some time to make it. The dataset where the corrupted data is, is 9TB, so removing files from there in a clone will probably take some time. Next week I will post an update, if we can trap it in a clone stream. Thank you for your help! |
|
I managed to narrow the possible bug in the zfs dataset. It seems the dataset is very picky on what system it can run. On an other newer system which does work: However, I tested another machine, with the same(!) zfs version as the one that is working: I also tested an older zfs system, which also worked: It's very random where it works and where not. $ gzip -dc zfs-bad-input-output.zfs.gz|zfs recv "yourpool"/bad I did a strace on the bad system when I do an The good system says: It seems it's some getxattr that gives different results. I tried Tune in a bit deeper, good system: bad system: We do use ACL's on our systems, so we do need them, I cannot just get rid of them. I normally set: But I can confirm, the systems where I don't get input/output errors are the ones where I don't have acltype set. PS. resetting acls does not work also: |
|
Cool, it is easily reproducible with your stream (receive into a pool/dataset with default settings => works / receive into a pool/dataset with acltype=posixacl => error). With |
|
On my dataset, the issue does "not" seem to be connected to this ACL, because I am using the same settings across the pool and only one dataset is impacted by such a corruption. ACL is the same (as it looks) for the good and bad files. |
|
@shuther FreeBSD and Linux use different ACL systems. Indeed, previously we had nfsv4 ACL's on our datasets from FreeBSD, but when I transferred them to Linux, it was incompatible and I had to recreate the ACL's (posix) for the Linux system. My idea was to temporary disable acltype, then I can delete (or copy first) the error directory and put acltype back on again. Would it delete all the ACL's of the complete dataset with this operation? |
|
I can confirm the bug persists in the 2.0 branch also! Now, try to check if we lose previous ACL's when disabling posixacl: Now I try to delete the input-output error: Works! All files recovered and ACL's retained. No more input-output errors. |
|
This should be checked and send upstream. In case you missed the small stream in the lenghty report: |
|
So, from the stream, I see that zdb -vvvvvv -ddddd brokenpool/brokenfs 1525568Dataset brokenpool/brokenfs [ZPL], ID 389, cr_txg 11, 2.13M, 2074 objects, rootbp DVA[0]=<0:8048e00:200> DVA[1]=<0:18052a00:200> [L0 DMU objset] fletcher4 lz4 unencrypted LE contiguous unique double size=1000L/200P birth=614L/614P fill=2074 cksum=161558d1d1:6c21d6ed532:12cf12503a5c9:260b771f9953a4 1525568 1 128K 512 0 512 512 0.00 ZFS plain file (K=inherit) (Z=inherit=uncompressed) This with the caveat that I didn't manage to reproduce from userspace, but didn't try very hard. (apologies, but I will not be sending a pull request with a proposed fix) |
|
Looks like that /video_analysis/uav_london-input-output-error//system.posix_acl_access exists, but has zero size. Speculation by reading the code on the web browser: zfs/module/os/linux/zfs/zpl_xattr.c Lines 980 to 991 in 1c2358c The kernel seems also willing to pass such NULL pointer if userspace writes an empty value. https://github.com/torvalds/linux/blob/fcadab740480e0e0e9fa9bd272acd409884d431a/fs/posix_acl.c#L860-L896 However, zpl_get_acl will return -EIO if it reads a zero-sized xattr : zfs/module/os/linux/zfs/zpl_xattr.c Lines 1045 to 1047 in 1c2358c I couldn't manage to reproduce from userspace with purposely-wrong calls to setxattr or libacl, but tried only for a few minutes. (apologies in advance, I will not be sending a pull request with a proposed fix) |
|
So I can confirm I am facing a different issue. Using SystemRescue+ZFS, I cam read partially a file (using cat - it is a picture), and I face the input/output error in the middle of the file (so not ACL related). Still nothing reported in dmesg or zpool status. I tried also an strace using strace file xxx |
|
Thanks for pointing this out @maxximino ! The ACL is set like this and normally is inherited from the shared folder: This is a 9TB dataset, this zero ACL only happened on the uav_londen directory. It looks like this is a very rare case then? |
behlendorf
added
Component: Send/Recv
|
As usual, the catch is just one layer below where I stopped looking. zfs/module/os/linux/zfs/zpl_xattr.c Lines 482 to 488 in 1c2358c ... now I don't have anymore a clue of a around-the-ACLs code path that could mistakenly trigger this. |
|
This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions. |
System information
Describe the problem you're observing
I have an input/output error on a directory on a raidz2 zfs filesystem with lz4 compression, but there is no sign of corruption of the disks and it is not detected by scrub.
The error is propagated in the snapshots and the zfs send/recv streams as as well. The original is on a Ubuntu 18.04 with 0.7.5 zfs version which I transferred to a new Ubuntu 20.04 with zfs version 0.8.3. I will be keeping the update only, so I want to delete the bad I/O error directory on the new Ubuntu 20.04 system. The new system uses LVM partitions, which indeed could be the problem, but the original Ubuntu 18.04 has raw disks without LVM and has this fault propagated to the new Ubuntu 20.04. They both have the same behaviour.
I can still read the contents with zdb and extract the contents of the files correctly. I managed to recover the files, but I cannot delete the directory and free the space!
Both .ts streams can be read and reconstructed as well. As far as I can tell, this is the only input/output error directory on a 35TB zpool which I have detected. I doubled checked both systems with md5 checksums which pointed me to this abnormally. If I did not checked both systems I would have not known the error, so that's why I put
silentin the title and could be possible with numerous zpool systems. Both systems have ECC memory.How do I free up the space? But before I do, can I run some tests to find the cause or even better fix the abnormally in case others have this as well?
Describe how to reproduce the problem
Include any warning/errors/backtraces from the system logs
No errors in the system logs.
The text was updated successfully, but these errors were encountered: