I/O errors on zvol, but no erros reported by zpool scrub #15720
Replies: 7 comments 54 replies
-
|
I am currently destroying all the datasets/zvols affected by this issue on the backup machine. However, I'll leave at least one intact for debugging. |
Beta Was this translation helpful? Give feedback.
-
|
You should probably read #12014 before destroying everything in a fire. Assuming you're using native encryption ,of course. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I might have not reported the issue clearly - it is somewhat complex. On the backup receiver side:
On the backup sender side:
Fragment of output from backup machine for |
Beta Was this translation helpful? Give feedback.
-
|
That's quite strange. Are both sides running the same version? What are the properties on one of the example zvols where this is happening? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I agree, this is very strange. Both sides are running the same version presently. However it might have been the case that backup receiver has run a newer 2.1.X version than sender. In this case, sender has run 2.1.12 version, receiver might have run 2.1.13 or above. Props of affected zvol on the backup receiver side: Props on the sender side: |
Beta Was this translation helpful? Give feedback.
-
|
It would be useful if you could find a position in the zvol that has an error on read, and then use zdb to poke around in the zvol's metadata to find out things about that region, and try poking around. Basically do a dd, see what offset it errors off at, then use something like Then probably try something like The contents of |
Beta Was this translation helpful? Give feedback.
-
|
Thank You, I really appreciate Your help, honestly. However, since it is almost 1am where I live, I'd like to postpone this exercise for tomorrow :) Best regards and happy new year for all ZFS developers and contributors 🥇 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @rincebrain, for this zdb test/investigation I decided to use a dataset which I can share. If sharing the output of Also, somewhere in the past I've changed Output of Contents of ... with Output of Finding out the offset... I assume From part of At the end of
So, L0 DVA in this case will be |
Beta Was this translation helpful? Give feedback.
-
|
That'd be fine. I mean, setting 512 to zfs_flags won't break anything, the dbg log will just be very verbose. I would be very explicit if I was suggesting anything risky. |
Beta Was this translation helpful? Give feedback.
-
|
re zstd - thanks for info, I must've misunderstood some issues/merge requests |
Beta Was this translation helpful? Give feedback.
-
|
I don't mean risk, but rather that Anyway, I shall prepare |
Beta Was this translation helpful? Give feedback.
-
|
zstd early abort is a useful feature, but that's not to say zstd-fast won't also skip different sized incompressible segments, I believe both lz4 and zstd-fast will do similar things about skipping to the next chunk to apply compression to, but because they work differently, they will fail or succeed differently, and that's why using lz4 (cheap) as a heuristic for more expensive zstd levels is beneficial. zstd-fast-X are all fast or faster than lz4 but often worse at compression anyway, so I think switching to lz4 was probably a win for you, but the rationale wasn't. (I did not go check the zstd code to confirm that, but from what I recall, it has a similar principle about jumping forward if it gets X% into the data and jumping forward to the next 4k chunk or something, but neither of them will outright just give up entirely until they get to the end unless they overrun the output buffer size they were told, which is currently 87.5% of the input buffer.) |
Beta Was this translation helpful? Give feedback.
-
|
it's always fun to do the "right" thing for "wrong" reasons ;) |
Beta Was this translation helpful? Give feedback.
-
|
Here is the link to output of |
Beta Was this translation helpful? Give feedback.
-
|
I see the problem, but not how it happened in the first place. That record, e0000, is not a valid compressed block of any kind. The on-disk checksum is correct, but it can't be decompressed by anything. So that produces the strange result that you trip the error that, in the code, claims: And you get back EIO without tripping any error counters, since you didn't fail checksum, or on-disk read, so you don't increment those, but you got back EIO anyway, which shouldn't be possible. Looking at the naked contents of that block, it seems like the first KiB or so have some nonzero contents, but the rest is just zeroes. Overwriting the block if you overwrite the whole 128k record should work without EIO, since it doesn't have to read-modify-write, but that doesn't explain what happened to the contents originally. I am...not sure I've ever seen a bug that would cause this. The simplest way for this to happen would be if the sender screwed up and checksummed a "compressed" nonsense block, sent it along, that passed checksum because the nonsense was the thing checksummed, and then it wrote it out to disk without verifying it decompresses correctly (which it can't do in e.g. the encrypted case, since it doesn't necessarily have the key). But that doesn't answer how the sender emitted nonsense... What was the reason you needed to restore these to the production machine in the first place? What broke there? |
Beta Was this translation helpful? Give feedback.
-
|
TLDR; If this is a completely unique situation, which nobody ever came across, then please don't waste time on it - especially since it is impossible to repro. On the prod. machine, after "hard" power failure, some VMs failed to boot due to FS errors on zvols. The VM which prompted me to restore from backup machine ( Additionally, on production machine, we had some "weird" problems through December including:
No I/O errors on ZVOLs ever occurred on production machine, weirdly enough As for the possible scenario... probably sender got abruptly shut down while receiver stayed online - can this explain anything? If not, then L2ARC failure and reboots are to blame? |
Beta Was this translation helpful? Give feedback.
-
|
It shouldn't be possible to emit something like that from that kind of failure, I think, because zfs send shouldn't be taking blocks that don't pass checksum as input anyway, and if it doesn't IO error on the source, it should be passing checksum. Strange. |
Beta Was this translation helpful? Give feedback.
-
|
After setting up a new pool (
At present, I do not have a full list of affected zvols/datasets. Possible causes - guessing
|
Beta Was this translation helpful? Give feedback.
-
|
The records should all be checksummed in flight, so it shouldn't be the case that transport errors can silently do anything no matter how you're sending things. So the zvol in question has no errors being read on the source but errors now on the receiver after being received? |
Beta Was this translation helpful? Give feedback.
-
Ok, one thing less to suspect
Yes, there are 63 such zvols presently. I'm in the process of manual zfs send / recv some of them - for testing. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, I understand now. |
Beta Was this translation helpful? Give feedback.
-
|
...unless broken output of |
Beta Was this translation helpful? Give feedback.
-
|
Here's link to results of
I shall get into |
Beta Was this translation helpful? Give feedback.
-
|
The problem is that I'm trying to figure out what it looks like on the sender, and comparing that to what send/recv outputs, yes, thank you. It's not worthless, it's just that I'm trying to figure out if the problem is that something on the sender is wrong in how the snapshot is captured or how send is processing it or what, because this really shouldn't be happening, and so only seeing the send output is only half the information, unfortunately. |
Beta Was this translation helpful? Give feedback.
-
|
So, ZDB and snapshots.... Sadly, a number of zrepl-created snapshots were auto-pruned by zrepl itself. On the sender:Running On the receiver:Running |
Beta Was this translation helpful? Give feedback.
-
Thanks, I understand that this is exceptionally weird. I really appreciate Your help. |
Beta Was this translation helpful? Give feedback.
-
|
I like how you said "no weird settings" as the premise of your post, and then had that set. That still shouldn't happen, of course, but. |
Beta Was this translation helpful? Give feedback.
-
|
Reading further, the bug that @scineram linked is about breaking |
Beta Was this translation helpful? Give feedback.
-
|
Yes sorry, it was a setting that I forgot about.... Also, it didn't seem all that "weird", I've used ZFS since 0.6.x - before compressed ARC. |
Beta Was this translation helpful? Give feedback.
-
|
Hardly wasted, this should still be fixed. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Another problem caused by |
Beta Was this translation helpful? Give feedback.
-
System information
Describe the problem you're observing
I have a backup machine, to which backups are sent using
zrepltool. Recently, due to power failure (which also disabled UPS - it was "weird" power failure, forced UPS to shutdown) I was faced with a need to restore some ZFS ZVOLs from backup. I usedzfs send | zfs recvto accomplish this goal.(The backup machine was connected to the same UPS and was also shut down abruptly.)
zfs send -Lec | zfs recv ...completed without issue, however the result on the production machine was a ZVOL which exhibited I/O errors upon trying toddit (also, was impossible tofsckand reported I/O errors).zpool status -von the production machine reported errors correctly:The names under
errors:are lost, because I deleted the affected ZVOLs - earlier it was showingbelt/vms/vdi-proxy/rootwhich is a path to ZVOL being restored.Weirdly, when trying
ddon the backup machine, using the same snapshots, and evenzfs clonethem to ZVOLs, anddd-ing them to/dev/nullno errors were reported byzpool status -vDescribe how to reproduce the problem
I do not know how to reproduce these specific state, since, apparently it has taken a long time to occur without any error messages being reported. The last "integral" snapshot (without I/O errors) is from 2023-11-08.
Include any warning/errors/backtraces from the system logs
Output of
zpool events -von production machine (restore target) afterddfrom broken ZVOL:Output from
dmesgafterddaffected ZVOL:Output of dmesg during
e2fsckon affected ZVOL:e2fsckdue to write I/O errors.Changed ZFS kernel params (
modprobe.d/zfs.conf):Questions - most important
zfs destroy-ing affected ZVOLs?Possible causes
zrepl? However,zrepluses ZFS shell utils.fstrim -ain all VMs, which makes all ZVOLs receive TRIM/DISCARD/UNMAP dailyNon-causes
memtest86+zfs scrubon backup machinesync=never- not usedGoals of this report
Non-goals of this report
Questions and doubts
zpool scrubon the backup machine detect those?EDIT1:
Beta Was this translation helpful? Give feedback.
All reactions