Incremental zfs send does not transmit hole #4050
Comments
|
@Ringdingcoder Is not this corruption reproducible with 4.1.13? |
|
Yes, it does. Given that it happens with Illumos as well, I wouldn’t have expected anything else. It seems like this bug has been there "forever" (probably only since version 5000). |
|
@Ringdingcoder, do you have compression enabled on your dataset and hole_birth feature activated your pool ? If so, then I think I know what is going on. This is likely the same issue as described in issue #4023 where it is reproduced on zvols, but I can also reproduce it on files. The problem is as follows. When a large hole (L1 and up) is created in a ZFS file and then partially filled with data, the remaining (non-filled) portions of that hole are assigned birth epoch zero. This breaks the following assumption made in zfs send implementation: all the holes created after the hole_birth feature is activated must have non-zero birth time. Based on this assumption, zfs send does not transmit these holes, so the source of your zfs send will have zero ranges in non-filled portions, whereas the target of your zfs receive will have old data (from the previous snapshot) in these ranges. If you'd like to play with this, here is a set of commands to reproduce the issue: This creates a large sparse file with the 0.5M-1.5M range filled with random data. Let's create a hole in the middle and then partly fill it with data: Now let's zfs send/recv this to another dataset in the pool, and compare the files: Now, the files are of the same size, but have different contents: So, what's different ? Let's see if the zero ranges in the partially filled L1 hole might be an issue: So, the source has zeros but the destination still has data from snapshot s1. |
|
No, I don't have hole_birth activated and never have: http://list.zfsonlinux.org/pipermail/zfs-discuss/2015-November/023910.html. Compression is enabled. |
|
Hm, the fact that the file is sparse suggests that it was truncated then written past the end, as I have done above or that writes of zero blocks were converted to holes because of compression. But without birth_hole feature enabled, all the holes are transmitted, so there must be something else going on. You can use to see where the holes are where N is the dnode number (run P.S. You do 'sync' before backing stuff up? This is not a 'kernel page cache holding on to some data' issue? |
|
All the info is in the mailing list links. There I have posted zdb output which clearly shows the holes. The file has been written in full, from beginning to end in one go. ZFS creates the holes by itself. http://list.zfsonlinux.org/pipermail/zfs-discuss/2015-November/023909.html |
Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Chris Williamson <chris.williamson@delphix.com> In certain circumstances, "zfs send -i" (incremental send) can produce a stream which will result in incorrect sparse file contents on the target. The problem manifests as regions of the received file that should be sparse (and read a zero-filled) actually contain data from a file that was deleted (and which happened to share this file's object ID). Note: this can happen only with filesystems (not zvols, because they do not free (or reuse) object IDs). Note: This can happen only if, since the incremental source (FromSnap), a file was deleted and then another file was created, and the new file is sparse (i.e. has areas that were never written to and should be implicitly zero-filled). We suspect that this was introduced by 4370 (applies only if hole_birth feature is enabled), and made worse by 5243 (applies if hole_birth feature is disabled, and we never send any holes). The bug is caused by the hole birth feature. When an object is deleted and replaced, all the holes in the object have birth time zero. However, zfs send cannot tell that the holes are new since the file was replaced, so it doesn't send them in an incremental. As a result, you can end up with invalid data when you receive incremental send streams. As a short-term fix, we can always send holes with birth time 0 (unless it's a zvol or a dataset where we can guarantee that no objects have been reused).
|
We were seeing similar things as described in the linked email on pools running 0.6.5 working with pools that had feature@hole_birth disabled. Rolling back to 0.6.4 fixed the incremental sendfile creation for new data, but (somewhat obviously) is not able to fix historical data that was sent wrong. So basically: If for any reason you are running 0.6.5 with pools that have feature@hole_birth disabled, please check your receiving dataset to ensure that it matches your sending dataset (in our case an md5sum of the sparse files revealed different checksums) |
|
Referencing: ahrens/illumos@ca3f86b 6370 ZFS send fails to transmit some holes https://www.illumos.org/issues/6370 ZFS send fails to transmit some holes |
|
Those do seem related, but only if the description is incorrect. The above links claim "This can happen only if, since the incremental source (FromSnap), a file was deleted and then another file was created, and the new file is sparse (i.e. has areas that were never written to and should be implicitly zero-filled)." However in our case (and I believe in the case in this ticket) the sparse files were not deleted. As best we can tell, at some point the incremental send code was altered in a way that any file with holes would generate an incorrect incremental sendfile, resulting in dataset mismatches. We have some evidence that this is only on system running up to date ZFS code, but with pools that have feature@hole_birth disabled. This might also explain why both @bprotopopov and @Ringdingcoder applied the patch and did not see any improvement. |
|
The claim might well be correct. I can confidently say that some files have been deleted on my file system. It does not matter if sparse or not. The only other ingredient is a new sparse file being created that somehow shares some metadata structures (?) or inode numbers (?) with the deleted files. Unfortunately, it does not take much for a file to become sparse. A few blocks of zeroes are enough for that to happen. They specifically do not have to be created in an unusual way. |
|
How did you guys manage to create a pool with this feature disabled ? I tried this on Linux at pool create time, and it did not work (error returned). I assume this is the only way to do it because once it is enabled/activated, you cannot disable it either. I believe that there appear to be assumptions made in the code that once it is activated, you cannot disable it. Boris. From: Steven Burgess notifications@github.com We were seeing similar things as described in the linked email on pools running 0.6.5 working with pools that had feature@hole_birth disabled. Rolling back to 0.6.4 fixed the incremental sendfile creation for new data, but (somewhat obviously) is not able to fix historical data that was sent wrong. So basically: If for any reason you are running 0.6.5 with pools that have feature@hole_birth disabled, please check your receiving dataset to ensure that it matches your sending dataset (in our case an md5sum of the sparse files revealed different checksums) Reply to this email directly or view it on GitHubhttps://github.com//issues/4050#issuecomment-165136030. [https://avatars1.githubusercontent.com/u/173088?v=3&s=400]#4050 (comment) Incremental zfs send does not transmit hole · Issue #4050 ...#4050 (comment) |
|
I created the pool several years ago. I'm not even sure on which OS. Probably OpenIndiana. |
|
Us as well, we have many pools that existed prior to hole_birth that went un-upgraded, and also some ancient pools that are created Because some people still believe that they can switch over to zfs-fuse with no ill-consiquence. I know I have seen some people on the mailing list and GH issues make similar claims, so I was specifically trying to let them know, if they update the code but did not upgrade the pool, they should check the integrity of their sparse files. |
|
@stevenburgess I just downgraded to 0.6.4.2 in order to test your assertion, however, this does not make any difference for me. I still get the same corruption. And as I already mentioned earlier, OpenIndiana 151a8, which I tend to keep on an USB stick, and which is a good deal older than ZoL 0.6.4, behaves exactly the same. |
|
So, is there a simple recipe you can describe to reproduce this issue ? I would be interested to understand the mechanics of the failure. Boris. From: Stefan Ring notifications@github.com @stevenburgesshttps://github.com/stevenburgess I just downgraded to 0.6.4.2 in order to test your assertion, however, this does not make any difference for me. I still get the same corruption. And as I already mentioned earlier, OpenIndiana 151a8, which I tend to keep on an USB stick, and which is a good deal older than ZoL 0.6.4, behaves exactly the same. [https://avatars3.githubusercontent.com/u/2973190?v=3&s=400]https://github.com/stevenburgess stevenburgess (Steven Burgess) · GitHubhttps://github.com/stevenburgess Reply to this email directly or view it on GitHubhttps://github.com//issues/4050#issuecomment-165594521. |
|
No, I just have a state where I can rerun zfs send and zfs receive and check the resulting md5sum. I don't know how to manufacture that state in the first place. |
|
Stefan, if you'd like, I can work with you to get more debug info. This will involve locating a file that was updated incorrectly by send/recv and running zdb on that filesystem in the snapshots and in the resulting filesystem. Typos courtesy of my iPhone On Dec 18, 2015, at 4:08 AM, Stefan Ring <notifications@github.commailto:notifications@github.com> wrote: No, I just have a state where I can rerun zfs send and zfs receive and check the resulting md5sum. I don't know how to manufacture that state in the first place. Reply to this email directly or view it on GitHubhttps://github.com//issues/4050#issuecomment-165719000. |
|
I have already posted most of this: http://list.zfsonlinux.org/pipermail/zfs-discuss/2015-November/023909.html |
|
@stevenburgess No, you were right. I made a mistake when testing, and I can confirm that the corruption does not happen with 0.6.4.2. |
|
Good to hear! I was a little worried when I read that you were having this problem even on much older systems. |
|
@behlendorf @fling- pointed this issue out to me. Any issue where the term corruption is mentioned is something we should prioritize. We should add it to the milestone for the next release. |
|
@ryao agreed that its important, I brought it up in IRC and elsewhere because I feel like ZoL users are particularly prone to having new code with old pools (I know some people do it on purpose for zfs-fuse compatibility). |
6370 ZFS send fails to transmit some holes Reviewed by: Matthew Ahrens <mahrens@delphix.com> Reviewed by: Chris Williamson <chris.williamson@delphix.com> Reviewed by: Stefan Ring <stefanrin@gmail.com> Reviewed by: Steven Burgess <sburgess@datto.com> Reviewed by: Arne Jansen <sensille@gmx.net> Approved by: Robert Mustacchi <rm@joyent.com> References: https://www.illumos.org/issues/6370 illumos/illumos-gate@286ef71 In certain circumstances, "zfs send -i" (incremental send) can produce a stream which will result in incorrect sparse file contents on the target. The problem manifests as regions of the received file that should be sparse (and read a zero-filled) actually contain data from a file that was deleted (and which happened to share this file's object ID). Note: this can happen only with filesystems (not zvols, because they do not free (and thus can not reuse) object IDs). Note: This can happen only if, since the incremental source (FromSnap), a file was deleted and then another file was created, and the new file is sparse (i.e. has areas that were never written to and should be implicitly zero-filled). We suspect that this was introduced by 4370 (applies only if hole_birth feature is enabled), and made worse by 5243 (applies if hole_birth feature is disabled, and we never send any holes). The bug is caused by the hole birth feature. When an object is deleted and replaced, all the holes in the object have birth time zero. However, zfs send cannot tell that the holes are new since the file was replaced, so it doesn't send them in an incremental. As a result, you can end up with invalid data when you receive incremental send streams. As a short-term fix, we can always send holes with birth time 0 (unless it's a zvol or a dataset where we can guarantee that no objects have been reused). Ported-by: Steven Burgess <sburgess@datto.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes #4369 Closes #4050
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Adds a module option which disables the hole_birth optimization which has been responsible for several recent bugs, including issue openzfs#4050. Original-patch: https://gist.github.com/pcd1193182/2c0cd47211f3aee623958b4698836c48 Signed-off-by: Rich Ercolani <rincebrain@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#4833
Apparently, there are various related bug reports floating around, but AFAICT they all concern the @hole_birth feature. I have a case that seems very similar, but without having ever touched @hole_birth. The symptom is that on the receiving clone, some holes get filled with semi-random data.
Details on the mailing list:
http://list.zfsonlinux.org/pipermail/zfs-discuss/2015-November/023899.html
and
http://list.zfsonlinux.org/pipermail/zfs-discuss/2015-November/023909.html (self-reply to previous mail)
I've also tested this with the patch from openzfs/openzfs#37 applied, but this does not make the least bit of a difference.
The text was updated successfully, but these errors were encountered: