zfs diff : unable to generate diffs #1426

timemaster67 · 2013-04-25T00:03:24Z

running Arch Linux with zfs 0.6.1, I am having a little issue.

zfs diff GirlsPool/Animes@2012-10-04 GirlsPool/Animes
Cannot stat /media/GirlsPool/Animes/.zfs/shares/: unable to generate diffs

I think this is related to

cd /media/GirlsPool/Animes/.zfs
ls -l
ls: cannot access shares: Input/output error
total 0
d????????? ? ? ? ? ? shares
dr-xr-xr-x 2 root root 2 24 avr 19:41 snapshot
cd shares
ls -al
ls: cannot access .: Input/output error
total 0
?????????? ? ? ? ? ? .
dr-xr-xr-x 1 root root 0 23 avr 23:01 ..

the zfs diff command must be trying to access the shares folder, but since it's unable to, it stop working there.
I am guessing that the top patch to comment out access to the share directory would work, but I have not tried.
#481

Most of the zfs diff on snapshot works. When they works,
ls -al
total 2
dr-xr-xr-x 1 root root 0 23 avr 23:01 .
drwx------ 3 timemaster users 3 16 mar 13:36 ..
dr-xr-xr-x 2 root root 2 24 avr 19:49 shares
dr-xr-xr-x 2 root root 2 23 avr 23:07 snapshot

the share folder is accessible.

At first, I thought that this issue was because I used zfs-fuse before zfsonlinux. However, even new snapshot taken on 2013-04-22 have this issue.

Could this be related to a previous version of zfs not properly creating the .zfs folder for a new file system? My top file system is 2010-09-21, and all file system on which there is this issue are from this time.

behlendorf · 2013-04-25T23:14:28Z

@timemaster67 I suspect you're right, the issue is with the failed shares access. Unfortunately, I'm not able to recreate a problem with the shares folder? When this occurs are there any errors logged to the console, run dmesg.

timemaster67 · 2013-04-26T01:40:50Z

Unfortunately, no error messages are reported to dmesg.
Maybe when the filesystem was created with zfs-fuse, it was missing the .share folder and it's still missing. New shapshot cannot diff on old filesystem, however, new filesystem work correctly. (filesystem = new zfs create).
Let me test zfs-fuse behaviour for you, I will report back in a few days.

I created this bug for the community, I don't plan to use zfs diff often but maybe this bug report would be useful for other people.

I see some workaround for this :

Do a diff with rsync using the .zfs/snapshot/xxx. This does work as it do a basic diff
patch the zfd diff program to ignore the share folder like in #481. Do we ever need the .share folder in zfs diff ? Maybe if it return an error, then just don't use the share folder instead of breaking there ? my other file system have nothing in that folder and I can still diff. (pardon the lack of knowledge for the internal programming of zfs).

I do understand that if these filesystem were created by a previous zfs version for linux, that's not supported and you should not patch your software to workaround old bugs from different software.

So instead,
3. Is there a way to fix a broken .share folder on a particular filesystem, instead of patching the zfs diff command ?

gbooker · 2013-06-08T23:43:33Z

I'm running into the same issue. Also no error messages in dmesg. Ubuntu Precise, all up to date with stable.

My filesystem was created in FreeBSD 8. I'm pretty certain it was zpool version 13 when created, and ZFS version 4. It has since been upgraded to 28 and 5 (though old snapshots are version 4) using zfs on linux. I tried exporting the pool and re-importing it and saw no change.

If I create a new filesystem in the same pool, it doesn't not have a broken shares folder.

Any ideas on how I can resolve this?

ramonfernandez · 2013-07-10T18:07:51Z

I am seeing the same behavior (ls: cannot access shares: Input/output error). I had a pool created using Solaris10 x86 using ZFS v15. I then imported the pool into Ubuntu 12.04 running the stable ZFS PPA, then run 'zfs upgrade pool'. I'm wondering if it's related to moving from v15 to v28, or from x86 to x86-64. Will try with the daily PPA and see what happens.

Some additional information in case it helps:

root# zfs set sharenfs=on tst1/fs1
cannot share 'tst1/fs1': share(1M) failed
property may be set but unable to reshare filesystem

share(1M) is a solaris tool, not a linux one, so maybe something is hardcoded somewhere? This system is not in production yet, so I'll be happy to run more tests if anyone has an idea of what could be the issue here.

gbooker · 2013-12-02T15:17:12Z

In the hopes that it is useful for debugging purposes, I created a pool in a FreeBSD 8 virtual machine using the same mechanism I used to create my production pool. I then attached it to a new VM running Ubuntu LTS 12.04.2 and upgraded the pool and filesystem versions to the latest. I then attempted to run a zfs diff across its two snapshots. It failed with the same error as described by the OP.

Then, I cloned the repo from this project (checked out the 0.6.2 tag), and modified it as described in #481. I built the libs and the user space tool zfs, and reran the diff using the compiled binary. It successfully generated the diff.

So, here is my vdi created within VirtualBox (zipped due to radical space savings): http://www.cod3r.com/ZFSNoSharesDir.vdi.zip Hopefully this will help in diagnosing the problem.


        Workaround for 'zfs diff' shares error

For some reason which is not yet fully explained it's possible for stat64() to fail with EIO on the .zfs/shares directory. This then causes the entire 'zfs diff' command to fail even though the shares directory is not needed. Until the root cause of the stat64() failure can be determined a viable work around is to explicitly set the di->shares inode number. The Linux .zfs directory implementation will always use the same well known inode numbers for the top level directories. Obtaining the number through stat64() is cleaner, but in case that fails it can be safely assigned based on the known constant. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#1426 Issue openzfs#481


        Workaround for 'zfs diff' shares error

For some reason which is not yet fully explained it's possible for stat64() to fail with EIO on the .zfs/shares directory. This then causes the entire 'zfs diff' command to fail even though the shares directory is not needed. Until the root cause of the stat64() failure can be determined a viable work around is to explicitly set the di->shares inode number. The Linux .zfs directory implementation will always use the same well known inode numbers for the top level directories. Obtaining the number through stat64() is cleaner, but in case that fails it can be safely assigned based on the known constant. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#1426 Issue openzfs#481

behlendorf · 2013-12-03T22:26:17Z

@gbooker @b333z I'd like to get to the root cause of this issue and determine why stat64() is failing for the shares directory. But in the meantime what do you two think about applying the workaround in #1922. It's safe and should prevent people from hitting this issue until it can be resolved properly.

gbooker · 2013-12-04T04:24:59Z

@behlendorf I tried the patch in #1922 and it seems to work. I cherry-picked the commit on top of the 0.6.2 tag in my VM and it is able to do the diff where as the straight 0.6.2 tag code cannot. I'm running 0.6.2 code from the PPA and only compiling the libzfs and zfs binaries. I assume this is a sufficient test.

As per why it is failing, it seems that some implementations of ZFS create the filesystem without the shares directory entirely (such as FreeBSD 8). For example, in the vdi image I posted, if you change to the /testshare/.zfs directory and perform an ls, you will see the error:
ls: cannot access shares: Input/output error
because the shares directory was never created in this filesystem. I dug around in the code when I first commented here and discovered the shares directory created upon filesystem creation in zfsonlinux, but nothing to ensure its existence later on. I don't know if any other zfs implementations create this dir.


        Fix 'zfs diff' shares error

When creating a dataset with ZoL a zsb->z_shares_dir ZAP object will not be created because shares are unimplemented. Instead ZoL just sets zsb->z_shares_dir to zero to indicate there are no shares. However, if you import a pool which was created with a different ZFS implementation then the shares ZAP object may exist. Code was added to handle this case but it clearly wasn't sufficiently tested with other ZFS pools. There was a bug in the zpl_shares_getattr() function which passed the wrong inode to zfs_getattr_fast() for the case where are shares ZAP object does exist. This causes an EIO to be returned to stat64() which in turn causes 'zfs diff' to fail. This fix is the pass the correct inode after a sucessful zfs_zget(). Additionally, only put away the references if we were able to get one. Signed-off-by: Brian Behlendorf behlendorf1@llnl.gov Issue openzfs#1426 Issue openzfs#481


        Fix 'zfs diff' shares error

When creating a dataset with ZoL a zsb->z_shares_dir ZAP object will not be created because shares are unimplemented. Instead ZoL just sets zsb->z_shares_dir to zero to indicate there are no shares. However, if you import a pool which was created with a different ZFS implementation then the shares ZAP object may exist. Code was added to handle this case but it clearly wasn't sufficiently tested with other ZFS pools. There was a bug in the zpl_shares_getattr() function which passed the wrong inode to zfs_getattr_fast() for the case where are shares ZAP object does exist. This causes an EIO to be returned to stat64() which in turn causes 'zfs diff' to fail. This fix is the pass the correct inode after a sucessful zfs_zget(). Additionally, only put away the references if we were able to get one. Signed-off-by: Brian Behlendorf behlendorf1@llnl.gov Issue openzfs#1426 Issue openzfs#481

behlendorf · 2013-12-04T22:44:02Z

@gbooker @b333z Thank you for posting the vdi image. Once I imported the pool I was able to see exactly why EIO was being returned. I've opened #1927 with a proper fix for this issue. I've validated it using your image but if I could get a few more eyes on this fix that would be good.

timemaster67 · 2013-12-05T03:25:43Z

I just tested @behlendorf #1927 commit on top of zfs 0.6.2 on a lubuntu 13.04.
I installed zfs-fuse from the ubuntu repository and created a new pool called fusetestpool. Note : last time I used it, so long long ago, zfs-fuse does not create a mbr or gpt, it use the raw disk. anyway, you should detect the pool with an import. I created some test data and snapshot inside of it.

Then, I downloaded the zfs 0.6.2 tar.gz source and build it. Had to install the ppa:zfs-native to install libnvpair1 required by zpool.
I imported the fusetestpool and upgraded the zpool and the zfs file system. I also imported @gbooker FreeBSD pool. the 0.6.2 version could not zfs diff @gbooker pool and my fusetestpool. The error was the same as written at the beginning of this thread.
I created a new filesystem inside fusetestpool called new * native* (don't remember exactly). zfs diff worked on it.

I manually (I am not good with git) applied the patch on top of 0.6.2, rebuilded, reinstalled, reimported.
The zfs diff command would work on both the FreeBSD pool and the zfs-fuse pool. in the fusepooltest, zfs diff on both the old filesystem and the new one worked.

I had 1 issue : space character appear like \040
example :

M /fusetestpool/

/fusetestpool/fuse\040test\0402
/fusetestpool/fuse\040test\0403

this might be related to a default console char instead of zfs. still reporting it just in case.
If possible, I am not 100% confident of my testing, because it did a small test on a new vm, and might have done the install/things in a unsupported way. If someone else could do a better test than me and confirm the patch, it would be great.

You can download the virtualbox hard drive here : https://drive.google.com/file/d/0BzAEM7sxOch8b1ZndmpOR3pmNW8/edit?usp=sharing


        Fix 'zfs diff' shares error

When creating a dataset with ZoL a zsb->z_shares_dir ZAP object will not be created because shares are unimplemented. Instead ZoL just sets zsb->z_shares_dir to zero to indicate there are no shares. However, if you import a pool which was created with a different ZFS implementation then the shares ZAP object may exist. Code was added to handle this case but it clearly wasn't sufficiently tested with other ZFS pools. There was a bug in the zpl_shares_getattr() function which passed the wrong inode to zfs_getattr_fast() for the case where are shares ZAP object does exist. This causes an EIO to be returned to stat64() which in turn causes 'zfs diff' to fail. This fix is the pass the correct inode after a sucessful zfs_zget(). Additionally, only put away the references if we were able to get one. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Graham Booker <https://github.com/gbooker> Signed-off-by: timemaster67 <https://github.com/timemaster67> Closes openzfs#1426 Closes openzfs#481


        Fix 'zfs diff' shares error

When creating a dataset with ZoL a zsb->z_shares_dir ZAP object will not be created because shares are unimplemented. Instead ZoL just sets zsb->z_shares_dir to zero to indicate there are no shares. However, if you import a pool which was created with a different ZFS implementation then the shares ZAP object may exist. Code was added to handle this case but it clearly wasn't sufficiently tested with other ZFS pools. There was a bug in the zpl_shares_getattr() function which passed the wrong inode to zfs_getattr_fast() for the case where are shares ZAP object does exist. This causes an EIO to be returned to stat64() which in turn causes 'zfs diff' to fail. This fix is the pass the correct inode after a sucessful zfs_zget(). Additionally, only put away the references if we were able to get one. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Graham Booker <https://github.com/gbooker> Signed-off-by: timemaster67 <https://github.com/timemaster67> Closes openzfs#1426 Closes openzfs#481


        Fix 'zfs diff' shares error

When creating a dataset with ZoL a zsb->z_shares_dir ZAP object will not be created because shares are unimplemented. Instead ZoL just sets zsb->z_shares_dir to zero to indicate there are no shares. However, if you import a pool which was created with a different ZFS implementation then the shares ZAP object may exist. Code was added to handle this case but it clearly wasn't sufficiently tested with other ZFS pools. There was a bug in the zpl_shares_getattr() function which passed the wrong inode to zfs_getattr_fast() for the case where are shares ZAP object does exist. This causes an EIO to be returned to stat64() which in turn causes 'zfs diff' to fail. This fix is the pass the correct inode after a sucessful zfs_zget(). Additionally, only put away the references if we were able to get one. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Graham Booker <https://github.com/gbooker> Signed-off-by: timemaster67 <https://github.com/timemaster67> Closes openzfs#1426 Closes openzfs#481

behlendorf mentioned this issue Dec 3, 2013

Workaround for 'zfs diff' shares error #1922

Closed

behlendorf mentioned this issue Dec 4, 2013

Fix 'zfs diff' shares error #1927

Closed

behlendorf closed this in 90ee9ed Dec 6, 2013

openzfs / zfs

zfs diff : unable to generate diffs #1426

zfs diff : unable to generate diffs #1426

timemaster67 commented Apr 25, 2013

This comment has been minimized.

behlendorf commented Apr 25, 2013

This comment has been minimized.

timemaster67 commented Apr 26, 2013

This comment has been minimized.

gbooker commented Jun 8, 2013

This comment has been minimized.

ramonfernandez commented Jul 10, 2013

This comment has been minimized.

gbooker commented Dec 2, 2013

This comment has been minimized.

behlendorf commented Dec 3, 2013

This comment has been minimized.

gbooker commented Dec 4, 2013

This comment has been minimized.

behlendorf commented Dec 4, 2013

This comment has been minimized.

timemaster67 commented Dec 5, 2013

openzfs / zfs

Join GitHub today

zfs diff : unable to generate diffs #1426

zfs diff : unable to generate diffs #1426

Comments

timemaster67 commented Apr 25, 2013

This comment has been minimized.

behlendorf commented Apr 25, 2013

This comment has been minimized.

timemaster67 commented Apr 26, 2013

This comment has been minimized.

gbooker commented Jun 8, 2013

This comment has been minimized.

ramonfernandez commented Jul 10, 2013

This comment has been minimized.

gbooker commented Dec 2, 2013

This comment has been minimized.

behlendorf commented Dec 3, 2013

This comment has been minimized.

gbooker commented Dec 4, 2013

This comment has been minimized.

behlendorf commented Dec 4, 2013

This comment has been minimized.

timemaster67 commented Dec 5, 2013