avl_find avl_add z_unmount panic on zfs with snapshots #3243
Milestone
Comments
cat /sys/module/zfs/parameters/zfs_expire_snapshot300 after reading zfs_ctldir, I figure I can avoid this entire codepath by going hidden: It's seems like there's plenty of opportunity for this race condition to get triggered. this is the third panic since I've setup zfs, but I didn't take pictures to be sure it was the same. |
|
still panic'd. this time, there was nothing I did that should have mounted the snapshot that it would need to be unmounted. I'm setting expire to zero, let see if that helps cat /sys/module/zfs/parameters/zfs_expire_snapshot0 |
|
same problem for me on Fedora21 3.19.3-200.fc21.x86_64 posted here #3257 |
|
Hi, I'm trying this debugging patch. Which, I hope, does two things:
|
|
okay. in playing around with quick snapshot creation after applying my patch, I hit a would-have-been-panic, as well as the other issues people were reporting in linked tickets (i.e.: ?????? for directory permissions). I could not create a simple reproducer for finding avl in unmount_snapshot, but I did see that happen a few times (with my logging, which shows surprising to me things like a large series of unmount_snapshot calls firing in the same second. Ultimately, I'm giving up again, because the kernel still hung, with the last kernel message being: "BUG: unable to handle kernel paging request at 0000000400000029" no issues in the rest of zfs (when not using snapshots and leaving snapdir=hidden) I'm not sure about how automount is suppose to work, but it was mounting, even when I was cd'ing into subdirectories of the snapshot mount point. I wasn't printing out path & name, and I don't know what the shell was doing, so not sure if that is anything. the shell was printing out too many links and was getting confused about cwd. |
|
btw - I didn't find an entry in /proc/mounts for the snapshot, even though it was mounted.. } |
|
I confirm this bug with I think it's something related to standby of the disks, because I didn't notice this problem with manual import/export |
|
Anyone seeing this might like to try applying #3381 / @f5882e2 to see if it helps. |
|
@chrisrd , are there other changes in your branch? On Fedora, I'm seeing a build error when using your patch that avl_walk is a different type than expected. Unfortunately, my DKMS seems to be causing cc to be a little more strict than usual, but hey, we are talking about kernel modules here. ;-) |
|
@GalenOfTheShadows no, no other changes. |
|
Go the same panic : I've created a snapshot (zfs snapshot /tank/root@install), went into /.zfs/snapshot/install" and, several minutes after : |
|
I can also report that this is still an issue on Fedora 21 running 4.0.4-201 w/0.6.4.1. The funny part is, for me, it only triggers when more than 3 snapshot functions are scheduled. No issues when its just cycling out an hourly snapshot for a new one, but when daily/weekly/monthly hit along with the hourly, it craps out. Also, with kernel debugging turned on, a running scrub seems to hold it off until about 25-35 minutes after the scrub finishes, no matter what the status of the snapshot jobs are. |
|
The change referenced above:
...has been merged into master @7224c67. Can anyone seeing this please let us know if this change does or doesn't fix the problem? |
|
I'm also seeing this issue. I'm running a build from master @2a34db1b. If anyone needs me to extract any more information from the core dump, please let me know. kdump essentials follow: |
|
@fvanniere your stacktrace looks identical to one I posted on #3257 on 13th April. I was able to fix kernel panic by upgrading to newer kernel (3.18.14 but I also tested 4.0.4), this seems to have been fixed in torvalds/linux@8f502d5 (I also applied #3344 but it probably was not necessary). The current situation is still not perfect, there are kernel "Oopses" when trying to access non-existing snapshots (which depending on your configuration may lead to panic, for example see #3030 ). I have no other problems if I avoid accessing non-existing snapshots, |
|
Is anyone still actively investigating this? |
|
@dylanpiergies yes although I haven't had time to focus on it. |
|
Hi there, This patch is to address the avl_find avl_add z_unmount panic on zfs with snapshots. The cause of the problem is that avl_add() panics if it finds a node that already exists. This patch addresses this by removing the unnecessary calls to avl_add() and potential panic. The root of the problem is the avl_remove() is called before an "unmount" and if EBUSY tries to re-add a node. This is logically incorrect, because the EBUSY does not accurately reflect the filesystem state. The solution in the following patch is to: a) properly guard using ZFS_ENTER/ZFS_EXIT operations around the filesystem. Linux (and Solaris) kernel mutex's are not-reentrant, I have some evidence this might cause other problems with other processes accessing the filesystem during these operations. Other developers please take note. b) a simple userspace program that returns a side-effect "free" state of mount (by reading /proc/mounts) and therefore can determine the state of the filesystem before unmount. If a mount is not present it is ignored. An EBUSY is returned ONLY if a mounted was present before the attempted umount. c) the addition of ENOENT state for __zfsctl_unmount_snapshot() to reflect the snapshot was not mounted. The machine this was tested on is running linux 4.1.1, zfs/spl 0.6.4.2-1, 32 threads, 256G memory, and correctly creates/removes and mounts and unmounts snapshots. There remains a problem for someone else who knows this code better than I. If you cd $snapshot && cat /proc/mounts the mount for $snapshot is shown. Immediately after it disappears. I have instrumented the unmount code in zfs_ctrdir.c so I know it is not there. There also remains the problems of too many links, which I think is connected to the ".." structure. My simple userspace code to test a mounted snapshot, has no license so add whatever you feel is necessary for inclusion. I could of course make it a module function, but I wanted a simple drop in for "stat" and "mountpoint" programs both of which force an automount in testing for the mountpoint. Cheers P. simple program for side-effect free mount check. install as "/sbin/sidefree" |
|
@neurotensin it would be great if you could open a new pull request with your proposed fix. That will make it much easier to review and allow it to run through the automated testing. https://help.github.com/articles/creating-a-pull-request/ |
|
Hi, I'm working on a pull request now. I just wanted to get it out there as One question, where's the best place to add the "sidefree" help In the pull request I will add it as a binary to be install in /sbin/ Cheers P. On 07/10/2015 04:05 PM, Brian Behlendorf wrote:
|
|
Hi, This is the pull request. Sorry for rerouting, but I had to clean it up and make sure it applied to the master not just the release I am running. P. |
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Aug 30, 2015
Re-factor the .zfs/snapshot auto-mouting code to take in to account changes made to the upstream kernels. And to lay the groundwork for enabling access to .zfs snapshots via NFS clients. This patch makes the following core improvements. * All actively auto-mounted snapshots are now tracked in two global trees which are indexed by snapshot name and objset id respectively. This allows for fast lookups of any auto-mounted snapshot regardless without needing access to the parent dataset. * Snapshot entries are added to the tree in zfsctl_snapshot_mount(). However, they are now removed from the tree in the context of the unmount process. This eliminates the need complicated error logic in zfsctl_snapshot_unmount() to handle unmount failures. * References are now taken on the snapshot entries in the tree to ensure they always remain valid while a task is outstanding. * The MNT_SHRINKABLE flag is set on the snapshot vfsmount_t right after the auto-mount succeeds. This allows to kernel to unmount idle auto-mounted snapshots if needed removing the need for the zfsctl_unmount_snapshots() function. * Snapshots in active use will not be automatically unmounted. As long as at least one dentry is revalidated every zfs_expire_snapshot/2 seconds the auto-unmount expiration timer will be extended. * Commit torvalds/linux@bafc9b7 caused snapshots auto-mounted by ZFS to be immediately unmounted when the dentry was revalidated. This was a consequence of ZFS invaliding all snapdir dentries to ensure that negative dentries didn't mask new snapshots. This patch modifies the behavior such that only negative dentries are invalidated. This solves the issue and may result in a performance improvement. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3589 Issue openzfs#3344 Issue openzfs#3295 Issue openzfs#3257 Issue openzfs#3243 Issue openzfs#3030 Issue openzfs#2841
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Aug 31, 2015
Re-factor the .zfs/snapshot auto-mouting code to take in to account changes made to the upstream kernels. And to lay the groundwork for enabling access to .zfs snapshots via NFS clients. This patch makes the following core improvements. * All actively auto-mounted snapshots are now tracked in two global trees which are indexed by snapshot name and objset id respectively. This allows for fast lookups of any auto-mounted snapshot regardless without needing access to the parent dataset. * Snapshot entries are added to the tree in zfsctl_snapshot_mount(). However, they are now removed from the tree in the context of the unmount process. This eliminates the need complicated error logic in zfsctl_snapshot_unmount() to handle unmount failures. * References are now taken on the snapshot entries in the tree to ensure they always remain valid while a task is outstanding. * The MNT_SHRINKABLE flag is set on the snapshot vfsmount_t right after the auto-mount succeeds. This allows to kernel to unmount idle auto-mounted snapshots if needed removing the need for the zfsctl_unmount_snapshots() function. * Snapshots in active use will not be automatically unmounted. As long as at least one dentry is revalidated every zfs_expire_snapshot/2 seconds the auto-unmount expiration timer will be extended. * Commit torvalds/linux@bafc9b7 caused snapshots auto-mounted by ZFS to be immediately unmounted when the dentry was revalidated. This was a consequence of ZFS invaliding all snapdir dentries to ensure that negative dentries didn't mask new snapshots. This patch modifies the behavior such that only negative dentries are invalidated. This solves the issue and may result in a performance improvement. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3589 Closes openzfs#3344 Closes openzfs#3295 Closes openzfs#3257 Closes openzfs#3243 Closes openzfs#3030 Closes openzfs#2841
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Aug 31, 2015
Re-factor the .zfs/snapshot auto-mouting code to take in to account changes made to the upstream kernels. And to lay the groundwork for enabling access to .zfs snapshots via NFS clients. This patch makes the following core improvements. * All actively auto-mounted snapshots are now tracked in two global trees which are indexed by snapshot name and objset id respectively. This allows for fast lookups of any auto-mounted snapshot regardless without needing access to the parent dataset. * Snapshot entries are added to the tree in zfsctl_snapshot_mount(). However, they are now removed from the tree in the context of the unmount process. This eliminates the need complicated error logic in zfsctl_snapshot_unmount() to handle unmount failures. * References are now taken on the snapshot entries in the tree to ensure they always remain valid while a task is outstanding. * The MNT_SHRINKABLE flag is set on the snapshot vfsmount_t right after the auto-mount succeeds. This allows to kernel to unmount idle auto-mounted snapshots if needed removing the need for the zfsctl_unmount_snapshots() function. * Snapshots in active use will not be automatically unmounted. As long as at least one dentry is revalidated every zfs_expire_snapshot/2 seconds the auto-unmount expiration timer will be extended. * Commit torvalds/linux@bafc9b7 caused snapshots auto-mounted by ZFS to be immediately unmounted when the dentry was revalidated. This was a consequence of ZFS invaliding all snapdir dentries to ensure that negative dentries didn't mask new snapshots. This patch modifies the behavior such that only negative dentries are invalidated. This solves the issue and may result in a performance improvement. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3589 Closes openzfs#3344 Closes openzfs#3295 Closes openzfs#3257 Closes openzfs#3243 Closes openzfs#3030 Closes openzfs#2841
tomgarcia
pushed a commit
to tomgarcia/zfs
that referenced
this issue
Sep 11, 2015
Re-factor the .zfs/snapshot auto-mouting code to take in to account changes made to the upstream kernels. And to lay the groundwork for enabling access to .zfs snapshots via NFS clients. This patch makes the following core improvements. * All actively auto-mounted snapshots are now tracked in two global trees which are indexed by snapshot name and objset id respectively. This allows for fast lookups of any auto-mounted snapshot regardless without needing access to the parent dataset. * Snapshot entries are added to the tree in zfsctl_snapshot_mount(). However, they are now removed from the tree in the context of the unmount process. This eliminates the need complicated error logic in zfsctl_snapshot_unmount() to handle unmount failures. * References are now taken on the snapshot entries in the tree to ensure they always remain valid while a task is outstanding. * The MNT_SHRINKABLE flag is set on the snapshot vfsmount_t right after the auto-mount succeeds. This allows to kernel to unmount idle auto-mounted snapshots if needed removing the need for the zfsctl_unmount_snapshots() function. * Snapshots in active use will not be automatically unmounted. As long as at least one dentry is revalidated every zfs_expire_snapshot/2 seconds the auto-unmount expiration timer will be extended. * Commit torvalds/linux@bafc9b7 caused snapshots auto-mounted by ZFS to be immediately unmounted when the dentry was revalidated. This was a consequence of ZFS invaliding all snapdir dentries to ensure that negative dentries didn't mask new snapshots. This patch modifies the behavior such that only negative dentries are invalidated. This solves the issue and may result in a performance improvement. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3589 Closes openzfs#3344 Closes openzfs#3295 Closes openzfs#3257 Closes openzfs#3243 Closes openzfs#3030 Closes openzfs#2841
JKDingwall
pushed a commit
to JKDingwall/zfs
that referenced
this issue
Aug 11, 2016
Re-factor the .zfs/snapshot auto-mouting code to take in to account changes made to the upstream kernels. And to lay the groundwork for enabling access to .zfs snapshots via NFS clients. This patch makes the following core improvements. * All actively auto-mounted snapshots are now tracked in two global trees which are indexed by snapshot name and objset id respectively. This allows for fast lookups of any auto-mounted snapshot regardless without needing access to the parent dataset. * Snapshot entries are added to the tree in zfsctl_snapshot_mount(). However, they are now removed from the tree in the context of the unmount process. This eliminates the need complicated error logic in zfsctl_snapshot_unmount() to handle unmount failures. * References are now taken on the snapshot entries in the tree to ensure they always remain valid while a task is outstanding. * The MNT_SHRINKABLE flag is set on the snapshot vfsmount_t right after the auto-mount succeeds. This allows to kernel to unmount idle auto-mounted snapshots if needed removing the need for the zfsctl_unmount_snapshots() function. * Snapshots in active use will not be automatically unmounted. As long as at least one dentry is revalidated every zfs_expire_snapshot/2 seconds the auto-unmount expiration timer will be extended. * Commit torvalds/linux@bafc9b7 caused snapshots auto-mounted by ZFS to be immediately unmounted when the dentry was revalidated. This was a consequence of ZFS invaliding all snapdir dentries to ensure that negative dentries didn't mask new snapshots. This patch modifies the behavior such that only negative dentries are invalidated. This solves the issue and may result in a performance improvement. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#3589 Closes openzfs#3344 Closes openzfs#3295 Closes openzfs#3257 Closes openzfs#3243 Closes openzfs#3030 Closes openzfs#2841 Conflicts: config/kernel.m4 module/zfs/zfs_ctldir.c
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
[toppk@static ~]$ uname -a;rpm -q zfs
Linux static.bllue.org 3.19.3-200.fc21.x86_64 #1 SMP Thu Mar 26 21:39:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
zfs-0.6.3-1.3.fc21.x86_64
here is how I created the filesystem
zpool create t3 mirror /dev/mapper/hp35.cfs /dev/mapper/hp34.cfs
zfs create t3/home
zfs set compression=lz4 t3/home
zfs snapshot t3/home@$( date '+%Y%m%d_%H:%M:%S_%z' )
zfs set snapdir=visible t3/home
I was running some zfs list cmds and running "df /t3/home/.zfs/snaphots/" around the time it crashed
The text was updated successfully, but these errors were encountered: