Kernel panic while accessing snapshot directory #3948
Comments
|
I'm experiencing similar problem on Ubuntu 14.04, kernel 3.19.0-30-generic, ZFS 0.6.5.3
|
|
I do not suffer from this issue any longer after upgrade to non-hardened kernel 4.2.4 |
|
I've just installed kernel 4.1.12, let's see.. |
|
The same problem on 4.1.12 |
|
same problem here.
|
|
This happens only if there is concurrent mount access in the snapshot directory. Some clients are more prefetchy and access all the directories in parallel. If the client is opening mounting the snapshots sequentially everthing works. |
|
d287880 could fix our problem |
|
thanks, we are now building ZFS from git & test it |
|
Unfortunately, it does not help. |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
|
This should now be addressed in master and these fixes will appear in the next point release. @tuxoko provided patches for both known ways this could occur. If people impacted by this issue have time to independently verify the fix which was applied to master that would be appreciated. 24ef51f Use spa as key besides objsetid for snapentry |
|
I'm using Ubuntu 15.10 (pretty vanilla) and was using the latest stable zfsonlinux packages. The kernel would instantly fail as described above whenever there were concurrent mounts of snapshots. I installed master (built from source right now), and that problem completely vanished. So: this is independent verification by an impacted user. (By the way, I used to use zfs a lot, then switched to btrfs, and am now back to using zfs.) |
|
@williamstein thanks for posting. We're glad to have you back! |
|
Thanks -- I'm about to switch to using ZFS to serve the snapshots for 245,000 projects (=Linux accounts) for https://cloud.sagemath.com... |
|
@williamstein that's awesome. Just a quick FYI if you're going to want to access these snapshots via NFS you'll want to cherry-pick this small fix. If NFS isn't in your plans then don't worry about it. The fix should be in master this week and in the next point release. |
|
Thanks for the heads up regarding NFS and the workaround. I had tested using NFS last week and it consistently hung everything (as you know), so I gave up on that for now, and switched to using sshfs (with properly configuration to sshd with sftp chroot support to make it less insecure). I had to use keep the snapshots mounted so that they would be visible over sshfs, which is what led me to this ticket in the first place. I will likely switch to use NFS in the future. However, for my use case right now sshfs is fast enough (the local network in Google compute engine is extremely fast). |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
|
Unfortunately I'm still able to reproduce the bug. Kernel version: 3.19.0-31-generic running on Ubuntu 14.04; ZFS is from 10 Dec - a22502c |
|
I've had two machines crash with a similar log in the last day.... On Monday, December 21, 2015, Tomas Srnka notifications@github.com wrote:
Sent from my massive iPhone 6 plus. |
|
@williamstein specifically which back trace, and have you applied to the snapshot fixes? |
|
@tomassrnka |
|
@benalbrecht Here's the traceback I got in production yesterday: Maybe I'm wrong and this is a completely different problem. The mention of arc_prune makes me wonder if I have not configured the servers with sufficient RAM. Any advise? I have about 200-300GB of data, and I'm not use deduplication, though I did use dedup when originally copying the 200GB in (then I disabled it). This VM has 4GB RAM. |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue #3948 Issue #3786 Issue #3887
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
|
After a week of extensive usage on one of our backup servers, we are getting: Any ideas please? |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
|
on Linux odyssey 4.3.3 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 x86_64 GNU/Linux and zfs 0.6.5.3: |
|
The original issue here was fixed so I'm closing this out. The symbolic link issue still needs to be investigated. |
Kernel panic - not syncing: avl_find() succeeded inside avl_add()Versions:
ZFS setup:
zpool history
The text was updated successfully, but these errors were encountered: