Kernel panic on Ubuntu - zfs 0.6.5-1~trusty #3786
Comments
|
@Der-Jan thanks for reporting this. The stack appears to be caused by |
|
I had .zfs in the PRUNENAMES list for updatedb, but it was probably ignored. I had the very same idea, setting snapdir to hidden - some filesystems had it visible. For some reason - I didn't figure out - there was still a kernel panic the next morning (and yes it is panic, not warning, reboot via reset). Here's a more complete log: |
|
Same problem on my Ubuntu 14.04 desktop with ZFS 0.6.5.1-1~trusty installed. Added I can also confirm that setting |
When multiple processes simultaneously traverse in to a .zfs snapshot directory they each may trigger an auto-mount of that snapshot. Since each of these mounts may succeed that will result in multiple entries in the mounted snapshot tree triggering a panic in avl_add(). One solution to this issue is to hold the zfs_snapshot_lock over the tree lookup and subsequent mount to serialize the process. This has the advtange of being simple and easily understandable. It has the disadvantage that it means the zpl_mount() function must never take the zfs_snapshot_lock. But in practice this never happens and is easy to prevent. A technically superior approach would be to add a dummy entry to the snapshot during the mount. But this would add significant complexity because numberous places in the code would need to be updated to understand dummy entries. While not optimal this simpler solution is preferable for a point release. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3786
|
@dgtangman PRUNENAMES was not commented out, even apt-get remove didn't help. I wonder if one of the Linux Containers was causing this (didn't go through all of them). |
|
@behlendorf didn't really do the trick: |
|
@Der-Jan OK, thanks for the quick turn around. I'll work on reproducing it locally. |
|
Confirmed on Debian Wheezy and Jessie, with up-to-date kernels and spl/zfs modules (3.2.68-1+deb7u4 to 4.1.6-1~bpo8+1, zfs 0.6.5.1-4) Kernel Panic, avl_find() succeeded inside avl_add() - updatedb.mlocat Workaround: |
There exists a race where the kernel will auto-mount a snapshot in two different namespaces. This can result in the zfs_snapentry_t being added to the snapshot AVL trees twice. In order to prevent the panic check if it exists in the tree before inserting it. Longer term to correctly handle multiple mounts in different namespaces we may need to keep a list of all valid mount points for each entry. Otherwise we run the risk of them not automatically unmounting. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#3786 Issue openzfs#3887
|
@behlendorf Indeed - there's rpool and tank as pools |
|
@behlendorf Yep, multiple pools (3 to 8, 1 pool by DAS brick) on 2 servers. |
|
OK, then this makes sense. Basically the objset id tree needs to be per-spa because the objsetid's aren't unique. Thanks for the verification. |
|
@behlendorf Edit: I can reliably reproduce this with the following program |
|
Just bumping into another "set case" with same consequences: kernel:[ 213.949943] Kernel panic - not syncing: avl_find() succeeded inside avl_add() wheezy stock: 3.2.68-1+deb7u3 + zfs dkms 0.6.5.2-2-wheezy trying to upgrade the kernel as machine doesn't stand more than 2 minutes before panic again. EDIT: related to nfs exports... EDIT: my bad there WAS one last zfs export with snapsdir=visible set. Returning back to snapdir=hidden "seems" to do the job, for now (48h running, in production conditions, without any hang). Sorry for the inconvenience. |
|
@tuxoko thanks for digging in to this. The fix you proposed in #4018 looks right to me and I think it'll address the common case people are seeing. There's still a much less likely second case which can occur but it requires the system to have multiple pools mounted and that datasets with identical objset ids be auto-mounted concurrently. We can tackle that one in a follow up patch. |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
|
This should now be addressed in master and these fixes will appear in the next point release. @tuxoko provided patches for both known ways this could occur. If people impacted by this issue have time to independently verify the fix which was applied to master that would be appreciated. 24ef51f Use spa as key besides objsetid for snapentry |
|
@behlendorf I've applied the patches and installed the modules. Since I made sure mlocate won't access any zfs snapshots, I cannot really test it with mlocate. However I used @tuxoko's parallel mount without kernel panic. So seems to do the trick for me. |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue #3948 Issue #3786 Issue #3887
|
This has been resolved in master and the release branch. |
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
objsetid is not unique across pool, so using it solely as key would cause panic when automounting two snapshot on different pools with the same objsetid. We fix this by adding spa pointer as additional key. Signed-off-by: Chunwei Chen <david.chen@osnexus.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Issue openzfs#3948 Issue openzfs#3786 Issue openzfs#3887
After updating to zfs 0.6.5 on my Ubuntu server, I get a kernel panic each morning. I think it might be related to mlocate. I've disabled it for the moment - but I haven't had any issues before.
[84255.374379] CPU: 0 PID: 15716 Comm: updatedb.mlocat Tainted: P OX 3.13.0-63-generic #103-Ubuntu
[84255.401492] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 1.0a 03/06/2012
[84255.431562] 0000000000000009 ffff88107fc03d90 ffffffff81723cc0 0000000000000000
[84255.458151] ffff88107fc03dc8 ffffffff8106785d 0000000000000001 ffff88107fc13180
[84255.485714] 00000001014041ee 0000000000000000 ffff88107fc33180 ffff88107fc03dd8
[84255.514080] Call Trace:
[84255.527143] [] dump_stack+0x45/0x56
[84255.543818] [] warn_slowpath_common+0x7d/0xa0
[84255.560648] [] warn_slowpath_null+0x1a/0x20
[84255.577231] [] native_smp_send_reschedule+0x5d/0x60
[84255.594698] [] trigger_load_balance+0x16a/0x1e0
[84255.611747] [] scheduler_tick+0xa4/0xf0
[84255.628268] [] update_process_times+0x60/0x70
[84255.644967] [] tick_sched_handle.isra.17+0x25/0x60
[84255.661908] [] tick_sched_timer+0x41/0x60
[84255.678002] [] __run_hrtimer+0x77/0x1d0
[84255.693571] [] ? tick_sched_handle.isra.17+0x60/0x60
[84255.710216] [] hrtimer_interrupt+0xef/0x230
[84255.725929] [] local_apic_timer_interrupt+0x37/0x60
[84255.742135] [] smp_apic_timer_interrupt+0x3f/0x60
[84255.757983] [] apic_timer_interrupt+0x6d/0x80
[84255.773743] [] ? panic+0x193/0x1d7
[84255.788633] [] avl_add+0x4a/0x50 [zavl]
[84255.803095] [] zfsctl_snapshot_add+0x36/0x40 [zfs]
[84255.818878] [] zfsctl_snapshot_mount+0x3ad/0x440 [zfs]
[84255.834321] [] zpl_snapdir_automount+0x10/0x30 [zfs]
[84255.849493] [] follow_managed+0x13a/0x300
[84255.863640] [] ? path_lookupat+0x73/0x790
[84255.877231] [] lookup_fast+0x18b/0x2c0
[84255.891286] [] path_lookupat+0x155/0x790
[84255.904915] [] ? putname+0x29/0x40
[84255.918328] [] ? getname_flags+0x4f/0x190
[84255.931196] [] filename_lookup+0x2b/0xc0
[84255.944405] [] user_path_at_empty+0x54/0x90
[84255.956902] [] ? from_kgid_munged+0x12/0x20
[84255.969678] [] ? read_tsc+0x9/0x20
[84255.981437] [] ? __getnstimeofday+0x3a/0xc0
[84255.994077] [] user_path_at+0x11/0x20
[84256.005926] [] SyS_chdir+0x2f/0xc0
[84256.017377] [] sysenter_dispatch+0x7/0x21
The text was updated successfully, but these errors were encountered: