NULL pointer dereference in zfs_sb_teardown

[prakashsurya]

I hit this BUG today when running zpool destroy $POOL.

Running a down stream release of ZFS:

<5>ZFS: Loaded module v0.6.2-260.1, ZFS pool version 5000, ZFS filesystem version 5

# rpm -qa | grep zfs
zfs-debuginfo-0.6.2-260.1.ch5.2.x86_64
zfs-0.6.2-260.1.ch5.2.x86_64
zfs-test-0.6.2-260.1.ch5.2.x86_64
kmod-zfs-devel-2.6.32-431.11.2.1chaos.ch5.2.x86_64-0.6.2-260.1.ch5.2.x86_64
kmod-zfs-2.6.32-431.11.2.1chaos.ch5.2.x86_64-0.6.2-260.1.ch5.2.x86_64
zfs-kmod-debuginfo-0.6.2-260.1.ch5.2.x86_64
zfs-devel-0.6.2-260.1.ch5.2.x86_64
kmod-zfs-0.6.2-260.1.ch5.2.x86_64
zfs-dracut-0.6.2-260.1.ch5.2.x86_64
kmod-zfs-devel-0.6.2-260.1.ch5.2.x86_64

# uname -a
Linux hostname 2.6.32-431.11.2.1chaos.ch5.2.x86_64 #1 SMP Tue Mar 25 17:59:51 PDT 2014 x86_64 x86_64 x86_64 GNU/Linux

<1>BUG: unable to handle kernel NULL pointer dereference at (null)
<1>IP: [<ffffffff81296aa6>] __list_add+0x26/0xa0
<4>PGD 0 
<4>Oops: 0000 [#1] SMP 
<4>last sysfs file: /sys/module/zfs/initstate
<4>CPU 1 
<4>Modules linked in: zfs(P)(U) zcommon(P)(U) znvpair(P)(U) zavl(P)(U) zunicode(P)(U) spl(U) zlib_deflate nfs fscache nfsd lockd nfs_acl auth_rpcgss exportfs sunrpc ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 vhost_net macvtap macvlan tun kvm iTCO_wdt iTCO_vendor_support mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core sb_edac edac_core i2c_i801 lpc_ich mfd_core ioatdma sg igb dca i2c_algo_bit i2c_core ptp pps_core shpchp ext4 jbd2 mbcache sd_mod crc_t10dif ahci wmi mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib_deflate]
<4>
<4>Pid: 198279, comm: umount Tainted: P           ---------------    2.6.32-431.11.2.1chaos.ch5.2.x86_64 #1 Intel Corporation S2600GZ/S2600GZ
<4>RIP: 0010:[<ffffffff81296aa6>]  [<ffffffff81296aa6>] __list_add+0x26/0xa0
<4>RSP: 0018:ffff880ea3aabc98  EFLAGS: 00010246
<4>RAX: 0000000000000000 RBX: ffff880ea3aabcd8 RCX: 00000000ffffffff
<4>RDX: ffff880c7b9a12e0 RSI: 0000000000000000 RDI: ffff880ea3aabcd8
<4>RBP: ffff880ea3aabcb8 R08: 0000000000000000 R09: 00000000ffffffff
<4>R10: 0000000000000000 R11: 0000000000000000 R12: ffff880c7b9a12e0
<4>R13: 0000000000000000 R14: ffff880c7b9a12e0 R15: ffffffffffffffff
<4>FS:  00007f14a64d8740(0000) GS:ffff880061820000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
<4>CR2: 0000000000000000 CR3: 0000000ea3a94000 CR4: 00000000000407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process umount (pid: 198279, threadinfo ffff880ea3aaa000, task ffff880b0c39eae0)
<4>Stack:
<4> ffffffff8152c35d ffff880c7b9a12d8 ffff880b0c39eae0 ffff880c7b9a12dc
<4><d> ffff880ea3aabd28 ffffffff8152c3af ffff880ea3aabfd8 ffff880b0c39eae0
<4><d> 0000000000000001 ffff880b0c39eae0 ffffffff8109bbb0 ffff880ea3aabcf0
<4>Call Trace:
<4> [<ffffffff8152c35d>] ? __mutex_lock_slowpath+0x7d/0x180
<4> [<ffffffff8152c3af>] __mutex_lock_slowpath+0xcf/0x180
<4> [<ffffffff8109bbb0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff8152c2be>] mutex_lock+0x3e/0x60
<4> [<ffffffffa066f9ef>] dmu_objset_evict_dbufs+0x2f/0x150 [zfs]
<4> [<ffffffffa06e1c37>] zfs_sb_teardown+0x197/0x380 [zfs]
<4> [<ffffffffa06e1e70>] zfs_umount+0x30/0xb0 [zfs]
<4> [<ffffffffa06fdf7e>] zpl_put_super+0xe/0x10 [zfs]
<4> [<ffffffff8118c38b>] generic_shutdown_super+0x5b/0xe0
<4> [<ffffffff8118c476>] kill_anon_super+0x16/0x60
<4> [<ffffffffa06fdd9e>] zpl_kill_sb+0x1e/0x30 [zfs]
<4> [<ffffffff8118cc17>] deactivate_super+0x57/0x80
<4> [<ffffffff811abacf>] mntput_no_expire+0xbf/0x110
<4> [<ffffffff811ac61b>] sys_umount+0x7b/0x3a0
<4> [<ffffffff8108ae71>] ? sigprocmask+0x71/0x110
<4> [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
<4>Code: 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 4c 8b 42 08 49 89 f5 49 89 d4 49 39 f0 75 27 <4d> 8b 45 00 4d 39 c4 75 40 49 89 5c 24 08 4c 89 23 4c 89 6b 08 
<1>RIP  [<ffffffff81296aa6>] __list_add+0x26/0xa0
<4> RSP <ffff880ea3aabc98>
<4>CR2: 0000000000000000

[behlendorf]

This can be cleanly explained by #2523. If we lose the mutex_unlock() race cleaning up the dbuf then we could NULL dereference on the list handling in __mutex_lock_slowpath as shown here. Since we believe this has now been fixed I'm closing this issue out.