Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ZFS hangs when hotplugging disk #2185

Closed
arko-pl opened this issue Mar 12, 2014 · 2 comments
Closed

ZFS hangs when hotplugging disk #2185

arko-pl opened this issue Mar 12, 2014 · 2 comments
Labels
Component: ZVOL ZFS Volumes

Comments

@arko-pl
Copy link

arko-pl commented Mar 12, 2014

Hello,
I'm using ZoL 0.6.2 with Linux kernel 3.10.30. I've seen similar issues with SCST patched kernel but there are no SCST patches.
I created three zpools on three different drives with one zvol on each pool then I shut down the system and unplugged all three disks. Ten I booted up again system. Zpool status returned all three pools and they were marked as unavailable. I plugged all three disks one by one and executed "zpool status" command. Zpool status hanged and I couldn't kill this process. I displayed blocked tasks with SysRq + W:

[ 348.498695] SysRq : Show Blocked State
[ 348.498705] task PC stack pid father
[ 348.498786] zpool D 0000000000000000 0 27672 26976 0x00000000
[ 348.498794] ffff880073476e80 0000000000000002 ffffffff816c50a0 ffff88007e387500
[ 348.498801] ffff88007b22ffd8 ffff88007b22ffd8 ffff88007b22ffd8 ffff880073476e80
[ 348.498807] ffff88007b4cc148 ffff880073476ec8 ffff88007b4cc5fa ffffffff8107365f
[ 348.498813] Call Trace:
[ 348.498828] [] ? check_preempt_wakeup+0x12f/0x270
[ 348.498837] [] ? _raw_spin_unlock_irqrestore+0x8/0x10
[ 348.498844] [] ? try_to_wake_up+0xcb/0x290
[ 348.498851] [] ? set_user_nice+0xd5/0x190
[ 348.498871] [] ? taskq_create+0x2e5/0x4d0 [spl]
[ 348.498880] [] ? abort_exclusive_wait+0xb0/0xb0
[ 348.498935] [] ? spa_activate+0x2f3/0x400 [zfs]
[ 348.498986] [] ? spa_open_common+0x137/0x380 [zfs]
[ 348.499041] [] ? pool_status_check+0x41/0xa0 [zfs]
[ 348.499094] [] ? zfsdev_ioctl+0x1a5/0x1b0 [zfs]
[ 348.499101] [] ? do_vfs_ioctl+0x8a/0x4f0
[ 348.499108] [] ? get_vtime_delta+0x16/0x80
[ 348.499114] [] ? vtime_account_user+0x50/0x70
[ 348.499120] [] ? SyS_ioctl+0xa0/0xc0
[ 348.499127] [] ? tracesys+0xdd/0xe2
[ 348.499154] txg_sync D 0000000000000001 0 27891 2 0x00000000
[ 348.499161] ffff8800730ef500 0000000000000002 ffff880000000000 ffff8800730ee180
[ 348.499166] ffff88007177dfd8 ffff88007177dfd8 ffff88007177dfd8 ffff8800730ef500
[ 348.499172] ffff88007e412c40 0000000000000000 0000000000000000 ffffffff8106c95c
[ 348.499177] Call Trace:
[ 348.499185] [] ? check_preempt_curr+0x7c/0x90
[ 348.499192] [] ? ttwu_do_wakeup+0x11/0x90
[ 348.499198] [] ? try_to_wake_up+0xcb/0x290
[ 348.499204] [] ? __wake_up_common+0x4f/0x80
[ 348.499211] [] ? io_schedule+0x56/0x80
[ 348.499228] [] ? cv_wait_common+0xa5/0x1c0 [spl]
[ 348.499241] [] ? taskq_dispatch_ent+0x6d/0x1c0 [spl]
[ 348.499249] [] ? abort_exclusive_wait+0xb0/0xb0
[ 348.499306] [] ? zio_wait+0xeb/0x1a0 [zfs]
[ 348.499352] [] ? dsl_pool_sync+0xe6/0x530 [zfs]
[ 348.499401] [] ? spa_sync+0x3f9/0xac0 [zfs]
[ 348.499416] [] ? __gethrtime+0xc/0x30 [spl]
[ 348.499423] [] ? ktime_get_ts+0x3d/0xd0
[ 348.499473] [] ? txg_sync_thread+0x2eb/0x550 [zfs]
[ 348.499481] [] ? sched_clock+0x5/0x10
[ 348.499532] [] ? txg_thread_wait.isra.2+0x30/0x30 [zfs]
[ 348.499545] [] ? thread_generic_wrapper+0x75/0x90 [spl]
[ 348.499559] [] ? __thread_create+0x310/0x310 [spl]
[ 348.499565] [] ? kthread+0xb3/0xc0
[ 348.499571] [] ? alloc_pid+0x1e0/0x490
[ 348.499578] [] ? kthread_freezable_should_stop+0x60/0x60
[ 348.499584] [] ? ret_from_fork+0x7c/0xb0
[ 348.499591] [] ? kthread_freezable_should_stop+0x60/0x60

and after 40 seconds I got RCU stall:

[ 386.240004] INFO: rcu_sched self-detected stall on CPU { 0} (t=6000 jiffies g=3771 c=3770 q=0)
[ 386.240005] sending NMI to all CPUs:
[ 386.240005] NMI backtrace for cpu 0
[ 386.240005] CPU: 0 PID: 27908 Comm: vol_id Tainted: P O 3.10.30-oe64-00000-g8b3a21b #50
[ 386.240005] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 386.240005] task: ffff880071f15b00 ti: ffff880075c6e000 task.ti: ffff880075c6e000
[ 386.240005] RIP: 0010:[] [] kasprintf+0x50/0x50
[ 386.240005] RSP: 0000:ffff88007e403e40 EFLAGS: 00000096
[ 386.240005] RAX: 0000000000000c00 RBX: 0000000000002710 RCX: 0000000000000006
[ 386.240005] RDX: 0000000000000007 RSI: 0000000000000080 RDI: ffffffff81a19b60
[ 386.240005] RBP: ffffffff8198dd40 R08: 20676e69646e6573 R09: 61206f7420494d4e
[ 386.240005] R10: 00000000000003a5 R11: 3a73555043206c6c R12: ffffffff8198dd40
[ 386.240005] R13: 0000000000000000 R14: ffffffff81a1a040 R15: ffff880075c6e000
[ 386.240005] FS: 0000000000000000(0000) GS:ffff88007e400000(0063) knlGS:00000000f75e66c0
[ 386.240005] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 386.240005] CR2: 00000000f76a5230 CR3: 0000000075c39000 CR4: 00000000000007f0
[ 386.240005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 386.240005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 386.240005] Stack:
[ 386.240005] ffffffff8102d548 ffff88007e40e640 ffffffff810b1dbb ffff88007e412c40
[ 386.240005] 0000000000000000 ffffffff810121af ffffffff81012215 ffffffff81070b85
[ 386.240005] ffff880071f15b00 0000000000000000 0000000000000000 ffff88007e40e198
[ 386.240005] Call Trace:
[ 386.240005]
[ 386.240005] [] ? arch_trigger_all_cpu_backtrace+0x78/0x90
[ 386.240005] [] ? rcu_check_callbacks+0x1eb/0x5b0
[ 386.240005] [] ? native_sched_clock+0xf/0x70
[ 386.240005] [] ? sched_clock+0x5/0x10
[ 386.240005] [] ? sched_clock_local+0x15/0x80
[ 386.240005] [] ? update_process_times+0x3f/0x80
[ 386.240005] [] ? tick_sched_timer+0x66/0x90
[ 386.240005] [] ? __run_hrtimer.isra.31+0x4e/0xe0
[ 386.240005] [] ? hrtimer_interrupt+0xfc/0x250
[ 386.240005] [] ? smp_apic_timer_interrupt+0x63/0xa0
[ 386.240005] [] ? apic_timer_interrupt+0x6d/0x80
[ 386.240005]
[ 386.240005] [] ? try_module_get+0x3/0x20
[ 386.240005] [] ? get_disk+0x2e/0x80
[ 386.240005] [] ? exact_lock+0xc/0x20
[ 386.240005] [] ? kobj_lookup+0xd9/0x160
[ 386.240005] [] ? disk_map_sector_rcu+0x70/0x70
[ 386.240005] [] ? get_gendisk+0x34/0x120
[ 386.240005] [] ? __blkdev_get+0x11a/0x410
[ 386.240005] [] ? blkdev_get_block+0x20/0x20
[ 386.240005] [] ? blkdev_get+0x4b/0x2f0
[ 386.240005] [] ? unlock_new_inode+0x3a/0x60
[ 386.240005] [] ? bdget+0x112/0x130
[ 386.240005] [] ? blkdev_get+0x2f0/0x2f0
[ 386.240005] [] ? do_dentry_open+0x235/0x2b0
[ 386.240005] [] ? finish_open+0x28/0x40
[ 386.240005] [] ? do_last+0x7ca/0xe30
[ 386.240005] [] ? __inode_permission+0x29/0xa0
[ 386.240005] [] ? link_path_walk+0x245/0x920
[ 386.240005] [] ? flush_tlb_page+0x42/0xb0
[ 386.240005] [] ? path_openat+0xc6/0x500
[ 386.240005] [] ? handle_pte_fault+0xaa/0x930
[ 386.240005] [] ? do_filp_open+0x45/0xb0
[ 386.240005] [] ? __alloc_fd+0x3d/0x110
[ 386.240005] [] ? do_sys_open+0xfe/0x1f0
[ 386.240005] [] ? compat_SyS_open+0x6c/0x110
[ 386.240005] [] ? get_vtime_delta+0x16/0x80
[ 386.240005] [] ? vtime_account_user+0x50/0x70
[ 386.240005] [] ? syscall_trace_enter+0x1a/0x1f0
[ 386.240005] [] ? ia32_do_call+0x13/0x13
[ 386.240005] Code: 24 20 4c 89 4c 24 48 c7 44 24 08 10 00 00 00 48 89 44 24 18 e8 22 ff ff ff 48 83 c4 58 c3 90 90 90 90 90 90 90 90 90 90 90 90 90 <8d> 4e 3f 85 f6 0f 49 ce c1 f9 06 85 c9 7e 65 31 c0 48 83 3f 00
[ 386.240553] NMI backtrace for cpu 2
[ 386.240566] CPU: 2 PID: 0 Comm: swapper/2 Tainted: P O 3.10.30-oe64-00000-g8b3a21b #50
[ 386.240571] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 386.240578] task: ffff88007e387500 ti: ffff88007e3c8000 task.ti: ffff88007e3c8000
[ 386.240583] RIP: 0010:[] [] intel_idle+0xc7/0x140
[ 386.240597] RSP: 0000:ffff88007e3c9e48 EFLAGS: 00000046
[ 386.240602] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[ 386.240607] RDX: 0000000000000000 RSI: ffff88007e3c9fd8 RDI: 0000000000000002
[ 386.240611] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000582f10
[ 386.240616] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[ 386.240620] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88007e3c9fd8
[ 386.240626] FS: 0000000000000000(0000) GS:ffff88007e500000(0000) knlGS:0000000000000000
[ 386.240631] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 386.240636] CR2: 00000000f7762000 CR3: 000000007aedf000 CR4: 00000000000007e0
[ 386.240641] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 386.240645] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 386.240649] Stack:
[ 386.240652] 0000000000000000 000000020a61a3ff ffffffff8157ae3e ffff88007e518800
[ 386.240660] 00000059ed731d9a ffffffff819a36b0 ffffffff819a3640 ffffffff8157ae2c
[ 386.240667] 0000000000000000 000000000a61a3ff 0000000000000000 ffff88007e518800
[ 386.240674] Call Trace:
[ 386.240686] [] ? cpuidle_enter_state+0x5e/0xf0
[ 386.240693] [] ? cpuidle_enter_state+0x4c/0xf0
[ 386.240701] [] ? cpuidle_idle_call+0xa1/0x150
[ 386.240710] [] ? arch_cpu_idle+0x9/0x30
[ 386.240718] [] ? cpu_startup_entry+0x83/0x170
[ 386.240723] Code: 48 8b 34 25 f0 c5 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <85> 1d d3 b7 63 00 75 0f 48 8d 74 24 0c bf 05 00 00 00 e8 52 a0
[ 386.240795] NMI backtrace for cpu 3
[ 386.240803] CPU: 3 PID: 0 Comm: swapper/3 Tainted: P O 3.10.30-oe64-00000-g8b3a21b #50
[ 386.240808] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 386.240814] task: ffff88007e3d0000 ti: ffff88007e3ca000 task.ti: ffff88007e3ca000
[ 386.240819] RIP: 0010:[] [] interruptible_sleep_on+0x20/0x20
[ 386.240829] RSP: 0000:ffff88007e3cbf00 EFLAGS: 00000296
[ 386.240834] RAX: ffff88007e3d0000 RBX: ffff88007e3cbfd8 RCX: 0000000000000020
[ 386.240838] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000082
[ 386.240843] RBP: ffffffff81a1a030 R08: 0000000000000000 R09: 0000000000000000
[ 386.240847] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88007e3cbfd8
[ 386.240852] R13: ffff88007e3cbfd8 R14: ffff88007e3cbfd8 R15: ffff88007e3cbfd8
[ 386.240857] FS: 0000000000000000(0000) GS:ffff88007e580000(0000) knlGS:0000000000000000
[ 386.240862] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 386.240867] CR2: ffffffffff600400 CR3: 000000007a3fd000 CR4: 00000000000007e0
[ 386.240871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 386.240876] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 386.240880] Stack:
[ 386.240883] ffffffff816b17c9 ffff88007e3cbfd8 ffffffff8107c44c 0000000000000001
[ 386.240890] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 386.240896] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 386.240903] Call Trace:
[ 386.240912] [] ? schedule_preempt_disabled+0x9/0x10
[ 386.240920] [] ? cpu_startup_entry+0x12c/0x170
[ 386.240927] Code: 00 00 00 e9 f3 fe ff ff 0f 1f 00 48 ba ff ff ff ff ff ff ff 7f be 01 00 00 00 e9 dc fe ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 <41> 57 65 48 8b 04 25 f0 c5 00 00 41 56 65 48 8b 14 25 00 c6 00
[ 386.240011] NMI backtrace for cpu 1
[ 386.240011] CPU: 1 PID: 0 Comm: swapper/1 Tainted: P O 3.10.30-oe64-00000-g8b3a21b #50
[ 386.240011] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 386.240011] task: ffff88007e386e80 ti: ffff88007e3be000 task.ti: ffff88007e3be000
[ 386.240011] RIP: 0010:[] [] intel_idle+0xc7/0x140
[ 386.240011] RSP: 0018:ffff88007e3bfe48 EFLAGS: 00000046
[ 386.240011] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[ 386.240011] RDX: 0000000000000000 RSI: ffff88007e3bffd8 RDI: 0000000000000001
[ 386.240011] RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000066d51
[ 386.240011] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[ 386.240011] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88007e3bffd8
[ 386.240011] FS: 0000000000000000(0000) GS:ffff88007e480000(0000) knlGS:0000000000000000
[ 386.240011] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 386.240011] CR2: 00000000f77a2000 CR3: 0000000072108000 CR4: 00000000000007e0
[ 386.240011] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 386.240011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 386.240011] Stack:
[ 386.240011] 0000000000000000 0000000101ad1892 ffffffff8157ae3e ffff88007e498800
[ 386.240011] 00000059ed1aabee ffffffff819a36b0 ffffffff819a3640 ffffffff8157ae2c
[ 386.240011] 0000000000000000 0000000001ad1892 0000000000000000 ffff88007e498800
[ 386.240011] Call Trace:
[ 386.240011] [] ? cpuidle_enter_state+0x5e/0xf0
[ 386.240011] [] ? cpuidle_enter_state+0x4c/0xf0
[ 386.240011] [] ? cpuidle_idle_call+0xa1/0x150
[ 386.240011] [] ? arch_cpu_idle+0x9/0x30
[ 386.240011] [] ? cpu_startup_entry+0x83/0x170
[ 386.240011] Code: 48 8b 34 25 f0 c5 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <85> 1d d3 b7 63 00 75 0f 48 8d 74 24 0c bf 05 00 00 00 e8 52 a0

This is 100% reproducible with 3 drives.
@arko-pl
Copy link
Author

arko-pl commented Mar 13, 2014

I got more logs from kernel with debug:

After SysRq + W

[ 400.980936] zd0: unknown partition table
[ 414.294065] SysRq : Show Blocked State
[ 414.294081] task PC stack pid father
[ 414.294246] zpool D ffff880072319c90 0 25569 24497 0x00000000
[ 414.294262] ffff88007a30d6f8 0000000000000002 ffff88007a30dfd8 ffff88007a30dfd8
[ 414.294278] ffff88007a30dfd8 00000000001d3880 ffff88007b7f0000 ffff880077708000
[ 414.294294] 0000000000000000 ffff880072319ca8 ffff88007a30d678 ffffffff810a2049
[ 414.294310] Call Trace:
[ 414.294331] [] ? __lock_acquired+0x269/0x2b0
[ 414.294345] [] ? prepare_to_wait_exclusive+0x25/0x90
[ 414.294357] [] ? prepare_to_wait_exclusive+0x25/0x90
[ 414.294371] [] schedule+0x24/0x70
[ 414.294382] [] io_schedule+0x5a/0x80
[ 414.294411] [] cv_wait_common+0xb8/0x1f0 [spl]
[ 414.294489] [] ? zio_wait+0xeb/0x1f0 [zfs]
[ 414.294501] [] ? __init_waitqueue_head+0x60/0x60
[ 414.294524] [] __cv_wait_io+0x13/0x20 [spl]
[ 414.294592] [] zio_wait+0x11b/0x1f0 [zfs]
[ 414.294645] [] dbuf_read+0x155/0x5b0 [zfs]
[ 414.294699] [] dmu_buf_hold+0x105/0x200 [zfs]
[ 414.294767] [] zap_lockdir+0x4f/0x2f0 [zfs]
[ 414.294834] [] zap_lookup_norm+0x45/0x1c0 [zfs]
[ 414.294902] [] zap_lookup+0x2e/0x30 [zfs]
[ 414.294962] [] dsl_prop_get_dd+0xf4/0x210 [zfs]
[ 414.295024] [] dsl_prop_get_ds+0x164/0x230 [zfs]
[ 414.295082] [] dsl_dataset_get_ref+0x8fb/0xd40 [zfs]
[ 414.295142] [] dsl_dataset_hold_obj+0x1b/0x40 [zfs]
[ 414.295201] [] dsl_pool_open+0x205/0x400 [zfs]
[ 414.295222] [] ? nvlist_add_common+0x2a/0x40 [znvpair]
[ 414.295286] [] spa_load+0x916/0x12c0 [zfs]
[ 414.295300] [] ? lockdep_init_map+0x3e/0x90
[ 414.295312] [] ? debug_mutex_init+0x38/0x50
[ 414.295375] [] spa_load_best+0x49/0x250 [zfs]
[ 414.295440] [] spa_open_common+0x167/0x3b0 [zfs]
[ 414.295457] [] ? dataset_namecheck.part.1+0x8d/0x190 [zcommon]
[ 414.295524] [] ? zfs_secpolicy_none+0x10/0x10 [zfs]
[ 414.295587] [] spa_open+0xe/0x10 [zfs]
[ 414.295655] [] pool_status_check+0x40/0xa0 [zfs]
[ 414.295722] [] zfsdev_ioctl+0x1ad/0x1c0 [zfs]
[ 414.295736] [] do_vfs_ioctl+0x81/0x330
[ 414.295748] [] SyS_ioctl+0x91/0xb0
[ 414.295761] [] tracesys+0xdd/0xe2
[ 414.295876] Sched Debug Version: v0.10, 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 414.295884] ktime : 414295.876393
[ 414.295891] sched_clk : 414183.990906
[ 414.295897] cpu_clk : 414295.873290
[ 414.295904] jiffies : 4294978726
[ 414.295910] sched_clock_stable : 0
[ 414.295915]
[ 414.295920] sysctl_sched
[ 414.295927] .sysctl_sched_latency : 18.000000
[ 414.295934] .sysctl_sched_min_granularity : 2.250000
[ 414.295940] .sysctl_sched_wakeup_granularity : 3.000000
[ 414.295946] .sysctl_sched_child_runs_first : 0
[ 414.295952] .sysctl_sched_features : 11899
[ 414.295959] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 414.295965]
[ 414.295972] cpu#0, 1795.503 MHz
[ 414.295978] .nr_running : 0
[ 414.295984] .load : 0
[ 414.295990] .nr_switches : 93098
[ 414.295996] .nr_load_updates : 41427
[ 414.296002] .nr_uninterruptible : -2308
[ 414.296009] .next_balance : 4294.978727
[ 414.296015] .curr->pid : 0
[ 414.296022] .clock : 414293.654292
[ 414.296028] .cpu_load[0] : 0
[ 414.296034] .cpu_load[1] : 0
[ 414.296040] .cpu_load[2] : 0
[ 414.296046] .cpu_load[3] : 0
[ 414.296051] .cpu_load[4] : 0
[ 414.296057] .yld_count : 0
[ 414.296064] .sched_count : 93658
[ 414.296070] .sched_goidle : 32061
[ 414.296076] .avg_idle : 1000000
[ 414.296082] .ttwu_count : 45248
[ 414.296088] .ttwu_local : 23001
[ 414.296097]
[ 414.296097] cfs_rq[0]:/
[ 414.296105] .exec_clock : 68306.944965
[ 414.296113] .MIN_vruntime : 0.000001
[ 414.296120] .min_vruntime : 73344.976298
[ 414.296127] .max_vruntime : 0.000001
[ 414.296133] .spread : 0.000000
[ 414.296140] .spread0 : 0.000000
[ 414.296146] .nr_spread_over : 6480
[ 414.296152] .nr_running : 0
[ 414.296158] .load : 0
[ 414.296163] .runnable_load_avg : 0
[ 414.296169] .blocked_load_avg : 0
[ 414.296176] .tg_load_avg : 1722
[ 414.296182] .tg_load_contrib : 0
[ 414.296188] .tg_runnable_contrib : 3
[ 414.296194] .tg->runnable_avg : 509
[ 414.296200] .avg->runnable_avg_sum : 176
[ 414.296206] .avg->runnable_avg_period : 47585
[ 414.296213]
[ 414.296213] rt_rq[0]:
[ 414.296220] .rt_nr_running : 0
[ 414.296226] .rt_throttled : 0
[ 414.296233] .rt_time : 0.000000
[ 414.296239] .rt_runtime : 950.000000
[ 414.296247]
[ 414.296247] runnable tasks:
[ 414.296247] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 414.296247] ----------------------------------------------------------------------------------------------------------
[ 414.296414]
[ 414.296421] cpu#1, 1795.503 MHz
[ 414.296427] .nr_running : 1
[ 414.296433] .load : 1024
[ 414.296439] .nr_switches : 87543
[ 414.296445] .nr_load_updates : 14660
[ 414.296451] .nr_uninterruptible : 2799
[ 414.296458] .next_balance : 4294.978732
[ 414.296464] .curr->pid : 25830
[ 414.296471] .clock : 414290.030155
[ 414.296477] .cpu_load[0] : 1024
[ 414.296483] .cpu_load[1] : 512
[ 414.296489] .cpu_load[2] : 256
[ 414.296495] .cpu_load[3] : 128
[ 414.296501] .cpu_load[4] : 68
[ 414.296507] .yld_count : 0
[ 414.296513] .sched_count : 88141
[ 414.296519] .sched_goidle : 29546
[ 414.296525] .avg_idle : 1000000
[ 414.296531] .ttwu_count : 48632
[ 414.296538] .ttwu_local : 24252
[ 414.296545]
[ 414.296545] cfs_rq[1]:/
[ 414.296553] .exec_clock : 57065.112653
[ 414.296561] .MIN_vruntime : 0.000001
[ 414.296568] .min_vruntime : 95585.107419
[ 414.296574] .max_vruntime : 0.000001
[ 414.296581] .spread : 0.000000
[ 414.296587] .spread0 : 22240.131121
[ 414.296593] .nr_spread_over : 5601
[ 414.296599] .nr_running : 1
[ 414.296605] .load : 1024
[ 414.296611] .runnable_load_avg : 7
[ 414.296617] .blocked_load_avg : 0
[ 414.296623] .tg_load_avg : 1722
[ 414.296629] .tg_load_contrib : 7
[ 414.296635] .tg_runnable_contrib : 8
[ 414.296641] .tg->runnable_avg : 509
[ 414.296647] .avg->runnable_avg_sum : 374
[ 414.296654] .avg->runnable_avg_period : 46639
[ 414.296660]
[ 414.296660] rt_rq[1]:
[ 414.296667] .rt_nr_running : 0
[ 414.296673] .rt_throttled : 0
[ 414.296679] .rt_time : 0.000000
[ 414.296686] .rt_runtime : 950.000000
[ 414.296692]
[ 414.296692] runnable tasks:
[ 414.296692] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 414.296692] ----------------------------------------------------------------------------------------------------------
[ 414.296856] R bash 25830 95579.778772 44 120 95579.778772 36.185442 10107.571103 /
[ 414.296877]
[ 414.296884] cpu#2, 1795.503 MHz
[ 414.296890] .nr_running : 7
[ 414.296896] .load : 2655
[ 414.296902] .nr_switches : 108569
[ 414.296908] .nr_load_updates : 21837
[ 414.296914] .nr_uninterruptible : -1940
[ 414.296921] .next_balance : 4294.978752
[ 414.296927] .curr->pid : 25828
[ 414.296933] .clock : 401620.062158
[ 414.296939] .cpu_load[0] : 2655
[ 414.296945] .cpu_load[1] : 2655
[ 414.296951] .cpu_load[2] : 2655
[ 414.296958] .cpu_load[3] : 2655
[ 414.296964] .cpu_load[4] : 2655
[ 414.296970] .yld_count : 0
[ 414.296976] .sched_count : 109048
[ 414.296982] .sched_goidle : 37000
[ 414.296988] .avg_idle : 1000000
[ 414.296994] .ttwu_count : 63624
[ 414.297000] .ttwu_local : 32492
[ 414.297007]
[ 414.297007] cfs_rq[2]:/
[ 414.297015] .exec_clock : 73933.772619
[ 414.297023] .MIN_vruntime : 68187.652803
[ 414.297030] .min_vruntime : 68196.652803
[ 414.297037] .max_vruntime : 68193.922563
[ 414.297043] .spread : 6.269760
[ 414.297050] .spread0 : -5148.323495
[ 414.297056] .nr_spread_over : 6003
[ 414.297062] .nr_running : 5
[ 414.297068] .load : 2655
[ 414.297074] .runnable_load_avg : 1585
[ 414.297080] .blocked_load_avg : 118
[ 414.297086] .tg_load_avg : 1725
[ 414.297092] .tg_load_contrib : 1703
[ 414.297099] .tg_runnable_contrib : 466
[ 414.297105] .tg->runnable_avg : 508
[ 414.297111] .avg->runnable_avg_sum : 21511
[ 414.297117] .avg->runnable_avg_period : 47188
[ 414.297123]
[ 414.297123] rt_rq[2]:
[ 414.297130] .rt_nr_running : 1
[ 414.297136] .rt_throttled : 0
[ 414.297143] .rt_time : 0.000000
[ 414.297149] .rt_runtime : 950.000000
[ 414.297156]
[ 414.297156] runnable tasks:
[ 414.297156] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 414.297156] ----------------------------------------------------------------------------------------------------------
[ 414.297175] watchdog/2 24 -6.000000 104 0 -6.000000 5.820351 18.575568 /
[ 414.297195] migration/2 25 0.000000 1535 0 0.000000 248.098756 0.030965 /
[ 414.297227] kworker/2:2 1904 68187.652803 3058 120 68187.652803 828.833630 396669.333278 /
[ 414.297320] z_rd_int/2 25665 68193.922563 38 139 68193.922563 5.798316 984.291495 /
[ 414.297366] z_rd_iss/2 25750 68187.652803 2 139 68187.652803 0.034941 91.233620 /
[ 414.297387] z_rd_int/2 25758 68187.652803 2 139 68187.652803 0.032207 90.345448 /
[ 414.297430] R zvol_id 25828 68211.014295 1 118 68211.014295 27.015218 0.030434 /
[ 414.297450]
[ 414.297456] cpu#3, 1795.503 MHz
[ 414.297462] .nr_running : 1
[ 414.297468] .load : 1024
[ 414.297474] .nr_switches : 109152
[ 414.297480] .nr_load_updates : 20634
[ 414.297487] .nr_uninterruptible : 1450
[ 414.297493] .next_balance : 4294.978727
[ 414.297499] .curr->pid : 2038
[ 414.297506] .clock : 414297.492186
[ 414.297512] .cpu_load[0] : 0
[ 414.297518] .cpu_load[1] : 128
[ 414.297524] .cpu_load[2] : 144
[ 414.297530] .cpu_load[3] : 105
[ 414.297536] .cpu_load[4] : 70
[ 414.297542] .yld_count : 0
[ 414.297548] .sched_count : 109393
[ 414.297554] .sched_goidle : 39086
[ 414.297561] .avg_idle : 716377
[ 414.297566] .ttwu_count : 62035
[ 414.297572] .ttwu_local : 31720
[ 414.297579]
[ 414.297579] cfs_rq[3]:/
[ 414.297588] .exec_clock : 62930.719521
[ 414.297595] .MIN_vruntime : 0.000001
[ 414.297602] .min_vruntime : 110675.134204
[ 414.297608] .max_vruntime : 0.000001
[ 414.297615] .spread : 0.000000
[ 414.297621] .spread0 : 37330.157906
[ 414.297627] .nr_spread_over : 7198
[ 414.297633] .nr_running : 0
[ 414.297639] .load : 0
[ 414.297645] .runnable_load_avg : 0
[ 414.297651] .blocked_load_avg : 17
[ 414.297657] .tg_load_avg : 1727
[ 414.297663] .tg_load_contrib : 17
[ 414.297669] .tg_runnable_contrib : 34
[ 414.297675] .tg->runnable_avg : 511
[ 414.297681] .avg->runnable_avg_sum : 1607
[ 414.297688] .avg->runnable_avg_period : 47252
[ 414.297694]
[ 414.297694] rt_rq[3]:
[ 414.297701] .rt_nr_running : 0
[ 414.297707] .rt_throttled : 0
[ 414.297713] .rt_time : 0.000000
[ 414.297719] .rt_runtime : 950.000000
[ 414.297726]
[ 414.297726] runnable tasks:
[ 414.297726] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 414.297726] ----------------------------------------------------------------------------------------------------------
[ 414.297892]
[ 416.630007] INFO: rcu_sched self-detected stall on CPU
[ 416.630007] 2: (1500 ticks this GP) idle=125/140000000000001/0 softirq=6800/6800 last_accelerate: 0000/2d90, nonlazy_posted: 0, L.
[ 416.630007](t=1501 jiffies g=4967 c=4966 q=0)
[ 416.630007] sending NMI to all CPUs:
[ 416.630007] NMI backtrace for cpu 2
[ 416.630007] CPU: 2 PID: 25828 Comm: zvol_id Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 416.630007] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 416.630007] task: ffff8800720e6220 ti: ffff880071ff0000 task.ti: ffff880071ff0000
[ 416.630007] RIP: 0010:[] [] __bitmap_empty+0x15/0x80
[ 416.630007] RSP: 0018:ffff88007dc03dd8 EFLAGS: 00000046
[ 416.630007] RAX: 0000000000000000 RBX: 0000000000002710 RCX: 0000000000000002
[ 416.630007] RDX: ffff8800720e6220 RSI: 0000000000000080 RDI: ffffffff81b49920
[ 416.630007] RBP: ffff88007dc03dd8 R08: 0000000000000002 R09: 0000000000000000
[ 416.630007] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[ 416.630007] R13: ffffffff81b49e00 R14: ffffffff8109a940 R15: ffff88007dc0e3d8
[ 416.630007] FS: 00007f9c90cd06d0(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 416.630007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 416.630007] CR2: 00007f9c9082d1e0 CR3: 0000000078523000 CR4: 00000000000007e0
[ 416.630007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 416.630007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 416.630007] Stack:
[ 416.630007] ffff88007dc03df8 ffffffff810301b8 0000000000000000 ffffffff81a9b340
[ 416.630007] ffff88007dc03e28 ffffffff810d9807 0000000000000000 ffff88007ddce9e0
[ 416.630007] ffffffff81a9b340 ffffffff81a9b340 ffff88007dc03e58 ffffffff810d9b22
[ 416.630007] Call Trace:
[ 416.630007]
[ 416.630007] [] arch_trigger_all_cpu_backtrace+0x88/0xa0
[ 416.630007] [] print_cpu_stall+0xd7/0x140
[ 416.630007] [] __rcu_pending+0x72/0x1c0
[ 416.630007] [] rcu_check_callbacks+0x10d/0x120
[ 416.630007] [] update_process_times+0x43/0x80
[ 416.630007] [] tick_sched_timer+0x51/0xb0
[ 416.630007] [] __run_hrtimer.isra.29+0x61/0x100
[ 416.630007] [] hrtimer_interrupt+0x103/0x250
[ 416.630007] [] smp_apic_timer_interrupt+0x64/0xa0
[ 416.630007] [] apic_timer_interrupt+0x72/0x80
[ 416.630007]
[ 416.630007] [] ? debug_mutex_free_waiter+0x8/0x90
[ 416.630007] [] mutex_lock_nested+0x2b7/0x3a0
[ 416.630007] [] ? zvol_open+0x60/0x130 [zfs]
[ 416.630007] [] ? zvol_open+0x60/0x130 [zfs]
[ 416.630007] [] ? disk_get_part+0x18/0x250
[ 416.630007] [] zvol_open+0x60/0x130 [zfs]
[ 416.630007] [] __blkdev_get+0xb1/0x420
[ 416.630007] [] ? bd_acquire+0x1f/0x100
[ 416.630007] [] blkdev_get+0x59/0x1e0
[ 416.630007] [] ? _raw_spin_unlock+0x26/0x30
[ 416.630007] [] ? blkdev_get+0x1e0/0x1e0
[ 416.630007] [] blkdev_open+0x67/0x90
[ 416.630007] [] do_dentry_open+0x22e/0x2b0
[ 416.630007] [] finish_open+0x46/0x60
[ 416.630007] [] do_last+0x4fe/0x8d0
[ 416.630007] [] ? link_path_walk+0x235/0x950
[ 416.630007] [] path_openat+0xb6/0x510
[ 416.630007] [] ? __alloc_fd+0x29/0x220
[ 416.630007] [] do_filp_open+0x3d/0xa0
[ 416.630007] [] ? __alloc_fd+0x163/0x220
[ 416.630007] [] do_sys_open+0xf9/0x1e0
[ 416.630007] [] SyS_open+0x5a/0xe0
[ 416.630007] [] ? vtime_account_user+0x68/0x80
[ 416.630007] [] ? trace_hardirqs_on+0xd/0x10
[ 416.630007] [] ? syscall_trace_enter+0x1c/0x1f0
[ 416.630007] [] tracesys+0xdd/0xe2
[ 416.630007] Code: b8 10 00 00 00 48 89 45 c8 e8 38 ff ff ff c9 c3 90 90 90 90 90 90 8d 4e 3f 85 f6 55 0f 49 ce 48 89 e5 c1 f9 06 85 c9 7e 61 31 c0 <48> 83 3f 00 75 52 48 89 f8 31 d2 eb 10 66 0f 1f 44 00 00 48 83
[ 416.630647] NMI backtrace for cpu 3
[ 416.630653] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 416.630656] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 416.630659] task: ffff88007b7f2740 ti: ffff88007b7ee000 task.ti: ffff88007b7ee000
[ 416.630668] RIP: 0010:[] [] intel_idle+0xc7/0x130
[ 416.630671] RSP: 0000:ffff88007b7efdf8 EFLAGS: 00000046
[ 416.630673] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[ 416.630676] RDX: 0000000000000000 RSI: ffff88007b7effd8 RDI: 0000000000000003
[ 416.630679] RBP: ffff88007b7efe28 R08: 0000000000000c69 R09: 000000000012e6f1
[ 416.630682] R10: 0000000000001bc3 R11: 0000000000000000 R12: 0000000000000001
[ 416.630684] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[ 416.630688] FS: 0000000000000000(0000) GS:ffff88007de00000(0000) knlGS:0000000000000000
[ 416.630691] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 416.630694] CR2: 00000000f7747000 CR3: 0000000077901000 CR4: 00000000000007e0
[ 416.630697] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 416.630700] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 416.630701] Stack:
[ 416.630710] ffff88007b7efe28 0000000381092d2f ffff88007dfd9900 0000006100f1e6f9
[ 416.630718] ffffffff81abbb30 ffffffff81abbac0 ffff88007b7efe88 ffffffff816044ea
[ 416.630726] 0000000000000000 000000000026e46a 0000000000000000 000000000026e46a
[ 416.630728] Call Trace:
[ 416.630737] [] cpuidle_enter_state+0x4a/0xe0
[ 416.630745] [] cpuidle_idle_call+0xa1/0x150
[ 416.630752] [] ? rcu_eqs_enter+0x60/0xa0
[ 416.630759] [] arch_cpu_idle+0x9/0x30
[ 416.630766] [] cpu_idle_loop+0x8a/0x190
[ 416.630773] [] cpu_startup_entry+0x1e/0x20
[ 416.630781] [] start_secondary+0x85/0x87
[ 416.630856] Code: 48 8b 34 25 70 c7 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 13 30 6e 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 33 f6 cb
[ 416.630008] NMI backtrace for cpu 1
[ 416.630008] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 416.630008] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 416.630008] task: ffff88007b7f0000 ti: ffff88007b7ea000 task.ti: ffff88007b7ea000
[ 416.630008] RIP: 0010:[] [] intel_idle+0xc7/0x130
[ 416.630008] RSP: 0000:ffff88007b7ebdf8 EFLAGS: 00000046
[ 416.630008] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[ 416.630008] RDX: 0000000000000000 RSI: ffff88007b7ebfd8 RDI: 0000000000000001
[ 416.630008] RBP: ffff88007b7ebe28 R08: 00000000000025fd R09: 000000000179aa21
[ 416.630008] R10: 0000000000001c57 R11: 0000000000000000 R12: 0000000000000001
[ 416.630008] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[ 416.630008] FS: 0000000000000000(0000) GS:ffff88007da00000(0000) knlGS:0000000000000000
[ 416.630008] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 416.630008] CR2: ffffffffff600400 CR3: 0000000072526000 CR4: 00000000000007e0
[ 416.630008] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 416.630008] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 416.630008] Stack:
[ 416.630008] ffff88007b7ebe28 0000000181092d2f ffff88007dbd9900 0000006100ea20c4
[ 416.630008] ffffffff81abbb30 ffffffff81abbac0 ffff88007b7ebe88 ffffffff816044ea
[ 416.630008] 0000000000000000 000000000e798d07 0000000000000000 000000000e798d07
[ 416.630008] Call Trace:
[ 416.630008] [] cpuidle_enter_state+0x4a/0xe0
[ 416.630008] [] cpuidle_idle_call+0xa1/0x150
[ 416.630008] [] ? rcu_eqs_enter+0x60/0xa0
[ 416.630008] [] arch_cpu_idle+0x9/0x30
[ 416.630008] [] cpu_idle_loop+0x8a/0x190
[ 416.630008] [] cpu_startup_entry+0x1e/0x20
[ 416.630008] [] start_secondary+0x85/0x87
[ 416.630008] Code: 48 8b 34 25 70 c7 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 13 30 6e 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 33 f6 cb
[ 416.631098] NMI backtrace for cpu 0
[ 416.631105] CPU: 0 PID: 25840 Comm: dmesg Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 416.631107] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 416.631110] task: ffff8800721d3ae0 ti: ffff880072572000 task.ti: ffff880072572000
[ 416.631118] RIP: 0010:[] [] cfb_imageblit+0x2d2/0x4e0
[ 416.631120] RSP: 0000:ffff8800725738e8 EFLAGS: 00000246
[ 416.631123] RAX: ffffc900045f5cd8 RBX: ffffc900045f5cdc RCX: 0000000000000002
[ 416.631125] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
[ 416.631127] RBP: ffff880072573968 R08: 00000000ad55ad55 R09: ffff8800777e8fb5
[ 416.631129] R10: 0000000000000002 R11: ffffffff817dda00 R12: ffff8800777e8fa3
[ 416.631131] R13: ffffc900045f5da0 R14: 0000000000000003 R15: ffffc900045f5bb0
[ 416.631134] FS: 0000000000000000(0000) GS:ffff88007d800000(0063) knlGS:00000000f7699aa0
[ 416.631137] CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b
[ 416.631139] CR2: ffffffffff600400 CR3: 0000000072526000 CR4: 00000000000007f0
[ 416.631141] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 416.631143] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 416.631144] Stack:
[ 416.631151] 0000000000000046 0000000000000000 0000000000000000 ffff88007dbd3898
[ 416.631158] 000000f872573988 0000000000000046 0000000000000640 00000000000001f0
[ 416.631164] 000000000000001f ffffc900045f0a70 ffff8800721d3ae0 ffff88006e9c9000
[ 416.631165] Call Trace:
[ 416.631172] [] bit_putcs+0x305/0x5a0
[ 416.631179] [] ? __lock_acquired+0x269/0x2b0
[ 416.631185] [] ? arch_vtime_task_switch+0x51/0x90
[ 416.631190] [] ? soft_cursor+0x1ae/0x230
[ 416.631196] [] ? bit_cursor+0x5bb/0x680
[ 416.631202] [] ? get_color.isra.22+0x3e/0x160
[ 416.631207] [] fbcon_putcs+0x12f/0x170
[ 416.631213] [] ? bit_cursor+0x680/0x680
[ 416.631218] [] fbcon_redraw.isra.31+0x154/0x1c0
[ 416.631224] [] fbcon_scroll+0x1f5/0xdf0
[ 416.631231] [] scrup+0x10b/0x120
[ 416.631236] [] lf+0x70/0x80
[ 416.631242] [] do_con_trol+0x95/0xe40
[ 416.631248] [] do_con_write.part.24+0x1c6/0x990
[ 416.631254] [] ? process_output+0x32/0x70
[ 416.631260] [] con_write+0x32/0x50
[ 416.631265] [] do_output_char+0x191/0x210
[ 416.631270] [] process_output+0x48/0x70
[ 416.631275] [] n_tty_write+0x134/0x2f0
[ 416.631280] [] ? try_to_wake_up+0x160/0x160
[ 416.631285] [] tty_write+0x156/0x2b0
[ 416.631290] [] ? n_tty_ioctl+0xf0/0xf0
[ 416.631296] [] vfs_write+0xc8/0x1d0
[ 416.631301] [] SyS_write+0x50/0xa0
[ 416.631308] [] ia32_do_call+0x13/0x13
[ 416.631363] Code: 89 e1 b9 08 00 00 00 4d 01 fd eb 2c 0f 1f 44 00 00 41 0f be 11 44 29 d1 48 8d 58 04 d3 fa 21 fa 41 8b 14 93 44 21 c2 31 f2 89 10 <85> c9 75 06 49 83 c1 01 b1 08 48 89 d8 4c 39 e8 75 d4 4c 03 7d

and soft lockup bugs:
[ 532.200007] BUG: soft lockup - CPU#2 stuck for 22s! [zvol_id:25828]
[ 532.200007] Modules linked in: rpcsec_gss_krb5 zfs(O) zcommon(O) znvpair(O) zavl(O) zunicode(O) spl(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bonding button processor ftdi_sio usbserial nls_iso8859_1 nls_cp437 virtio_scsi virtio_pci virtio_ring virtio vfat fat aufs
[ 532.200007] irq event stamp: 0
[ 532.200007] hardirqs last enabled at (0): < (null)>
[ 532.200007] hardirqs last disabled at (0): [] copy_process.part.41+0x3db/0xf30
[ 532.200007] softirqs last enabled at (0): [] copy_process.part.41+0x3db/0xf30
[ 532.200007] softirqs last disabled at (0): < (null)>
[ 532.200007] CPU: 2 PID: 25828 Comm: zvol_id Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 532.200007] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 532.200007] task: ffff8800720e6220 ti: ffff880071ff0000 task.ti: ffff880071ff0000
[ 532.200007] RIP: 0010:[] [] mutex_lock_nested+0x2a9/0x3a0
[ 532.200007] RSP: 0018:ffff880071ff19d8 EFLAGS: 00000246
[ 532.200007] RAX: ffff8800720e6220 RBX: ffffffff810a2049 RCX: 0000000000000000
[ 532.200007] RDX: ffffffff81ab5c10 RSI: ffff880071ff1a08 RDI: 0000000000000246
[ 532.200007] RBP: ffff880071ff1a58 R08: 0000000000000000 R09: 0000000000000000
[ 532.200007] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ab5c38
[ 532.200007] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000246
[ 532.200007] FS: 00007f9c90cd06d0(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 532.200007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 532.200007] CR2: 00007f9c9082d1e0 CR3: 0000000078523000 CR4: 00000000000007e0
[ 532.200007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 532.200007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 532.200007] Stack:
[ 532.200007] ffffffff8147900f 0000000000000046 ffffffff81ab5c10 ffffffff8147900f
[ 532.200007] ffffffff81ab5c38 ffffffff810a0f0d ffff880071ff1a08 ffff880071ff1a08
[ 532.200007] 0000000000000000 ffff880071ff1a08 ffffffffa02d4b50 000000000e600000
[ 532.200007] Call Trace:
[ 532.200007] [] ? kobj_lookup+0x4f/0x170
[ 532.200007] [] ? kobj_lookup+0x4f/0x170
[ 532.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 532.200007] [] kobj_lookup+0x4f/0x170
[ 532.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 532.200007] [] get_gendisk+0x21/0x50
[ 532.200007] [] __blkdev_get+0x120/0x420
[ 532.200007] [] ? bd_acquire+0x1f/0x100
[ 532.200007] [] blkdev_get+0x59/0x1e0
[ 532.200007] [] ? _raw_spin_unlock+0x26/0x30
[ 532.200007] [] ? blkdev_get+0x1e0/0x1e0
[ 532.200007] [] blkdev_open+0x67/0x90
[ 532.200007] [] do_dentry_open+0x22e/0x2b0
[ 532.200007] [] finish_open+0x46/0x60
[ 532.200007] [] do_last+0x4fe/0x8d0
[ 532.200007] [] ? link_path_walk+0x235/0x950
[ 532.200007] [] path_openat+0xb6/0x510
[ 532.200007] [] ? __alloc_fd+0x29/0x220
[ 532.200007] [] do_filp_open+0x3d/0xa0
[ 532.200007] [] ? __alloc_fd+0x163/0x220
[ 532.200007] [] do_sys_open+0xf9/0x1e0
[ 532.200007] [] SyS_open+0x5a/0xe0
[ 532.200007] [] ? vtime_account_user+0x68/0x80
[ 532.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 532.200007] [] ? syscall_trace_enter+0x1c/0x1f0
[ 532.200007] [] tracesys+0xdd/0xe2
[ 532.200007] Code: 48 8b 55 90 48 3b 53 50 48 89 43 60 75 06 c7 03 00 00 00 00 80 43 08 01 41 f7 c6 00 02 00 00 74 27 e8 0c 5a 92 ff 4c 89 f7 57 9d <0f> 1f 44 00 00 48 8d 7d b0 e8 39 0c 92 ff 48 83 c4 58 5b 41 5c
[ 551.750005] INFO: rcu_sched self-detected stall on CPU
[ 551.750005] 2: (15012 ticks this GP) idle=125/140000000000001/0 softirq=6800/6800 last_accelerate: 0000/6258, nonlazy_posted: 0, L.
[ 551.750005](t=15013 jiffies g=4967 c=4966 q=0)
[ 551.750005] sending NMI to all CPUs:
[ 551.750005] NMI backtrace for cpu 2
[ 551.750005] CPU: 2 PID: 25828 Comm: zvol_id Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 551.750005] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 551.750005] task: ffff8800720e6220 ti: ffff880071ff0000 task.ti: ffff880071ff0000
[ 551.750005] RIP: 0010:[] [] flat_send_IPI_mask+0xac/0xc0
[ 551.750005] RSP: 0018:ffff88007dc03dc0 EFLAGS: 00000046
[ 551.750005] RAX: 0000000000000c00 RBX: 0000000000002710 RCX: 0000000000000006
[ 551.750005] RDX: ffff8800720e6220 RSI: 0000000000000002 RDI: ffffffff810333d7
[ 551.750005] RBP: ffff88007dc03dc8 R08: 0000000000000002 R09: 0000000000000000
[ 551.750005] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[ 551.750005] R13: ffffffff81b49e00 R14: 000000000f000000 R15: ffff88007dc0e3d8
[ 551.750005] FS: 00007f9c90cd06d0(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 551.750005] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 551.750005] CR2: 00007f9c9082d1e0 CR3: 0000000078523000 CR4: 00000000000007e0
[ 551.750005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 551.750005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 551.750005] Stack:
[ 551.750005] ffffffff8109a940 ffff88007dc03dd8 ffffffff8103366d ffff88007dc03df8
[ 551.750005] ffffffff81030190 0000000000000000 ffffffff81a9b340 ffff88007dc03e28
[ 551.750005] ffffffff810d9807 0000000000000000 ffff88007ddce9e0 ffffffff81a9b340
[ 551.750005] Call Trace:
[ 551.750005]
[ 551.750005] [] ? tick_nohz_handler+0xd0/0xd0
[ 551.750005] [] flat_send_IPI_all+0x5d/0x60
[ 551.750005] [] arch_trigger_all_cpu_backtrace+0x60/0xa0
[ 551.750005] [] print_cpu_stall+0xd7/0x140
[ 551.750005] [] __rcu_pending+0x72/0x1c0
[ 551.750005] [] rcu_check_callbacks+0x10d/0x120
[ 551.750005] [] update_process_times+0x43/0x80
[ 551.750005] [] tick_sched_timer+0x51/0xb0
[ 551.750005] [] __run_hrtimer.isra.29+0x61/0x100
[ 551.750005] [] hrtimer_interrupt+0x103/0x250
[ 551.750005] [] smp_apic_timer_interrupt+0x64/0xa0
[ 551.750005] [] apic_timer_interrupt+0x72/0x80
[ 551.750005]
[ 551.750005] [] ? mutex_unlock+0x9/0x10
[ 551.750005] [] ? mutex_unlock+0x4/0x10
[ 551.750005] [] ? put_device+0x12/0x20
[ 551.750005] [] __blkdev_get+0xfd/0x420
[ 551.750005] [] ? bd_acquire+0x1f/0x100
[ 551.750005] [] blkdev_get+0x59/0x1e0
[ 551.750005] [] ? _raw_spin_unlock+0x26/0x30
[ 551.750005] [] ? blkdev_get+0x1e0/0x1e0
[ 551.750005] [] blkdev_open+0x67/0x90
[ 551.750005] [] do_dentry_open+0x22e/0x2b0
[ 551.750005] [] finish_open+0x46/0x60
[ 551.750005] [] do_last+0x4fe/0x8d0
[ 551.750005] [] ? link_path_walk+0x235/0x950
[ 551.750005] [] path_openat+0xb6/0x510
[ 551.750005] [] ? __alloc_fd+0x29/0x220
[ 551.750005] [] do_filp_open+0x3d/0xa0
[ 551.750005] [] ? __alloc_fd+0x163/0x220
[ 551.750005] [] do_sys_open+0xf9/0x1e0
[ 551.750005] [] SyS_open+0x5a/0xe0
[ 551.750005] [] ? vtime_account_user+0x68/0x80
[ 551.750005] [] ? trace_hardirqs_on+0xd/0x10
[ 551.750005] [] ? syscall_trace_enter+0x1c/0x1f0
[ 551.750005] [] tracesys+0xdd/0xe2
[ 551.750005] Code: 00 4c 89 ef 57 9d 0f 1f 44 00 00 5b 41 5c 41 5d 41 5e 5d c3 0f 1f 00 4c 89 ef 57 9d 0f 1f 44 00 00 e8 19 9a 06 00 5b 41 5c 41 5d <41> 5e 5d c3 ff 90 50 01 00 00 eb 96 0f 1f 84 00 00 00 00 00 55
[ 551.750795] NMI backtrace for cpu 3
[ 551.750805] CPU: 3 PID: 13 Comm: rcu_sched Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 551.750811] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 551.750818] task: ffff88007b7d0000 ti: ffff88007b7ce000 task.ti: ffff88007b7ce000
[ 551.750824] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x30/0x70
[ 551.750837] RSP: 0000:ffff88007b7cfcb8 EFLAGS: 00000096
[ 551.750843] RAX: 0000000000000003 RBX: 0000000000000282 RCX: 0000000100006259
[ 551.750850] RDX: ffffffff8177a153 RSI: ffff88007ac50018 RDI: ffff88007ac50000
[ 551.750856] RBP: ffff88007b7cfcc8 R08: 0000000000000000 R09: 0000000000000001
[ 551.750863] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88007ac50000
[ 551.750870] R13: 0000000100006258 R14: ffff88007ac50000 R15: ffff88007ac50000
[ 551.750877] FS: 0000000000000000(0000) GS:ffff88007de00000(0000) knlGS:0000000000000000
[ 551.750884] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 551.750890] CR2: 00000000f7747000 CR3: 0000000077901000 CR4: 00000000000007e0
[ 551.750897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 551.750903] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 551.750908] Stack:
[ 551.750913] ffff88007b7cfce0 000000010000625b ffff88007b7cfd98 ffffffff8177a153
[ 551.750929] ffffffff8106ed75 ffff88007ac50618 ffff88007ac50618 000000010000625b
[ 551.750944] ffff88007ac50000 ffffffff810562d0 ffff88007b7d0000 ffffffffffffffff
[ 551.750958] Call Trace:
[ 551.750970] [] schedule_timeout+0x133/0x1f0
[ 551.750982] [] ? prepare_to_wait+0x25/0x90
[ 551.750994] [] ? call_timer_fn+0x100/0x100
[ 551.751007] [] rcu_gp_kthread+0x1be/0x2c0
[ 551.751019] [] ? __init_waitqueue_head+0x60/0x60
[ 551.751031] [] ? rcu_gp_fqs+0x80/0x80
[ 551.751042] [] kthread+0xd6/0xe0
[ 551.751053] [] ? tick_nohz_task_switch+0x57/0xb0
[ 551.751067] [] ? __init_kthread_worker+0x70/0x70
[ 551.751078] [] ret_from_fork+0x7c/0xb0
[ 551.751090] [] ? __init_kthread_worker+0x70/0x70
[ 551.751095] Code: 83 ec 10 48 8b 55 08 4c 89 65 f8 49 89 fc 48 8d 7f 18 48 89 5d f0 48 89 f3 be 01 00 00 00 e8 18 5b 92 ff 4c 89 e7 e8 90 f0 c0 ff c7 02 74 1b e8 d6 2e 92 ff 48 89 df 57 9d 0f 1f 44 00 00 48
[ 551.751251] NMI backtrace for cpu 0
[ 551.751267] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 551.751274] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 551.751282] task: ffffffff81a6f440 ti: ffffffff81a5c000 task.ti: ffffffff81a5c000
[ 551.751289] RIP: 0010:[] [] poll_idle+0x42/0x90
[ 551.751306] RSP: 0000:ffffffff81a5dd98 EFLAGS: 00000246
[ 551.751312] RAX: 0000000000080000 RBX: 0000000000000000 RCX: 0000000000000019
[ 551.751319] RDX: ffffffff81a5dfd8 RSI: 0000000000000000 RDI: ffffffff8160410f
[ 551.751325] RBP: ffffffff81a5ddd8 R08: 0000000000000020 R09: 0000000000000000
[ 551.751331] R10: 0000000000002000 R11: 0000000000000000 R12: ffff88007d9d9900
[ 551.751338] R13: 0000008072c52b51 R14: ffffffff81abbac0 R15: 0000000000000000
[ 551.751345] FS: 0000000000000000(0000) GS:ffff88007d800000(0000) knlGS:0000000000000000
[ 551.751351] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 551.751358] CR2: 00000000f77a0000 CR3: 000000007a798000 CR4: 00000000000007f0
[ 551.751364] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 551.751370] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 551.751375] Stack:
[ 551.751380] 0000000000000000 ffffffff81782820 ffffffff81a5ddd8 ffffffff81092d2f
[ 551.751396] ffffffff81abbac0 ffff88007d9d9900 0000008072c52896 ffffffff81abbad8
[ 551.751410] ffffffff81a5de38 ffffffff816044ea 0000000000000000 0000000000000000
[ 551.751424] Call Trace:
[ 551.751438] [] ? notifier_call_chain+0x110/0x110
[ 551.751451] [] ? ktime_get+0x4f/0xe0
[ 551.751462] [] cpuidle_enter_state+0x4a/0xe0
[ 551.751474] [] cpuidle_idle_call+0xa1/0x150
[ 551.751486] [] ? rcu_eqs_enter+0x60/0xa0
[ 551.751498] [] arch_cpu_idle+0x9/0x30
[ 551.751509] [] cpu_idle_loop+0x8a/0x190
[ 551.751520] [] cpu_startup_entry+0x1e/0x20
[ 551.751532] [] rest_init+0x12b/0x140
[ 551.751543] [] ? csum_partial_copy_generic+0x170/0x170
[ 551.751555] [] start_kernel+0x3b5/0x3c2
[ 551.751566] [] ? repair_env_string+0x5a/0x5a
[ 551.751576] [] ? early_idt_handlers+0x120/0x120
[ 551.751587] [] x86_64_start_reservations+0x2a/0x2c
[ 551.751598] [] x86_64_start_kernel+0xee/0xfd
[ 551.751604] Code: 49 89 c5 e8 f1 cd a9 ff fb 66 0f 1f 44 00 00 65 48 8b 14 25 70 c7 00 00 48 8b 82 38 e0 ff ff a8 08 75 13 66 0f 1f 44 00 00 f3 90 <48> 8b 82 38 e0 ff ff a8 08 74 f3 e8 9e eb a8 ff 4c 29 e8 48 89
[ 551.750013] NMI backtrace for cpu 1
[ 551.750013] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 551.750013] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 551.750013] task: ffff88007b7f0000 ti: ffff88007b7ea000 task.ti: ffff88007b7ea000
[ 551.750013] RIP: 0010:[] [] intel_idle+0xc7/0x130
[ 551.750013] RSP: 0018:ffff88007b7ebdf8 EFLAGS: 00000046
[ 551.750013] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001
[ 551.750013] RDX: 0000000000000000 RSI: ffff88007b7ebfd8 RDI: 0000000000000001
[ 551.750013] RBP: ffff88007b7ebe28 R08: 00000000000017ed R09: 0000000000de1471
[ 551.750013] R10: 0000000000000803 R11: 0000000000000000 R12: 0000000000000001
[ 551.750013] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000001
[ 551.750013] FS: 0000000000000000(0000) GS:ffff88007da00000(0000) knlGS:0000000000000000
[ 551.750013] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 551.750013] CR2: ffffffffff600400 CR3: 0000000073406000 CR4: 00000000000007e0
[ 551.750013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 551.750013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 551.750013] Stack:
[ 551.750013] ffff88007b7ebe28 0000000181092d2f ffff88007dbd9900 00000080764519d1
[ 551.750013] ffffffff81abbb30 ffffffff81abbac0 ffff88007b7ebe88 ffffffff816044ea
[ 551.750013] 0000000000000000 0000000002666768 0000000000000000 0000000002666768
[ 551.750013] Call Trace:
[ 551.750013] [] cpuidle_enter_state+0x4a/0xe0
[ 551.750013] [] cpuidle_idle_call+0xa1/0x150
[ 551.750013] [] ? rcu_eqs_enter+0x60/0xa0
[ 551.750013] [] arch_cpu_idle+0x9/0x30
[ 551.750013] [] cpu_idle_loop+0x8a/0x190
[ 551.750013] [] cpu_startup_entry+0x1e/0x20
[ 551.750013] [] start_secondary+0x85/0x87
[ 551.750013] Code: 48 8b 34 25 70 c7 00 00 48 89 d1 48 8d 86 38 e0 ff ff 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <85> 1d 13 30 6e 00 75 0e 48 8d 75 dc bf 05 00 00 00 e8 33 f6 cb
[ 576.200007] BUG: soft lockup - CPU#2 stuck for 22s! [zvol_id:25828]
[ 576.200007] Modules linked in: rpcsec_gss_krb5 zfs(O) zcommon(O) znvpair(O) zavl(O) zunicode(O) spl(O) iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bonding button processor ftdi_sio usbserial nls_iso8859_1 nls_cp437 virtio_scsi virtio_pci virtio_ring virtio vfat fat aufs
[ 576.200007] irq event stamp: 0
[ 576.200007] hardirqs last enabled at (0): < (null)>
[ 576.200007] hardirqs last disabled at (0): [] copy_process.part.41+0x3db/0xf30
[ 576.200007] softirqs last enabled at (0): [] copy_process.part.41+0x3db/0xf30
[ 576.200007] softirqs last disabled at (0): < (null)>
[ 576.200007] CPU: 2 PID: 25828 Comm: zvol_id Tainted: G O 3.10.30-oe64-00000-gdfc5587-dirty #101
[ 576.200007] Hardware name: To be filled by O.E.M. WDBLGT0080KBK-40/GA8-IBLV, BIOS 4.6.4 10/18/2011
[ 576.200007] task: ffff8800720e6220 ti: ffff880071ff0000 task.ti: ffff880071ff0000
[ 576.200007] RIP: 0010:[] [] mutex_lock_nested+0x2a9/0x3a0
[ 576.200007] RSP: 0018:ffff880071ff19d8 EFLAGS: 00000246
[ 576.200007] RAX: ffff8800720e6220 RBX: ffffffff810a2049 RCX: 0000000000000000
[ 576.200007] RDX: ffffffff81ab5c10 RSI: ffff880071ff1a08 RDI: 0000000000000246
[ 576.200007] RBP: ffff880071ff1a58 R08: 0000000000000000 R09: 0000000000000000
[ 576.200007] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ab5c38
[ 576.200007] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000246
[ 576.200007] FS: 00007f9c90cd06d0(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 576.200007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 576.200007] CR2: 00007f9c9082d1e0 CR3: 0000000078523000 CR4: 00000000000007e0
[ 576.200007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 576.200007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 576.200007] Stack:
[ 576.200007] ffffffff8147900f 0000000000000046 ffffffff81ab5c10 ffffffff8147900f
[ 576.200007] ffffffff81ab5c38 ffffffff810a0f0d ffff880071ff1a08 ffff880071ff1a08
[ 576.200007] 0000000000000000 ffff880071ff1a08 ffffffffa02d4b50 000000000e600000
[ 576.200007] Call Trace:
[ 576.200007] [] ? kobj_lookup+0x4f/0x170
[ 576.200007] [] ? kobj_lookup+0x4f/0x170
[ 576.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 576.200007] [] kobj_lookup+0x4f/0x170
[ 576.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 576.200007] [] get_gendisk+0x21/0x50
[ 576.200007] [] __blkdev_get+0x120/0x420
[ 576.200007] [] ? bd_acquire+0x1f/0x100
[ 576.200007] [] blkdev_get+0x59/0x1e0
[ 576.200007] [] ? _raw_spin_unlock+0x26/0x30
[ 576.200007] [] ? blkdev_get+0x1e0/0x1e0
[ 576.200007] [] blkdev_open+0x67/0x90
[ 576.200007] [] do_dentry_open+0x22e/0x2b0
[ 576.200007] [] finish_open+0x46/0x60
[ 576.200007] [] do_last+0x4fe/0x8d0
[ 576.200007] [] ? link_path_walk+0x235/0x950
[ 576.200007] [] path_openat+0xb6/0x510
[ 576.200007] [] ? __alloc_fd+0x29/0x220
[ 576.200007] [] do_filp_open+0x3d/0xa0
[ 576.200007] [] ? __alloc_fd+0x163/0x220
[ 576.200007] [] do_sys_open+0xf9/0x1e0
[ 576.200007] [] SyS_open+0x5a/0xe0
[ 576.200007] [] ? vtime_account_user+0x68/0x80
[ 576.200007] [] ? trace_hardirqs_on+0xd/0x10
[ 576.200007] [] ? syscall_trace_enter+0x1c/0x1f0
[ 576.200007] [] tracesys+0xdd/0xe2
[ 576.200007] Code: 48 8b 55 90 48 3b 53 50 48 89 43 60 75 06 c7 03 00 00 00 00 80 43 08 01 41 f7 c6 00 02 00 00 74 27 e8 0c 5a 92 ff 4c 89 f7 57 9d <0f> 1f 44 00 00 48 8d 7d b0 e8 39 0c 92 ff 48 83 c4 58 5b 41 5c

@behlendorf behlendorf added this to the 0.6.4 milestone Mar 21, 2014
@behlendorf behlendorf added the Bug label Mar 21, 2014
@behlendorf
Copy link
Contributor

Things have improved considerably in the master source. I would be very interested to see if you can reproduce this issue with the latest code.

@behlendorf behlendorf removed this from the 0.6.4 milestone Oct 29, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: ZVOL ZFS Volumes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@behlendorf @arko-pl