Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System hangs (rsync from other host) #2957

Closed
TioNisla opened this issue Dec 11, 2014 · 11 comments
Closed

System hangs (rsync from other host) #2957

TioNisla opened this issue Dec 11, 2014 · 11 comments

Comments

@TioNisla
Copy link 

root@amnesiac:~# uname -a
Linux amnesiac 3.18.0 #3 SMP Wed Dec 10 10:41:07 YEKT 2014 x86_64 Intel(R) Xeon(R) CPU X5660 @ 2.80GHz GenuineIntel GNU/Linux

spl 52479ecf58
zfs c944be5d7e

root@amnesiac:~# zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

    NAME                                      STATE     READ WRITE CKSUM
    storage                                   ONLINE       0     0     0
      wwn-0x60060e80102d34f00511c6d70000000a  ONLINE       0     0     0

errors: No known data errors

Dec 11 14:25:03 amnesiac kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=6002 jiffies, g=258, c=257, q=24151)
Dec 11 14:25:04 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:25:04 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:25:04 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:25:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:25:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:25:04 amnesiac kernel: Call Trace:
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:25:04 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:25:04 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:25:26 amnesiac kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 3} (detected by 1, t=6002 jiffies, g=-297, c=-298, q=1)
Dec 11 14:25:26 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:25:26 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:25:26 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:25:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:25:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:25:26 amnesiac kernel: Call Trace:
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:25:26 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:25:26 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:28:04 amnesiac kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=24007 jiffies, g=258, c=257, q=96724)
Dec 11 14:28:04 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:28:04 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:28:04 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:28:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:28:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:28:04 amnesiac kernel: Call Trace:
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:28:04 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:28:04 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:28:26 amnesiac kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 3} (detected by 1, t=24007 jiffies, g=-297, c=-298, q=2)
Dec 11 14:28:26 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:28:26 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:28:26 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:28:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:28:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:28:26 amnesiac kernel: Call Trace:
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:28:26 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:28:26 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:30:24 amnesiac kernel: SysRq : Show backtrace of all active CPUs
Dec 11 14:30:24 amnesiac kernel: sending NMI to all CPUs:
Dec 11 14:30:24 amnesiac kernel: NMI backtrace for cpu 2
Dec 11 14:30:24 amnesiac kernel: CPU: 2 PID: 559 Comm: bash Tainted: P           O   3.18.0 #3
Dec 11 14:30:24 amnesiac kernel: task: ffff880232246720 ti: ffff8800b8800000 task.ti: ffff8800b8800000
Dec 11 14:30:24 amnesiac kernel: RIP: 0010:[<ffffffff8101cf7c>]  [<ffffffff8101cf7c>] native_apic_mem_write+0x8/0x9
Dec 11 14:30:24 amnesiac kernel: RSP: 0018:ffff8800b8803e50  EFLAGS: 00000046
Dec 11 14:30:24 amnesiac kernel: RAX: 0000000000000000 RBX: 0000000000000292 RCX: 0000000000000007
Dec 11 14:30:24 amnesiac kernel: RDX: 0000000000000007 RSI: 0000000000000c00 RDI: 0000000000000300
Dec 11 14:30:24 amnesiac kernel: RBP: 0000000000000002 R08: 0000000000000400 R09: 0000000000000000
Dec 11 14:30:24 amnesiac kernel: R10: 00000000ffffffff R11: ffff88023ff9e61c R12: 0000000000000c00
Dec 11 14:30:24 amnesiac kernel: R13: 000000000000000f R14: ffffffffffffffff R15: 000000000076ffe8
Dec 11 14:30:24 amnesiac kernel: FS:  00007f78c3bd4700(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
Dec 11 14:30:24 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 11 14:30:24 amnesiac kernel: CR2: 00000000007dd000 CR3: 00000002222a2000 CR4: 00000000000007a0
Dec 11 14:30:24 amnesiac kernel: Stack:
Dec 11 14:30:24 amnesiac kernel:  ffffffff8101d1c4 000000000000fbfb 0000000000000001 0000000000000004
Dec 11 14:30:24 amnesiac kernel:  000000000000006c 0000000000000000 ffffffff8101aae0 ffffffff81634640
Dec 11 14:30:24 amnesiac kernel:  ffffffff811bec88 000000000000006c 0000000000000002 ffff8802210c7e00
Dec 11 14:30:24 amnesiac kernel: Call Trace:
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8101d1c4>] ? _flat_send_IPI_mask+0x59/0x63
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8101aae0>] ? arch_trigger_all_cpu_backtrace+0x78/0xa2
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff811bec88>] ? __handle_sysrq+0x89/0x112
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff811befcc>] ? write_sysrq_trigger+0x1f/0x2d
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810e67ec>] ? proc_reg_write+0x56/0x78
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810a7467>] ? vfs_write+0x98/0xfa
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810a7a15>] ? SyS_write+0x3e/0x72
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:30:24 amnesiac kernel: Code: e9 78 24 00 89 e8 eb 05 b8 ed ff ff ff 5b 5d 41 5c 41 5d 41 5e c3 8a 87 ac 00 00 00 c0 e8 02 83 e0 01 c3 89 ff 89 b7 00 d0 5f ff <c3> 89 ff 8b 87 00 d0 5f ff c3 48 8b 05 8b c5 3e 00 c3 31 c0 81 
Dec 11 14:30:24 amnesiac kernel: NMI backtrace for cpu 1
Dec 11 14:30:24 amnesiac kernel: CPU: 1 PID: 592 Comm: sshd Tainted: P           O   3.18.0 #3
Dec 11 14:30:24 amnesiac kernel: task: ffff8800003ba940 ti: ffff8800af014000 task.ti: ffff8800af014000
Dec 11 14:30:24 amnesiac kernel: RIP: 0010:[<ffffffff81062920>]  [<ffffffff81062920>] csd_lock_wait.isra.1+0x7/0xa
Dec 11 14:30:24 amnesiac kernel: RSP: 0018:ffff8800af017c30  EFLAGS: 00000202
Dec 11 14:30:24 amnesiac kernel: RAX: 0000000000000003 RBX: ffff88023fc90300 RCX: 0000000000000020
Dec 11 14:30:24 amnesiac kernel: RDX: ffff88023fd92e68 RSI: 0000000000000020 RDI: ffff88023fd92e80
Dec 11 14:30:24 amnesiac kernel: RBP: 0000000000000003 R08: ffff88023fc90308 R09: 0000000000000000
Dec 11 14:30:24 amnesiac kernel: R10: 000000000000003f R11: 0000000000000010 R12: ffff88023fd92e68
Dec 11 14:30:24 amnesiac kernel: R13: ffff88023fc90308 R14: 0000000000000020 R15: 0000000000000001
Dec 11 14:30:24 amnesiac kernel: FS:  00007fb878aa0700(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
Dec 11 14:30:24 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 11 14:30:24 amnesiac kernel: CR2: 00007fb877c53406 CR3: 00000000af7b3000 CR4: 00000000000007a0
Dec 11 14:30:24 amnesiac kernel: Stack:
Dec 11 14:30:24 amnesiac kernel:  ffffffff81062d4d 0000000000000028 ffffffff8102231f ffff8800af017ce0
Dec 11 14:30:24 amnesiac kernel:  0000000000000003 ffffffff81642a80 ffff8800af017d38 ffffffff8102231f
Dec 11 14:30:24 amnesiac kernel:  ffff8800af017ce0 ffff8800af017db8 ffff88023fd8c570 0000000000000004
Dec 11 14:30:24 amnesiac kernel: Call Trace:
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81062d4d>] ? smp_call_function_many+0x1dc/0x1ed
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8102231f>] ? __native_flush_tlb+0x7/0x7
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8102231f>] ? __native_flush_tlb+0x7/0x7
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81062d78>] ? smp_call_function+0x1a/0x1e
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81062d8f>] ? on_each_cpu+0x13/0x2e
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81022643>] ? flush_tlb_kernel_range+0x50/0x5d
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81097856>] ? __purge_vmap_area_lazy+0x300/0x358
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810979f1>] ? vm_unmap_aliases+0x143/0x152
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810208bc>] ? change_page_attr_set_clr+0xeb/0x2f5
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810a0358>] ? kfree+0x12/0x8d
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81020c6b>] ? change_page_attr_clear+0x26/0x2b
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81020d27>] ? set_memory_ro+0x1a/0x1f
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81069472>] ? do_seccomp+0x388/0x6f3
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810349ed>] ? SyS_prctl+0x12e/0x382
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:30:24 amnesiac kernel: Code: ff ff ff 3b 05 2e 24 5e 00 89 c3 7d 15 65 8b 04 25 1c a0 00 00 39 c3 74 dc 89 df e8 ae bc fd ff eb d3 5b c3 f6 07 01 74 04 f3 90 <eb> f7 c3 48 c7 c0 00 03 01 00 53 48 8b 1c d5 e0 48 64 81 48 83 
Dec 11 14:30:24 amnesiac kernel: NMI backtrace for cpu 0
Dec 11 14:30:24 amnesiac kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: P           O   3.18.0 #3
Dec 11 14:30:24 amnesiac kernel: task: ffffffff81616440 ti: ffffffff81600000 task.ti: ffffffff81600000
Dec 11 14:30:24 amnesiac kernel: RIP: 0010:[<ffffffff810087ea>]  [<ffffffff810087ea>] default_idle+0x2/0x3
Dec 11 14:30:24 amnesiac kernel: RSP: 0018:ffffffff81603f80  EFLAGS: 00000246
Dec 11 14:30:24 amnesiac kernel: RAX: ffffffff810087e8 RBX: ffffffff81603fd8 RCX: 00000000ffffffff
Dec 11 14:30:24 amnesiac kernel: RDX: ffffffff81626e00 RSI: 0000000000000000 RDI: ffff88023fc0bb0c
Dec 11 14:30:24 amnesiac kernel: RBP: ffffffff81603fd8 R08: 0000000000000000 R09: 0000000000000000
Dec 11 14:30:24 amnesiac kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff81603fd8
Dec 11 14:30:24 amnesiac kernel: R13: ffffffff816c7090 R14: 0000000000000000 R15: 0000000000000000
Dec 11 14:30:24 amnesiac kernel: FS:  0000000000000000(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
Dec 11 14:30:24 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 11 14:30:24 amnesiac kernel: CR2: 00007ff78b5fd0d8 CR3: 00000000b9ae7000 CR4: 00000000000007b0
Dec 11 14:30:24 amnesiac kernel: Stack:
Dec 11 14:30:24 amnesiac kernel:  ffffffff81048b17 ffffffffffffffff ffffffff816c08d0 ffff88023ffc8b00
Dec 11 14:30:24 amnesiac kernel:  ffffffff8165dcce ffffffff8165d797 0000000000000000 0000000000000000
Dec 11 14:30:24 amnesiac kernel:  ffffffff816c7090 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:30:24 amnesiac kernel: Call Trace:
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81048b17>] ? cpu_startup_entry+0xb1/0x113
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8165dcce>] ? start_kernel+0x341/0x34c
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff8165d797>] ? set_init_arg+0x4e/0x4e
Dec 11 14:30:24 amnesiac kernel: Code: 63 c3 48 89 ef ff c3 48 6b c0 14 48 8b b0 f4 eb 6d 81 48 89 f2 48 03 90 fc eb 6d 81 48 ff ca e8 72 ff ff ff eb d1 5b 5d c3 fb f4 <c3> 48 89 fe 48 c7 c7 d0 f6 6d 81 e9 87 2b 03 00 48 89 fe 48 c7 
Dec 11 14:30:24 amnesiac kernel: NMI backtrace for cpu 3
Dec 11 14:30:24 amnesiac kernel: CPU: 3 PID: 585 Comm: rsync Tainted: P           O   3.18.0 #3
Dec 11 14:30:24 amnesiac kernel: task: ffff880236a61b80 ti: ffff880235eec000 task.ti: ffff880235eec000
Dec 11 14:30:24 amnesiac kernel: RIP: 0010:[<ffffffff810497dc>]  [<ffffffff810497dc>] do_raw_spin_lock+0x19/0x1c
Dec 11 14:30:24 amnesiac kernel: RSP: 0018:ffff880235eefad8  EFLAGS: 00000002
Dec 11 14:30:24 amnesiac kernel: RAX: 0000000000000001 RBX: ffff880222e5a000 RCX: 000000000000001a
Dec 11 14:30:24 amnesiac kernel: RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff880222e5a018
Dec 11 14:30:24 amnesiac kernel: RBP: ffff880236a61b80 R08: 0000000000012600 R09: e0525e8c9b98c0a6
Dec 11 14:30:24 amnesiac kernel: R10: ffff880222e96470 R11: 0000000000000001 R12: ffffffffffffffff
Dec 11 14:30:24 amnesiac kernel: R13: 0000000000000000 R14: ffff880235eefd38 R15: 0000000000000000
Dec 11 14:30:24 amnesiac kernel: FS:  00007feb2e821700(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
Dec 11 14:30:24 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 11 14:30:24 amnesiac kernel: CR2: 0000000001b7cc58 CR3: 0000000232137000 CR4: 00000000000007a0
Dec 11 14:30:24 amnesiac kernel: Stack:
Dec 11 14:30:24 amnesiac kernel:  ffffffff81265498 ffffffffa00bb154 0000000000000018 ffff880235eefb90
Dec 11 14:30:24 amnesiac kernel:  ffff880222e7bd40 ffff880236a61b80 0000000000000001 ffff880222e5a000
Dec 11 14:30:24 amnesiac kernel:  0000000000000002 0000000000000002 ffffffff81163094 0000000080000000
Dec 11 14:30:24 amnesiac kernel: Call Trace:
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:30:24 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:30:24 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:30:24 amnesiac kernel: Code: 00 01 00 00 f0 0f c1 07 84 c0 74 05 e9 5b 0d 00 00 c3 b8 00 01 00 00 f0 66 0f c1 07 0f b6 d4 38 c2 74 0a 8a 07 38 d0 74 04 f3 90 <eb> f6 c3 c3 48 8b 07 65 48 03 04 25 c8 b8 00 00 80 00 01 c3 48 
Dec 11 14:31:04 amnesiac kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=42012 jiffies, g=258, c=257, q=166858)
Dec 11 14:31:04 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:31:04 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:31:04 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:31:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:31:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:31:04 amnesiac kernel: Call Trace:
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:31:04 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:31:04 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:31:26 amnesiac kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 3} (detected by 1, t=42012 jiffies, g=-297, c=-298, q=2)
Dec 11 14:31:26 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:31:26 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:31:26 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:31:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:31:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:31:26 amnesiac kernel: Call Trace:
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:31:26 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:31:26 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:34:04 amnesiac kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 1, t=60017 jiffies, g=258, c=257, q=191407)
Dec 11 14:34:04 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:34:04 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:34:04 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:34:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:34:04 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:34:04 amnesiac kernel: Call Trace:
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:34:04 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:34:04 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17
Dec 11 14:34:26 amnesiac kernel: INFO: rcu_bh detected stalls on CPUs/tasks: { 3} (detected by 1, t=60017 jiffies, g=-297, c=-298, q=2)
Dec 11 14:34:26 amnesiac kernel: Task dump for CPU 3:
Dec 11 14:34:26 amnesiac kernel: rsync           R  running task        0   585    581 0x00000008
Dec 11 14:34:26 amnesiac kernel:  0000000000000002 0000000000000001 0000000000000000 0000000000000000
Dec 11 14:34:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:34:26 amnesiac kernel:  0000000000000000 0000000000000000 0000000000000000 0000000000000000
Dec 11 14:34:26 amnesiac kernel: Call Trace:
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00a6c6b>] ? dbuf_find+0xa2/0xce [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00a8f93>] ? __dbuf_hold_impl+0x38b/0x3b4 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00a7285>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00a73c8>] ? dbuf_read+0x111/0x508 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff81264b95>] ? mutex_lock+0x9/0x25
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff81265498>] ? rwsem_down_read_failed+0x32/0xd9
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00bb154>] ? dnode_rele+0x3c/0x60 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff81163094>] ? call_rwsem_down_read_failed+0x14/0x30
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff812650c2>] ? down_read+0x17/0x19
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00f4801>] ? zap_get_leaf_byblk+0x12f/0x190 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00f48c0>] ? zap_deref_leaf+0x5e/0x67 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00f6026>] ? fzap_cursor_retrieve+0x76/0x1a1 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00f864a>] ? zap_cursor_retrieve+0xb7/0x1bc [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00ae078>] ? RW_WRITE_HELD+0x37/0x69 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa00ae146>] ? dmu_prefetch+0x9c/0x171 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa01115ec>] ? zfs_readdir+0x1c5/0x351 [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff810b18a5>] ? path_openat+0x21a/0x475
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff810b2a3d>] ? do_filp_open+0x2b/0x6f
Dec 11 14:34:26 amnesiac kernel:  [<ffffffffa0124b61>] ? zpl_iterate+0x38/0x4d [zfs]
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff810b410d>] ? iterate_dir+0x63/0xe5
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff810b426a>] ? SyS_getdents+0x6f/0xc5
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff810b3f3a>] ? fillonedir+0xa0/0xa0
Dec 11 14:34:26 amnesiac kernel:  [<ffffffff81266392>] ? system_call_fastpath+0x12/0x17

screenshot
@Boldie
Copy link

Boldie commented Dec 11, 2014

Maybe I trapped to the same issue, read here: #2941

@umask
Copy link

umask commented Dec 11, 2014

Probably not all known zfs hangs with xattr=sa are fixed: #2945

I still have kernel panic with xattr=sa and latest zfs/spl from testing repo.

@TioNisla
Copy link
Author

xattr=sa, acltype=posixacl, aclinherit=passthrough
It is not any kind of GPF/panic, it is clearly deadlock on directory creation with many acl's attached.

@TioNisla
Copy link
Author

There is a bunch of locking-related bugs:

root@amnesiac:~# zpool list
NAME      SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
storage  23,9T  96,6G  23,8T         -     0%     0%  1.00x  DEGRADED  -
root@amnesiac:~# 
root@amnesiac:~# zpool status
  pool: storage
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
    invalid.  Sufficient replicas exist for the pool to continue
    functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 193G in 0h55m with 0 errors on Thu Dec 11 15:45:01 2014
config:

    NAME                                        STATE     READ WRITE CKSUM
    storage                                     DEGRADED     0     0     0
      mirror-0                                  DEGRADED     0     0     0
        wwn-0x60060e80102d34f00511c6d70000000a  UNAVAIL      0     0     0
        wwn-0x60060e80102d35100511c6d90000000a  ONLINE       0     0     0
        **wwn-0x60060e80102d37f00511c7070000000a**  UNAVAIL      0     0     0

errors: No known data errors
root@amnesiac:~# ls -la /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 400 дек 12 10:06 .
drwxr-xr-x 7 root root 140 дек 12 10:06 ..
lrwxrwxrwx 1 root root   9 дек 12 10:06 scsi-360060e80102d34f00511c6d70000000a -> ../../sdb
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d34f00511c6d70000000a-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d34f00511c6d70000000a-part9 -> ../../sdb9
lrwxrwxrwx 1 root root   9 дек 12 10:06 scsi-360060e80102d35100511c6d90000000a -> ../../sdc
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d35100511c6d90000000a-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d35100511c6d90000000a-part9 -> ../../sdc9
lrwxrwxrwx 1 root root   9 дек 12 10:06 scsi-360060e80102d37f00511c7070000000a -> ../../sdd
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d37f00511c7070000000a-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  10 дек 12 10:06 scsi-360060e80102d37f00511c7070000000a-part9 -> ../../sdd9
lrwxrwxrwx 1 root root   9 дек 12 10:06 wwn-0x60060e80102d34f00511c6d70000000a -> ../../sdb
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d34f00511c6d70000000a-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d34f00511c6d70000000a-part9 -> ../../sdb9
lrwxrwxrwx 1 root root   9 дек 12 10:06 wwn-0x60060e80102d35100511c6d90000000a -> ../../sdc
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d35100511c6d90000000a-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d35100511c6d90000000a-part9 -> ../../sdc9
lrwxrwxrwx 1 root root   9 дек 12 10:06 wwn-0x60060e80102d37f00511c7070000000a -> ../../sdd
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d37f00511c7070000000a-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  10 дек 12 10:06 wwn-0x60060e80102d37f00511c7070000000a-part9 -> ../../sdd9
root@amnesiac:~# 
root@amnesiac:~# zpool online wwn-0x60060e80102d37f00511c7070000000a
missing device name
usage:
    online <pool> <device> ...
root@amnesiac:~# zpool online storage **wwn-0x60060e80102d37f00511c7070000000a**

Message from syslogd@amnesiac at Fri Dec 12 10:08:21 2014 ...
amnesiac kernel: BUG: spinlock lockup suspected on CPU#3, updatedb/649

Message from syslogd@amnesiac at Fri Dec 12 10:08:21 2014 ...
amnesiac kernel:  lock: 0xffff880235928e18, .magic: 00000000, .owner: /0, .owner_cpu: 898797088
warning: device 'wwn-0x60060e80102d37f00511c7070000000a' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
root@amnesiac:~# 
root@amnesiac:~# zpool status

<hungs>

reset

Dec 12 10:17:10 amnesiac kernel: INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 3, t=6002 jiffies, g=227, c=226, q=53)
Dec 12 10:17:10 amnesiac kernel: INFO: Stall ended before state dump start
Dec 12 10:17:15 amnesiac kernel: BUG: spinlock lockup suspected on CPU#0, updatedb/613
Dec 12 10:17:15 amnesiac kernel:  lock: 0xffff8800b709bb18, .magic: 00000000, .owner: /0, .owner_cpu: -1224099040
Dec 12 10:17:15 amnesiac kernel: CPU: 0 PID: 613 Comm: updatedb Tainted: P           O   3.18.0 #2
Dec 12 10:17:15 amnesiac kernel:  ffff8800b709bb18 ffff8800b877fa00 ffffffff81283961 0000000081624544
Dec 12 10:17:15 amnesiac kernel:  ffff8800b709bb20 ffff8800b877fa20 ffffffff81281f93 ffff8800b709bb18
Dec 12 10:17:15 amnesiac kernel:  00000000a6c79a38 ffff8800b877fa50 ffffffff8104eee5 0000000080000000
Dec 12 10:17:15 amnesiac kernel: Call Trace:
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81283961>] dump_stack+0x46/0x58
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81281f93>] spin_dump+0x8c/0x91
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8104eee5>] do_raw_spin_lock+0xbc/0x101
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff812878e2>] _raw_spin_lock_irq+0xa/0xc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81286e74>] rwsem_down_read_failed+0x37/0xdc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01499fa>] ? dnode_rele+0x5e/0x7f [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81178264>] call_rwsem_down_read_failed+0x14/0x30
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81286a66>] ? down_read+0x1c/0x20
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0186b8c>] zap_get_leaf_byblk+0x132/0x194 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0186c4d>] zap_deref_leaf+0x5f/0x6a [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0188451>] fzap_cursor_retrieve+0x7b/0x1a7 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa018ac09>] zap_cursor_retrieve+0xbc/0x1c4 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa013bcf8>] ? dmu_prefetch+0x95/0x170 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01a4bc0>] zfs_readdir+0x1fb/0x3b0 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa002db51>] ? tsd_hash_search+0x75/0x7f [spl]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa002e1be>] ? tsd_exit+0x52/0x131 [spl]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810b51d5>] ? cp_new_stat+0x106/0x113
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01b9394>] zpl_iterate+0x3c/0x53 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810befa7>] iterate_dir+0x71/0x103
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810bf120>] SyS_getdents+0x74/0xc9
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810bedb1>] ? fillonedir+0xaf/0xaf
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287e52>] system_call_fastpath+0x12/0x17
Dec 12 10:17:15 amnesiac kernel: sending NMI to all CPUs:
Dec 12 10:17:15 amnesiac kernel: NMI backtrace for cpu 0
Dec 12 10:17:15 amnesiac kernel: CPU: 0 PID: 613 Comm: updatedb Tainted: P           O   3.18.0 #2
Dec 12 10:17:15 amnesiac kernel: task: ffff880235027300 ti: ffff8800b877c000 task.ti: ffff8800b877c000
Dec 12 10:17:15 amnesiac kernel: RIP: 0010:[<ffffffff8101ea76>]  [<ffffffff8101ea76>] native_apic_mem_write+0xc/0xe
Dec 12 10:17:15 amnesiac kernel: RSP: 0018:ffff8800b877f9c0  EFLAGS: 00000046
Dec 12 10:17:15 amnesiac kernel: RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000007
Dec 12 10:17:15 amnesiac kernel: RDX: 0000000081627170 RSI: 0000000000000c00 RDI: 0000000000000300
Dec 12 10:17:15 amnesiac kernel: RBP: ffff8800b877f9c0 R08: 0000000000000400 R09: ffff88023ff7cbe4
Dec 12 10:17:15 amnesiac kernel: R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
Dec 12 10:17:15 amnesiac kernel: R13: 0000000000000c00 R14: 000000000000000f R15: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: FS:  00007f865291a700(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
Dec 12 10:17:15 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 12 10:17:15 amnesiac kernel: CR2: 0000000000b2e000 CR3: 00000000b875c000 CR4: 00000000000007b0
Dec 12 10:17:15 amnesiac kernel: Stack:
Dec 12 10:17:15 amnesiac kernel:  ffff8800b877f9f0 ffffffff8101ed37 0000000000000001 00000000a6c79a38
Dec 12 10:17:15 amnesiac kernel:  00000000a6c79a38 ffff8800b877fd28 ffff8800b877fa00 ffffffff8101ed7d
Dec 12 10:17:15 amnesiac kernel:  ffff8800b877fa20 ffffffff8101c398 ffff8800b877f7d0 ffff8800b709bb18
Dec 12 10:17:15 amnesiac kernel: Call Trace:
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101ed37>] _flat_send_IPI_mask+0x60/0x6b
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101ed7d>] flat_send_IPI_mask+0xc/0xe
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101c398>] arch_trigger_all_cpu_backtrace+0x7d/0xa9
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8104eeef>] do_raw_spin_lock+0xc6/0x101
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff812878e2>] _raw_spin_lock_irq+0xa/0xc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81286e74>] rwsem_down_read_failed+0x37/0xdc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01499fa>] ? dnode_rele+0x5e/0x7f [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81178264>] call_rwsem_down_read_failed+0x14/0x30
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81286a66>] ? down_read+0x1c/0x20
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0186b8c>] zap_get_leaf_byblk+0x132/0x194 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0186c4d>] zap_deref_leaf+0x5f/0x6a [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa0188451>] fzap_cursor_retrieve+0x7b/0x1a7 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa018ac09>] zap_cursor_retrieve+0xbc/0x1c4 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa013bcf8>] ? dmu_prefetch+0x95/0x170 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01a4bc0>] zfs_readdir+0x1fb/0x3b0 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa002db51>] ? tsd_hash_search+0x75/0x7f [spl]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa002e1be>] ? tsd_exit+0x52/0x131 [spl]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810b51d5>] ? cp_new_stat+0x106/0x113
Dec 12 10:17:15 amnesiac kernel:  [<ffffffffa01b9394>] zpl_iterate+0x3c/0x53 [zfs]
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810befa7>] iterate_dir+0x71/0x103
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287904>] ? _raw_spin_unlock+0x9/0xb
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810bf120>] SyS_getdents+0x74/0xc9
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff810bedb1>] ? fillonedir+0xaf/0xaf
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff81287e52>] system_call_fastpath+0x12/0x17
Dec 12 10:17:15 amnesiac kernel: Code: ff 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8a 87 cc 00 00 00 55 48 89 e5 5d c0 e8 02 83 e0 01 c3 55 89 ff 48 89 e5 89 b7 00 d0 5f ff <5d> c3 55 89 ff 48 89 e5 8b 87 00 d0 5f ff 5d c3 55 48 8b 05 ca 
Dec 12 10:17:15 amnesiac kernel: NMI backtrace for cpu 2
Dec 12 10:17:15 amnesiac kernel: CPU: 2 PID: 0 Comm: swapper/2 Tainted: P           O   3.18.0 #2
Dec 12 10:17:15 amnesiac kernel: task: ffff8802368fc7e0 ti: ffff880236938000 task.ti: ffff880236938000
Dec 12 10:17:15 amnesiac kernel: RIP: 0010:[<ffffffff81008e06>]  [<ffffffff81008e06>] default_idle+0x6/0x8
Dec 12 10:17:15 amnesiac kernel: RSP: 0018:ffff88023693bee8  EFLAGS: 00000246
Dec 12 10:17:15 amnesiac kernel: RAX: 0000000000000000 RBX: ffff88023693bfd8 RCX: 0000000200000000
Dec 12 10:17:15 amnesiac kernel: RDX: ffff88023693bfd8 RSI: 0000000000000000 RDI: ffff88023fd0bb4c
Dec 12 10:17:15 amnesiac kernel: RBP: ffff88023693bee8 R08: 0000000000000000 R09: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff88023693bfd8
Dec 12 10:17:15 amnesiac kernel: R13: ffff88023693bfd8 R14: 0000000000000000 R15: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: FS:  0000000000000000(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
Dec 12 10:17:15 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 12 10:17:15 amnesiac kernel: CR2: 0000000000af7000 CR3: 0000000000314000 CR4: 00000000000007a0
Dec 12 10:17:15 amnesiac kernel: Stack:
Dec 12 10:17:15 amnesiac kernel:  ffff88023693bef8 ffffffff8100947c ffff88023693bf28 ffffffff8104cdeb
Dec 12 10:17:15 amnesiac kernel:  ffffffff81628ce0 0000000000000002 0000000000000000 0000000000000000
Dec 12 10:17:15 amnesiac kernel:  ffff88023693bf48 ffffffff8101a643 00000000ffff8802 0000000000000000
Dec 12 10:17:15 amnesiac kernel: Call Trace:
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8100947c>] arch_cpu_idle+0xa/0xc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8104cdeb>] cpu_startup_entry+0xb8/0x11a
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101a643>] start_secondary+0x101/0x11f
Dec 12 10:17:15 amnesiac kernel: Code: c3 48 6b c0 14 48 8b b0 f4 3b 6e 81 48 89 f2 48 03 90 fc 3b 6e 81 48 ff ca e8 67 ff ff ff eb d1 5b 41 5c 5d c3 55 48 89 e5 fb f4 <5d> c3 55 48 89 fe 48 c7 c7 a0 8a 61 81 48 89 e5 e8 69 5b 03 00 
Dec 12 10:17:15 amnesiac kernel: NMI backtrace for cpu 3
Dec 12 10:17:15 amnesiac kernel: CPU: 3 PID: 0 Comm: swapper/3 Tainted: P           O   3.18.0 #2
Dec 12 10:17:15 amnesiac kernel: task: ffff8802368fcf10 ti: ffff88023693c000 task.ti: ffff88023693c000
Dec 12 10:17:15 amnesiac kernel: RIP: 0010:[<ffffffff81008e06>]  [<ffffffff81008e06>] default_idle+0x6/0x8
Dec 12 10:17:15 amnesiac kernel: RSP: 0018:ffff88023693fee8  EFLAGS: 00000246
Dec 12 10:17:15 amnesiac kernel: RAX: 0000000000000000 RBX: ffff88023693ffd8 RCX: ffffffff81057cfb
Dec 12 10:17:15 amnesiac kernel: RDX: ffff88023693ffd8 RSI: 0000000000000000 RDI: ffff88023fd8bb4c
Dec 12 10:17:15 amnesiac kernel: RBP: ffff88023693fee8 R08: 0000000000000000 R09: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff88023693ffd8
Dec 12 10:17:15 amnesiac kernel: R13: ffff88023693ffd8 R14: 0000000000000000 R15: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: FS:  0000000000000000(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
Dec 12 10:17:15 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 12 10:17:15 amnesiac kernel: CR2: 00000000015ed8a0 CR3: 00000000b86f3000 CR4: 00000000000007a0
Dec 12 10:17:15 amnesiac kernel: Stack:
Dec 12 10:17:15 amnesiac kernel:  ffff88023693fef8 ffffffff8100947c ffff88023693ff28 ffffffff8104cdeb
Dec 12 10:17:15 amnesiac kernel:  ffffffff81628ce0 0000000000000003 0000000000000000 0000000000000000
Dec 12 10:17:15 amnesiac kernel:  ffff88023693ff48 ffffffff8101a643 00000000ffff8802 0000000000000000
Dec 12 10:17:15 amnesiac kernel: Call Trace:
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8100947c>] arch_cpu_idle+0xa/0xc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8104cdeb>] cpu_startup_entry+0xb8/0x11a
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101a643>] start_secondary+0x101/0x11f
Dec 12 10:17:15 amnesiac kernel: Code: c3 48 6b c0 14 48 8b b0 f4 3b 6e 81 48 89 f2 48 03 90 fc 3b 6e 81 48 ff ca e8 67 ff ff ff eb d1 5b 41 5c 5d c3 55 48 89 e5 fb f4 <5d> c3 55 48 89 fe 48 c7 c7 a0 8a 61 81 48 89 e5 e8 69 5b 03 00 
Dec 12 10:17:15 amnesiac kernel: NMI backtrace for cpu 1
Dec 12 10:17:15 amnesiac kernel: CPU: 1 PID: 0 Comm: swapper/1 Tainted: P           O   3.18.0 #2
Dec 12 10:17:15 amnesiac kernel: task: ffff8802368fc0b0 ti: ffff880236934000 task.ti: ffff880236934000
Dec 12 10:17:15 amnesiac kernel: RIP: 0010:[<ffffffff81008e06>]  [<ffffffff81008e06>] default_idle+0x6/0x8
Dec 12 10:17:15 amnesiac kernel: RSP: 0018:ffff880236937ee8  EFLAGS: 00000246
Dec 12 10:17:15 amnesiac kernel: RAX: 0000000000000000 RBX: ffff880236937fd8 RCX: 0000000100000000
Dec 12 10:17:15 amnesiac kernel: RDX: ffff880236937fd8 RSI: 0000000000000000 RDI: ffff88023fc8bb4c
Dec 12 10:17:15 amnesiac kernel: RBP: ffff880236937ee8 R08: 0000000000000000 R09: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880236937fd8
Dec 12 10:17:15 amnesiac kernel: R13: ffff880236937fd8 R14: 0000000000000000 R15: 0000000000000000
Dec 12 10:17:15 amnesiac kernel: FS:  0000000000000000(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
Dec 12 10:17:15 amnesiac kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 12 10:17:15 amnesiac kernel: CR2: 00007f1e26a060b8 CR3: 000000000038b000 CR4: 00000000000007a0
Dec 12 10:17:15 amnesiac kernel: Stack:
Dec 12 10:17:15 amnesiac kernel:  ffff880236937ef8 ffffffff8100947c ffff880236937f28 ffffffff8104cdeb
Dec 12 10:17:15 amnesiac kernel:  ffffffff81628ce0 0000000000000001 0000000000000000 0000000000000000
Dec 12 10:17:15 amnesiac kernel:  ffff880236937f48 ffffffff8101a643 00000000ffff8802 0000000000000000
Dec 12 10:17:15 amnesiac kernel: Call Trace:
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8100947c>] arch_cpu_idle+0xa/0xc
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8104cdeb>] cpu_startup_entry+0xb8/0x11a
Dec 12 10:17:15 amnesiac kernel:  [<ffffffff8101a643>] start_secondary+0x101/0x11f
Dec 12 10:17:15 amnesiac kernel: Code: c3 48 6b c0 14 48 8b b0 f4 3b 6e 81 48 89 f2 48 03 90 fc 3b 6e 81 48 ff ca e8 67 ff ff ff eb d1 5b 41 5c 5d c3 55 48 89 e5 fb f4 <5d> c3 55 48 89 fe 48 c7 c7 a0 8a 61 81 48 89 e5 e8 69 5b 03 00

UPDATE: after reboot

root@amnesiac:/var/log# zpool status
  pool: storage
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 48K in 0h0m with 0 errors on Fri Dec 12 10:15:46 2014
config:

    NAME                                        STATE     READ WRITE CKSUM
    storage                                     ONLINE       0     0     0
      mirror-0                                  ONLINE       0     0     0
        wwn-0x60060e80102d34f00511c6d70000000a  ONLINE       0     0     0
        wwn-0x60060e80102d35100511c6d90000000a  ONLINE       0     0     0
        **wwn-0x60060e80102d37f00511c7070000000a**  ONLINE       0     0     1

errors: No known data errors

@TioNisla
Copy link
Author 

[  248.050183] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 3, t=6002 jiffies, g=10, c=9, q=2171)
[  248.082732] Task dump for CPU 2:
[  248.082734] rsync           R  running task        0   601    597 0x00000008
[  248.082739]  ffff8800b820b738 0000000000000246 000000002f9ef8e8 0000000000000000
[  248.082742]  ffff88022f9ef8f8 ffffffff8105063d ffff88022f9ef938 0000000000000046
[  248.082744]  0000000100000006 0000000030e58000 0000000000000000 ffff8800b820b738
[  248.082747] Call Trace:
[  248.082754]  [<ffffffff8105063d>] ? arch_local_irq_save+0x9/0xc
[  248.082759]  [<ffffffff81295277>] ? spin_bug+0x15/0x28
[  248.082762]  [<ffffffff81184674>] ? __delay+0xa/0xc
[  248.082765]  [<ffffffff81055bf8>] ? do_raw_spin_lock+0xab/0x101
[  248.082768]  [<ffffffff8129b32f>] ? _raw_spin_lock_irq+0x3a/0x41
[  248.082771]  [<ffffffff8129a995>] ? rwsem_down_read_failed+0x37/0xdc
[  248.082773]  [<ffffffff8129a995>] ? rwsem_down_read_failed+0x37/0xdc
[  248.082775]  [<ffffffff8129a33b>] ? mutex_unlock+0x9/0xb
[  248.082778]  [<ffffffff81185574>] ? call_rwsem_down_read_failed+0x14/0x30
[  248.082820]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  248.082823]  [<ffffffff8104fd6f>] ? __down_read+0x12/0x14
[  248.082825]  [<ffffffff8129a571>] ? down_read+0x3c/0x43
[  248.082861]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  248.082892]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  248.082923]  [<ffffffffa0513c6e>] ? zap_deref_leaf+0x62/0x6d [zfs]
[  248.082954]  [<ffffffffa0516f74>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  248.082994]  [<ffffffffa05154f5>] ? fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  248.083031]  [<ffffffffa0517ea3>] ? zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  248.083033]  [<ffffffff8104fe41>] ? up_read+0x23/0x27
[  248.083051]  [<ffffffffa04c82ac>] ? dmu_prefetch+0x95/0x170 [zfs]
[  248.083089]  [<ffffffffa05320d5>] ? zfs_readdir+0x1fb/0x3b0 [zfs]
[  248.083093]  [<ffffffff810a13ef>] ? handle_mm_fault+0x3be/0x726
[  248.083095]  [<ffffffff8129b3aa>] ? _raw_spin_unlock+0x23/0x27
[  248.083098]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  248.083100]  [<ffffffff8105253d>] ? trace_hardirqs_on+0xd/0xf
[  248.083102]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  248.083104]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  248.083139]  [<ffffffffa0546c11>] ? zpl_iterate+0x3c/0x53 [zfs]
[  248.083141]  [<ffffffff810c84ad>] ? iterate_dir+0x73/0x105
[  248.083143]  [<ffffffff810c86fd>] ? SyS_getdents+0x76/0xd2
[  248.083145]  [<ffffffff810c853f>] ? iterate_dir+0x105/0x105
[  248.083147]  [<ffffffff810a158f>] ? handle_mm_fault+0x55e/0x726
[  248.083149]  [<ffffffff8129bc92>] ? system_call_fastpath+0x12/0x17
[  253.144731] BUG: spinlock lockup suspected on CPU#2, rsync/601
[  253.161281]  lock: 0xffff8800b820d218, .magic: 00000000, .owner: SH\xffffff89\xfffffff3H\xffffff83\xffffffe\xffffffe83\xffffff92\xffffffda\xffffffffH\xffffff89\xffffffdeH\xffffff89\xffffffc5L\xffffff89\xffffffc6@�/-1017298596, .owner_cpu: -1205808608
[  253.195322] CPU: 2 PID: 601 Comm: rsync Tainted: G           O   3.18.0 #3
[  253.195324]  ffff8800b820d218 ffff88022f9ef9b0 ffffffff81296d5c 000000000000f5f4
[  253.195327]  ffff8800b820d220 ffff88022f9ef9d0 ffffffff8129525d ffff8800b820d218
[  253.195330]  00000000a6c5aa20 ffff88022f9efa00 ffffffff81055c09 0000000000000000
[  253.195333] Call Trace:
[  253.195341]  [<ffffffff81296d5c>] dump_stack+0x46/0x58
[  253.195344]  [<ffffffff8129525d>] spin_dump+0x8c/0x91
[  253.195348]  [<ffffffff81055c09>] do_raw_spin_lock+0xbc/0x101
[  253.195350]  [<ffffffff8129b32f>] _raw_spin_lock_irq+0x3a/0x41
[  253.195353]  [<ffffffff8129a995>] ? rwsem_down_read_failed+0x37/0xdc
[  253.195355]  [<ffffffff8129a995>] rwsem_down_read_failed+0x37/0xdc
[  253.195357]  [<ffffffff8129a33b>] ? mutex_unlock+0x9/0xb
[  253.195361]  [<ffffffff81185574>] call_rwsem_down_read_failed+0x14/0x30
[  253.195413]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195417]  [<ffffffff8104fd6f>] ? __down_read+0x12/0x14
[  253.195419]  [<ffffffff8129a571>] down_read+0x3c/0x43
[  253.195450]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195481]  [<ffffffffa0513baa>] zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195512]  [<ffffffffa0513c6e>] zap_deref_leaf+0x62/0x6d [zfs]
[  253.195543]  [<ffffffffa0516f74>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  253.195574]  [<ffffffffa05154f5>] fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  253.195606]  [<ffffffffa0517ea3>] zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  253.195608]  [<ffffffff8104fe41>] ? up_read+0x23/0x27
[  253.195626]  [<ffffffffa04c82ac>] ? dmu_prefetch+0x95/0x170 [zfs]
[  253.195660]  [<ffffffffa05320d5>] zfs_readdir+0x1fb/0x3b0 [zfs]
[  253.195664]  [<ffffffff810a13ef>] ? handle_mm_fault+0x3be/0x726
[  253.195665]  [<ffffffff8129b3aa>] ? _raw_spin_unlock+0x23/0x27
[  253.195668]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.195670]  [<ffffffff8105253d>] ? trace_hardirqs_on+0xd/0xf
[  253.195672]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.195674]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.195708]  [<ffffffffa0546c11>] zpl_iterate+0x3c/0x53 [zfs]
[  253.195710]  [<ffffffff810c84ad>] iterate_dir+0x73/0x105
[  253.195713]  [<ffffffff810c86fd>] SyS_getdents+0x76/0xd2
[  253.195714]  [<ffffffff810c853f>] ? iterate_dir+0x105/0x105
[  253.195716]  [<ffffffff810a158f>] ? handle_mm_fault+0x55e/0x726
[  253.195718]  [<ffffffff8129bc92>] system_call_fastpath+0x12/0x17
[  253.195720] sending NMI to all CPUs:
[  253.195736] NMI backtrace for cpu 2
[  253.195738] CPU: 2 PID: 601 Comm: rsync Tainted: G           O   3.18.0 #3
[  253.195740] task: ffff880230e58000 ti: ffff88022f9ec000 task.ti: ffff88022f9ec000
[  253.195741] RIP: 0010:[<ffffffff8101eed4>]  [<ffffffff8101eed4>] native_apic_mem_write+0xc/0xe
[  253.195746] RSP: 0018:ffff88022f9ef970  EFLAGS: 00000046
[  253.195747] RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000007
[  253.195748] RDX: 000000000000605f RSI: 0000000000000c00 RDI: 0000000000000300
[  253.195750] RBP: ffff88022f9ef970 R08: 0000000000000001 R09: 0000000000000000
[  253.195751] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
[  253.195752] R13: 0000000000000c00 R14: 000000000000000f R15: 0000000000000000
[  253.195753] FS:  00007f0ddeb34700(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
[  253.195754] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  253.195783] CR2: 0000000001cc4a38 CR3: 000000022ff32000 CR4: 00000000000007a0
[  253.195796] Stack:
[  253.195797]  ffff88022f9ef9a0 ffffffff8101f001 0000000000000001 00000000a6c5aa20
[  253.195800]  00000000a6c5aa20 ffff88022f9efd18 ffff88022f9ef9b0 ffffffff8101f05a
[  253.195803]  ffff88022f9ef9d0 ffffffff8101c7f0 ffff88022f9ef780 ffff8800b820d218
[  253.195805] Call Trace:
[  253.195808]  [<ffffffff8101f001>] _flat_send_IPI_mask+0x66/0x84
[  253.195810]  [<ffffffff8101f05a>] flat_send_IPI_mask+0xc/0xe
[  253.195813]  [<ffffffff8101c7f0>] arch_trigger_all_cpu_backtrace+0x7d/0xa9
[  253.195815]  [<ffffffff81055c13>] do_raw_spin_lock+0xc6/0x101
[  253.195817]  [<ffffffff8129b32f>] _raw_spin_lock_irq+0x3a/0x41
[  253.195819]  [<ffffffff8129a995>] ? rwsem_down_read_failed+0x37/0xdc
[  253.195821]  [<ffffffff8129a995>] rwsem_down_read_failed+0x37/0xdc
[  253.195823]  [<ffffffff8129a33b>] ? mutex_unlock+0x9/0xb
[  253.195825]  [<ffffffff81185574>] call_rwsem_down_read_failed+0x14/0x30
[  253.195858]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195860]  [<ffffffff8104fd6f>] ? __down_read+0x12/0x14
[  253.195862]  [<ffffffff8129a571>] down_read+0x3c/0x43
[  253.195893]  [<ffffffffa0513baa>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195924]  [<ffffffffa0513baa>] zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  253.195955]  [<ffffffffa0513c6e>] zap_deref_leaf+0x62/0x6d [zfs]
[  253.195986]  [<ffffffffa0516f74>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  253.196017]  [<ffffffffa05154f5>] fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  253.196050]  [<ffffffffa0517ea3>] zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  253.196054]  [<ffffffff8104fe41>] ? up_read+0x23/0x27
[  253.196078]  [<ffffffffa04c82ac>] ? dmu_prefetch+0x95/0x170 [zfs]
[  253.196119]  [<ffffffffa05320d5>] zfs_readdir+0x1fb/0x3b0 [zfs]
[  253.196123]  [<ffffffff810a13ef>] ? handle_mm_fault+0x3be/0x726
[  253.196125]  [<ffffffff8129b3aa>] ? _raw_spin_unlock+0x23/0x27
[  253.196127]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.196129]  [<ffffffff8105253d>] ? trace_hardirqs_on+0xd/0xf
[  253.196131]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.196133]  [<ffffffff810c8475>] ? iterate_dir+0x3b/0x105
[  253.196168]  [<ffffffffa0546c11>] zpl_iterate+0x3c/0x53 [zfs]
[  253.196170]  [<ffffffff810c84ad>] iterate_dir+0x73/0x105
[  253.196172]  [<ffffffff810c86fd>] SyS_getdents+0x76/0xd2
[  253.196174]  [<ffffffff810c853f>] ? iterate_dir+0x105/0x105
[  253.196176]  [<ffffffff810a158f>] ? handle_mm_fault+0x55e/0x726
[  253.196178]  [<ffffffff8129bc92>] system_call_fastpath+0x12/0x17
[  253.196179] Code: ff 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8a 87 0c 01 00 00 55 48 89 e5 5d c0 e8 02 83 e0 01 c3 55 89 ff 48 89 e5 89 b7 00 d0 5f ff <5d> c3 55 89 ff 48 89 e5 8b 87 00 d0 5f ff 5d c3 55 48 8b 05 6c 
[  253.196209] NMI backtrace for cpu 1
[  253.196212] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O   3.18.0 #3
[  253.196214] task: ffff880236938f90 ti: ffff880236944000 task.ti: ffff880236944000
[  253.196215] RIP: 0010:[<ffffffff81009037>]  [<ffffffff81009037>] default_idle+0xb/0xd
[  253.196220] RSP: 0018:ffff880236947ee8  EFLAGS: 00000246
[  253.196221] RAX: 0000000000000000 RBX: ffff880236947fd8 RCX: ffff880236947fd8
[  253.196223] RDX: 0000000000000046 RSI: 0000000000000000 RDI: ffffffff81009035
[  253.196224] RBP: ffff880236947ee8 R08: 0000000000000000 R09: 0000000000000000
[  253.196225] R10: ffffffff81040608 R11: 0000000000000001 R12: ffff880236947fd8
[  253.196226] R13: ffff880236947fd8 R14: 0000000000000000 R15: 0000000000000000
[  253.196227] FS:  0000000000000000(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
[  253.196228] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  253.196249] CR2: 0000000001125808 CR3: 0000000001611000 CR4: 00000000000007a0
[  253.196261] Stack:
[  253.196262]  ffff880236947ef8 ffffffff8100969b ffff880236947f28 ffffffff8104f3e4
[  253.196265]  ffff880236938f90 0000000000000001 0000000000000000 0000000000000000
[  253.196267]  ffff880236947f48 ffffffff8101a9ed 00000000ffff8802 0000000000000000
[  253.196270] Call Trace:
[  253.196273]  [<ffffffff8100969b>] arch_cpu_idle+0xa/0xc
[  253.196276]  [<ffffffff8104f3e4>] cpu_startup_entry+0xcc/0x133
[  253.196278]  [<ffffffff8101a9ed>] start_secondary+0x106/0x124
[  253.196279] Code: 81 48 89 e5 e8 5b 75 03 00 5d c3 55 48 89 fe 48 c7 c7 40 97 61 81 48 89 e5 e8 85 75 03 00 5d c3 55 48 89 e5 e8 fb 94 04 00 fb f4 <5d> c3 55 48 89 e5 53 48 89 fb 52 48 8b 37 48 85 f6 74 13 48 8b 
[  253.196309] NMI backtrace for cpu 0
[  253.196312] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O   3.18.0 #3
[  253.196313] task: ffffffff81616490 ti: ffffffff81600000 task.ti: ffffffff81600000
[  253.196314] RIP: 0010:[<ffffffff81009037>]  [<ffffffff81009037>] default_idle+0xb/0xd
[  253.196318] RSP: 0018:ffffffff81603ef8  EFLAGS: 00000246
[  253.196319] RAX: 0000000000000000 RBX: ffffffff81603fd8 RCX: ffffffff81603fd8
[  253.196320] RDX: 0000000000000046 RSI: 0000000000000000 RDI: ffffffff81009035
[  253.196321] RBP: ffffffff81603ef8 R08: 0000000000000000 R09: 0000000000000000
[  253.196323] R10: ffffffff81040608 R11: 0000000000000001 R12: ffffffff81603fd8
[  253.196324] R13: ffffffff81603fd8 R14: ffffffff816d7090 R15: 0000000000000000
[  253.196325] FS:  0000000000000000(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[  253.196326] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  253.196344] CR2: 0000000000c1e538 CR3: 0000000234d9f000 CR4: 00000000000007b0
[  253.196357] Stack:
[  253.196358]  ffffffff81603f08 ffffffff8100969b ffffffff81603f38 ffffffff8104f3e4
[  253.196360]  0000000000000000 0000000000000002 ffffffff816d0900 ffff88023ff68b00
[  253.196363]  ffffffff81603f68 ffffffff81291a2b ffffffff8129197c ffff880000001000
[  253.196366] Call Trace:
[  253.196368]  [<ffffffff8100969b>] arch_cpu_idle+0xa/0xc
[  253.196371]  [<ffffffff8104f3e4>] cpu_startup_entry+0xcc/0x133
[  253.196373]  [<ffffffff81291a2b>] rest_init+0xaf/0xb6
[  253.196375]  [<ffffffff8129197c>] ? csum_partial_copy_generic+0x16c/0x16c
[  253.196380]  [<ffffffff8166bd4f>] start_kernel+0x362/0x36f
[  253.196382]  [<ffffffff8166b7e5>] ? set_init_arg+0x55/0x55
[  253.196384]  [<ffffffff8166b442>] x86_64_start_reservations+0x2a/0x2c
[  253.196386]  [<ffffffff8166b4f5>] x86_64_start_kernel+0xb1/0xb5
[  253.196387] Code: 81 48 89 e5 e8 5b 75 03 00 5d c3 55 48 89 fe 48 c7 c7 40 97 61 81 48 89 e5 e8 85 75 03 00 5d c3 55 48 89 e5 e8 fb 94 04 00 fb f4 <5d> c3 55 48 89 e5 53 48 89 fb 52 48 8b 37 48 85 f6 74 13 48 8b 
[  253.196417] NMI backtrace for cpu 3
[  253.196420] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O   3.18.0 #3
[  253.196421] task: ffff88023693aeb0 ti: ffff88023694c000 task.ti: ffff88023694c000
[  253.196423] RIP: 0010:[<ffffffff81009037>]  [<ffffffff81009037>] default_idle+0xb/0xd
[  253.196426] RSP: 0018:ffff88023694fee8  EFLAGS: 00000246
[  253.196427] RAX: 0000000000000000 RBX: ffff88023694ffd8 RCX: ffff88023694ffd8
[  253.196428] RDX: 0000000000000046 RSI: 0000000000000000 RDI: ffffffff81009035
[  253.196429] RBP: ffff88023694fee8 R08: 0000000000000000 R09: 0000000000000000
[  253.196430] R10: ffffffff81040608 R11: 0000000000000001 R12: ffff88023694ffd8
[  253.196431] R13: ffff88023694ffd8 R14: 0000000000000000 R15: 0000000000000000
[  253.196433] FS:  0000000000000000(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
[  253.196434] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  253.196451] CR2: 00000000022a8808 CR3: 0000000230d33000 CR4: 00000000000007a0
[  253.196464] Stack:
[  253.196465]  ffff88023694fef8 ffffffff8100969b ffff88023694ff28 ffffffff8104f3e4
[  253.196468]  ffff88023693aeb0 0000000000000003 0000000000000000 0000000000000000
[  253.196470]  ffff88023694ff48 ffffffff8101a9ed 00000000ffff8802 0000000000000000
[  253.196473] Call Trace:
[  253.196476]  [<ffffffff8100969b>] arch_cpu_idle+0xa/0xc
[  253.196478]  [<ffffffff8104f3e4>] cpu_startup_entry+0xcc/0x133
[  253.196480]  [<ffffffff8101a9ed>] start_secondary+0x106/0x124
[  253.196482] Code: 81 48 89 e5 e8 5b 75 03 00 5d c3 55 48 89 fe 48 c7 c7 40 97 61 81 48 89 e5 e8 85 75 03 00 5d c3 55 48 89 e5 e8 fb 94 04 00 fb f4 <5d> c3 55 48 89 e5 53 48 89 fb 52 48 8b 37 48 85 f6 74 13 48 8b

@TioNisla
Copy link
Author

UP kernel

root@amnesiac:~# uname -a
Linux amnesiac 3.18.0 #5 Fri Dec 12 15:09:36 YEKT 2014 x86_64 Intel(R) Xeon(R) CPU X5660 @ 2.80GHz GenuineIntel GNU/Linux
root@amnesiac:~# dmesg
....
[    1.025891] SPL: Loaded module v0.6.3-1
[    1.082584] ZFS: Loaded module v0.6.3-1, ZFS pool version 5000, ZFS filesystem version 5

[    1.114890] =============================================
[    1.115150] [ INFO: possible recursive locking detected ]
[    1.115411] 3.18.0 #5 Tainted: G           O  
[    1.115671] ---------------------------------------------
[    1.115932] vdev_open/0/195 is trying to acquire lock:
[    1.116192]  (&zio->io_lock){+.+...}, at: [<ffffffffa021aa51>] zio_add_child+0x68/0x13f [zfs]
[    1.116453] 
but task is already holding lock:
[    1.116713]  (&zio->io_lock){+.+...}, at: [<ffffffffa021aa34>] zio_add_child+0x4b/0x13f [zfs]
[    1.116974] 
other info that might help us debug this:
[    1.117234]  Possible unsafe locking scenario:

[    1.117495]        CPU0
[    1.117636]        ----
[    1.117775]   lock(&zio->io_lock);
[    1.118036]   lock(&zio->io_lock);
[    1.118297] 
 *** DEADLOCK ***

[    1.118502]  May be due to missing lock nesting notation

[    1.118763] 1 lock held by vdev_open/0/195:
[    1.119023]  #0:  (&zio->io_lock){+.+...}, at: [<ffffffffa021aa34>] zio_add_child+0x4b/0x13f [zfs]
[    1.122946] 
stack backtrace:
[    1.122950] CPU: 0 PID: 195 Comm: vdev_open/0 Tainted: G           O   3.18.0 #5
[    1.122953]  ffffffff81e7cb00 ffff880139b5ba08 ffffffff81278458 ffff880139b5bac8
[    1.122957]  ffffffff8104560e ffff8800ba89acd0 ffffffff00000000 0000000000000000
[    1.122961]  ffffffff81e7cb00 0000000039b5ba58 00000000003d81ec ffff880139b5ba78
[    1.122964] Call Trace:
[    1.122971]  [<ffffffff81278458>] dump_stack+0x19/0x1b
[    1.122978]  [<ffffffff8104560e>] __lock_acquire+0xc4c/0xe0c
[    1.122986]  [<ffffffff8105ed15>] ? __module_address+0x53/0x67
[    1.122993]  [<ffffffff81044386>] ? mark_lock+0x2e/0x226
[    1.122999]  [<ffffffff810428a2>] ? arch_local_irq_save+0x9/0xc
[    1.123006]  [<ffffffff81045ac7>] lock_acquire+0x81/0x97
[    1.123013]  [<ffffffffa021aa51>] ? zio_add_child+0x68/0x13f [zfs]
[    1.123020]  [<ffffffff8127ad9a>] mutex_lock_nested+0x45/0x2ea
[    1.123028]  [<ffffffffa021aa51>] ? zio_add_child+0x68/0x13f [zfs]
[    1.123035]  [<ffffffffa021aa51>] ? zio_add_child+0x68/0x13f [zfs]
[    1.123042]  [<ffffffffa021aa51>] zio_add_child+0x68/0x13f [zfs]
[    1.123049]  [<ffffffffa021ae5f>] zio_create+0x337/0x355 [zfs]
[    1.123056]  [<ffffffffa021b4e7>] zio_read_phys+0x6c/0x76 [zfs]
[    1.123063]  [<ffffffffa01e3960>] ? vdev_accessible+0x52/0x52 [zfs]
[    1.123071]  [<ffffffffa01e27fb>] vdev_probe+0x1ae/0x1dd [zfs]
[    1.123078]  [<ffffffffa01e3960>] ? vdev_accessible+0x52/0x52 [zfs]
[    1.123085]  [<ffffffffa01e4c00>] ? vdev_set_state+0x8d/0x1ae [zfs]
[    1.123092]  [<ffffffffa01e501d>] vdev_open+0x2fc/0x367 [zfs]
[    1.123099]  [<ffffffff8127c2bd>] ? _raw_spin_unlock_irqrestore+0x3a/0x41
[    1.123106]  [<ffffffffa01e5177>] vdev_open_child+0x1d/0x32 [zfs]
[    1.123113]  [<ffffffffa002e0e9>] taskq_thread+0x1e5/0x2bc [spl]
[    1.123121]  [<ffffffff8103d01b>] ? wake_up_process+0x34/0x34
[    1.123128]  [<ffffffffa002df04>] ? taskq_dispatch_delay+0x103/0x103 [spl]
[    1.123135]  [<ffffffff81039cf8>] kthread+0xb4/0xbc
[    1.123142]  [<ffffffff81039c44>] ? init_completion+0x24/0x24
[    1.123149]  [<ffffffff8127caaa>] ret_from_fork+0x7a/0xb0
[    1.123156]  [<ffffffff81039c44>] ? init_completion+0x24/0x24
[    1.281683] SPL: using hostid 0x00000000

@behlendorf
Copy link
Contributor

@TioNisla could you test the patch in openzfs/spl#421 and see if it resolves this issue.

As for the lockdep analysis thanks for providing it but this is actually I known false positive. We need to tweak the code a little bit to make the analyzer happy.

@TioNisla
Copy link
Author

spl: a3c1eb7
zfs: 74328ee

[    1.590722] SPL: Loaded module v0.6.3-1
[    1.631415] ZFS: Loaded module v0.6.3-1, ZFS pool version 5000, ZFS filesystem version 5

[    1.652404] =============================================
[    1.652685] [ INFO: possible recursive locking detected ]
[    1.652826] 3.18.0 #1 Tainted: G           O  
[    1.652967] ---------------------------------------------
[    1.653108] vdev_open/0/231 is trying to acquire lock:
[    1.653249]  (&zio->io_lock){+.+...}, at: [<ffffffffa065e465>] zio_add_child+0x69/0x16f [zfs]
[    1.653389] 
but task is already holding lock:
[    1.653530]  (&zio->io_lock){+.+...}, at: [<ffffffffa065e447>] zio_add_child+0x4b/0x16f [zfs]
[    1.653671] 
other info that might help us debug this:
[    1.653812]  Possible unsafe locking scenario:

[    1.653953]        CPU0
[    1.654092]        ----
[    1.654228]   lock(&zio->io_lock);
[    1.654369]   lock(&zio->io_lock);
[    1.654510] 
 *** DEADLOCK ***

[    1.654650]  May be due to missing lock nesting notation

[    1.654791] 1 lock held by vdev_open/0/231:
[    1.654932]  #0:  (&zio->io_lock){+.+...}, at: [<ffffffffa065e447>] zio_add_child+0x4b/0x16f [zfs]
[    1.658558] 
stack backtrace:
[    1.678073] CPU: 0 PID: 231 Comm: vdev_open/0 Tainted: G           O   3.18.0 #1
[    1.688079]  ffffffff81e88260 ffff8802325879e8 ffffffff812a5d40 ffffffff81059a90
[    1.698004] tsc: Refined TSC clocksource calibration: 2799.201 MHz
[    1.708088]  ffff8800b9faaeb0 ffff880232587aa8 ffffffff81054dbf ffff880232587a28
[    1.718096]  0000000000000000 ffff880000000000 ffffffff81e88260 00000000a00558e8
[    1.728101] Call Trace:
[    1.738112]  [<ffffffff812a5d40>] dump_stack+0x46/0x58
[    1.748120]  [<ffffffff81059a90>] ? console_unlock+0x365/0x394
[    1.758277]  [<ffffffff81054dbf>] __lock_acquire+0xc50/0xe10
[    1.768125]  [<ffffffff8105fbe2>] ? rcu_read_lock_held+0x29/0x2f
[    1.778134]  [<ffffffff810734c4>] ? __module_address+0x6a/0x96
[    1.778414]  [<ffffffff81053b02>] ? mark_lock+0x2e/0x226
[    1.788158]  [<ffffffff8105527b>] lock_acquire+0x82/0x98
[    1.798209]  [<ffffffffa065e465>] ? zio_add_child+0x69/0x16f [zfs]
[    1.808170]  [<ffffffff812a7bd2>] mutex_lock_nested+0x51/0x366
[    1.808860]  [<ffffffffa065e465>] ? zio_add_child+0x69/0x16f [zfs]
[    1.809180]  [<ffffffffa065e465>] ? zio_add_child+0x69/0x16f [zfs]
[    1.818206]  [<ffffffffa065e465>] zio_add_child+0x69/0x16f [zfs]
[    1.828211]  [<ffffffffa065e8a2>] zio_create+0x337/0x355 [zfs]
[    1.828537]  [<ffffffffa065ef2a>] zio_read_phys+0x6c/0x76 [zfs]
[    1.829053]  [<ffffffffa0625cf6>] ? vdev_accessible+0x52/0x52 [zfs]
[    1.838214]  [<ffffffffa0624a4a>] vdev_probe+0x1c6/0x1f5 [zfs]
[    1.848423]  [<ffffffffa0625cf6>] ? vdev_accessible+0x52/0x52 [zfs]
[    1.858764]  [<ffffffffa062712d>] ? vdev_set_state+0x8d/0x1ae [zfs]
[    1.868224]  [<ffffffffa062754a>] vdev_open+0x2fc/0x367 [zfs]
[    1.878204]  [<ffffffff812aa5f5>] ? _raw_spin_unlock_irqrestore+0x3a/0x41
[    1.888238]  [<ffffffffa06276a5>] vdev_open_child+0x1e/0x33 [zfs]
[    1.898215]  [<ffffffffa004d3ee>] taskq_thread+0x1e9/0x2cd [spl]
[    1.908217]  [<ffffffff81047104>] ? wake_up_state+0xd/0xd
[    1.918228]  [<ffffffffa004d205>] ? taskq_dispatch_delay+0x103/0x103 [spl]
[    1.928227]  [<ffffffff81040cb8>] kthread+0xb5/0xbd
[    1.938232]  [<ffffffff81040c03>] ? __kthread_parkme+0x5c/0x5c
[    1.948237]  [<ffffffff812aadec>] ret_from_fork+0x7c/0xb0
[    1.958240]  [<ffffffff81040c03>] ? __kthread_parkme+0x5c/0x5c
[    2.291491] SPL: using hostid 0x00000000
[    2.336928]  zd0: unknown partition table
[    2.368684] EXT4-fs (sda1): re-mounted. Opts: discard
[    2.400874] EXT4-fs (sda1): re-mounted. Opts: discard
[    2.411518] EXT4-fs (sda2): mounting with "discard" option, but the device does not support discard
[    2.428539] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: discard
[    2.440986] EXT4-fs (sda3): mounting with "discard" option, but the device does not support discard
[    2.448757] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: discard
[    2.698605] Switched to clocksource tsc
[    4.185042] ip_tables: (C) 2000-2006 Netfilter Core Team
[    4.194245] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    4.747717] vmxnet3 0000:0b:00.0 eth0: intr type 3, mode 0, 5 vectors allocated
[    4.749286] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
[   62.143521] random: nonblocking pool is initialized

now rsync

[  244.108073] INFO: rcu_sched detected stalls on CPUs/tasks:
[  244.112570] BUG: spinlock lockup suspected on CPU#2, rsync/612
[  244.112601]  lock: 0xffff880229b61e18, .magic: 00000000, .owner: /3, .owner_cpu: 699801120
[  244.112604] CPU: 2 PID: 612 Comm: rsync Tainted: G           O   3.18.0 #1
[  244.112607]  ffff880229b61e18 ffff88022a8a79b0 ffffffff812a5d40 0000000000000000
[  244.112610]  ffff880229b61e20 ffff88022a8a79d0 ffffffff812a41e6 ffff880229b61e18
[  244.112612]  00000000a6c71568 ffff88022a8a7a00 ffffffff81057642 0000000000000000
[  244.112612] Call Trace:
[  244.112618]  [<ffffffff812a5d40>] dump_stack+0x46/0x58
[  244.112622]  [<ffffffff812a41e6>] spin_dump+0x8c/0x91
[  244.112626]  [<ffffffff81057642>] do_raw_spin_lock+0xbc/0x101
[  244.112629]  [<ffffffff812aa507>] _raw_spin_lock_irq+0x3a/0x41
[  244.112632]  [<ffffffff812a9b88>] ? rwsem_down_read_failed+0x37/0xdc
[  244.112634]  [<ffffffff812a9b88>] rwsem_down_read_failed+0x37/0xdc
[  244.112674]  [<ffffffffa05efb19>] ? dnode_rele+0x81/0xa9 [zfs]
[  244.112677]  [<ffffffff8118bc04>] call_rwsem_down_read_failed+0x14/0x30
[  244.112721]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.112723]  [<ffffffff81051703>] ? __down_read+0x12/0x14
[  244.112726]  [<ffffffff812a9764>] down_read+0x3c/0x43
[  244.112766]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.112806]  [<ffffffffa062f956>] zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.112847]  [<ffffffffa062fa1a>] zap_deref_leaf+0x62/0x6d [zfs]
[  244.112887]  [<ffffffffa0632d51>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  244.112927]  [<ffffffffa06312d2>] fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  244.112969]  [<ffffffffa0633c9a>] zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  244.112971]  [<ffffffff810517d5>] ? up_read+0x23/0x27
[  244.112995]  [<ffffffffa05e16ee>] ? dmu_prefetch+0x95/0x170 [zfs]
[  244.113041]  [<ffffffffa064e6a4>] zfs_readdir+0x1fb/0x3b0 [zfs]
[  244.113045]  [<ffffffff810a4396>] ? handle_mm_fault+0x3be/0x726
[  244.113047]  [<ffffffff812aa582>] ? _raw_spin_unlock+0x23/0x27
[  244.113052]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113055]  [<ffffffff81053f02>] ? trace_hardirqs_on+0xd/0xf
[  244.113057]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113059]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113112]  [<ffffffffa0663cbc>] zpl_iterate+0x41/0x58 [zfs]
[  244.113116]  [<ffffffff810cb9e3>] iterate_dir+0x73/0x105
[  244.113118]  [<ffffffff810cbc33>] SyS_getdents+0x76/0xd2
[  244.113120]  [<ffffffff810cba75>] ? iterate_dir+0x105/0x105
[  244.113123]  [<ffffffff810a4536>] ? handle_mm_fault+0x55e/0x726
[  244.113126]  [<ffffffff812aae92>] system_call_fastpath+0x12/0x17
[  244.113136] sending NMI to all CPUs:
[  244.113178] NMI backtrace for cpu 2
[  244.113180] CPU: 2 PID: 612 Comm: rsync Tainted: G           O   3.18.0 #1
[  244.113182] task: ffff8800bb458f90 ti: ffff88022a8a4000 task.ti: ffff88022a8a4000
[  244.113187] RIP: 0010:[<ffffffff8101efc9>]  [<ffffffff8101efc9>] native_apic_mem_write+0xc/0xe
[  244.113188] RSP: 0018:ffff88022a8a7970  EFLAGS: 00000046
[  244.113189] RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000018
[  244.113190] RDX: 0000000000000006 RSI: 0000000000000c00 RDI: 0000000000000300
[  244.113191] RBP: ffff88022a8a7970 R08: 0000000000000001 R09: 0000000000000000
[  244.113192] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000002
[  244.113192] R13: 0000000000000c00 R14: 000000000000000f R15: 0000000000000000
[  244.113194] FS:  00007fa2d34a7700(0000) GS:ffff88023fd00000(0000) knlGS:0000000000000000
[  244.113195] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  244.113225] CR2: 000000000284f9b8 CR3: 000000022a848000 CR4: 00000000000007a0
[  244.113241] Stack:
[  244.113243]  ffff88022a8a79a0 ffffffff8101f0f6 0000000000000001 00000000a6c71568
[  244.113245]  00000000a6c71568 ffff88022a8a7d18 ffff88022a8a79b0 ffffffff8101f14f
[  244.113255]  ffff88022a8a79d0 ffffffff8101c8e5 ffff88022a8a7780 ffff880229b61e18
[  244.113255] Call Trace:
[  244.113259]  [<ffffffff8101f0f6>] _flat_send_IPI_mask+0x66/0x84
[  244.113261]  [<ffffffff8101f14f>] flat_send_IPI_mask+0xc/0xe
[  244.113263]  [<ffffffff8101c8e5>] arch_trigger_all_cpu_backtrace+0x7d/0xa9
[  244.113265]  [<ffffffff8105764c>] do_raw_spin_lock+0xc6/0x101
[  244.113267]  [<ffffffff812aa507>] _raw_spin_lock_irq+0x3a/0x41
[  244.113269]  [<ffffffff812a9b88>] ? rwsem_down_read_failed+0x37/0xdc
[  244.113270]  [<ffffffff812a9b88>] rwsem_down_read_failed+0x37/0xdc
[  244.113297]  [<ffffffffa05efb19>] ? dnode_rele+0x81/0xa9 [zfs]
[  244.113299]  [<ffffffff8118bc04>] call_rwsem_down_read_failed+0x14/0x30
[  244.113335]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.113338]  [<ffffffff81051703>] ? __down_read+0x12/0x14
[  244.113339]  [<ffffffff812a9764>] down_read+0x3c/0x43
[  244.113375]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.113411]  [<ffffffffa062f956>] zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.113447]  [<ffffffffa062fa1a>] zap_deref_leaf+0x62/0x6d [zfs]
[  244.113484]  [<ffffffffa0632d51>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  244.113520]  [<ffffffffa06312d2>] fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  244.113557]  [<ffffffffa0633c9a>] zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  244.113559]  [<ffffffff810517d5>] ? up_read+0x23/0x27
[  244.113582]  [<ffffffffa05e16ee>] ? dmu_prefetch+0x95/0x170 [zfs]
[  244.113624]  [<ffffffffa064e6a4>] zfs_readdir+0x1fb/0x3b0 [zfs]
[  244.113627]  [<ffffffff810a4396>] ? handle_mm_fault+0x3be/0x726
[  244.113629]  [<ffffffff812aa582>] ? _raw_spin_unlock+0x23/0x27
[  244.113632]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113633]  [<ffffffff81053f02>] ? trace_hardirqs_on+0xd/0xf
[  244.113635]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113637]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.113677]  [<ffffffffa0663cbc>] zpl_iterate+0x41/0x58 [zfs]
[  244.113680]  [<ffffffff810cb9e3>] iterate_dir+0x73/0x105
[  244.113681]  [<ffffffff810cbc33>] SyS_getdents+0x76/0xd2
[  244.113683]  [<ffffffff810cba75>] ? iterate_dir+0x105/0x105
[  244.113686]  [<ffffffff810a4536>] ? handle_mm_fault+0x55e/0x726
[  244.113688]  [<ffffffff812aae92>] system_call_fastpath+0x12/0x17
[  244.113706] Code: ff 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 8a 87 0c 01 00 00 55 48 89 e5 5d c0 e8 02 83 e0 01 c3 55 89 ff 48 89 e5 89 b7 00 d0 5f ff <5d> c3 55 89 ff 48 89 e5 8b 87 00 d0 5f ff 5d c3 55 48 8b 05 77 
[  244.113708] NMI backtrace for cpu 0
[  244.113710] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O   3.18.0 #1
[  244.113711] task: ffffffff81616490 ti: ffffffff81600000 task.ti: ffffffff81600000
[  244.113715] RIP: 0010:[<ffffffff811b5f2b>]  [<ffffffff811b5f2b>] setcolor+0x12/0x3b
[  244.113716] RSP: 0018:ffff88023fc03a08  EFLAGS: 00000046
[  244.113716] RAX: 0000000000000000 RBX: ffff880236a32800 RCX: 0000000000002d00
[  244.113717] RDX: 00000000000003ce RSI: ffff88023fc03af8 RDI: 0000000000000007
[  244.113718] RBP: ffff88023fc03a18 R08: 0000000000000013 R09: ffff880236a32bf8
[  244.113719] R10: 0000000000000019 R11: 0000000000000000 R12: 0000000000000007
[  244.113720] R13: ffff880235850390 R14: 0000000000000019 R15: ffff88023fc03af8
[  244.113721] FS:  0000000000000000(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[  244.113722] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  244.113746] CR2: 00007fff52a4afd0 CR3: 000000022a86f000 CR4: 00000000000007b0
[  244.113761] Stack:
[  244.113764]  ffff880236a32800 ffff8800000a2d13 ffff88023fc03a78 ffffffff811b62c8
[  244.113765]  ffff88023fc03a38 ffffffff811b4d2c ffff880236a32800 ffff8800000a913f
[  244.113767]  ffff88023fc03aa8 ffff880236a32800 ffff880236810000 00000000000001ff
[  244.113768] Call Trace:
[  244.113775]  <IRQ> 
[  244.113776]  [<ffffffff811b62c8>] vga16fb_imageblit+0x98/0x3c7
[  244.113779]  [<ffffffff811b4d2c>] ? setmode+0x33/0x3b
[  244.113789]  [<ffffffff811aeb2a>] bit_putcs+0x3dc/0x437
[  244.113792]  [<ffffffff811ae701>] ? bit_cursor+0x488/0x4d5
[  244.113794]  [<ffffffff811ae74e>] ? bit_cursor+0x4d5/0x4d5
[  244.113795]  [<ffffffff811aa17e>] fbcon_putcs+0xc1/0xd0
[  244.113798]  [<ffffffff811abd08>] fbcon_redraw.isra.22+0xc4/0x15c
[  244.113799]  [<ffffffff811ac379>] fbcon_scroll+0x5d9/0xac5
[  244.113803]  [<ffffffff811f3e73>] scrup+0xc6/0xd3
[  244.113811]  [<ffffffff811f3eaa>] lf+0x2a/0x63
[  244.113813]  [<ffffffff811f5d66>] vt_console_print+0x194/0x2a0
[  244.113815]  [<ffffffff81058ba9>] call_console_drivers.constprop.30+0x78/0x83
[  244.113817]  [<ffffffff8105987c>] console_unlock+0x151/0x394
[  244.113819]  [<ffffffff81059e65>] ? vprintk_emit+0x3a6/0x420
[  244.113821]  [<ffffffff81059ece>] vprintk_emit+0x40f/0x420
[  244.113823]  [<ffffffff812a42e9>] printk+0x48/0x4a
[  244.113825]  [<ffffffff81059a90>] ? console_unlock+0x365/0x394
[  244.113829]  [<ffffffff81062deb>] rcu_check_callbacks+0x30e/0x513
[  244.113831]  [<ffffffff810645c3>] update_process_times+0x39/0x62
[  244.113833]  [<ffffffff810694e7>] ? update_wall_time+0x375/0x384
[  244.113835]  [<ffffffff8106f9dd>] tick_nohz_handler+0x93/0xd5
[  244.113838]  [<ffffffff8101ba84>] local_apic_timer_interrupt+0x50/0x54
[  244.113839]  [<ffffffff8101bced>] smp_apic_timer_interrupt+0x2c/0x3e
[  244.113841]  [<ffffffff812abcac>] apic_timer_interrupt+0x6c/0x80
[  244.113842]  <EOI> 
[  244.113846]  [<ffffffff810415a2>] ? unregister_die_notifier+0x15/0x15
[  244.113849]  [<ffffffff8100912a>] ? default_idle+0x9/0xd
[  244.113850]  [<ffffffff8100912c>] ? default_idle+0xb/0xd
[  244.113852]  [<ffffffff8100912a>] ? default_idle+0x9/0xd
[  244.113854]  [<ffffffff81009790>] arch_cpu_idle+0xa/0xc
[  244.113856]  [<ffffffff81050d78>] cpu_startup_entry+0xcc/0x133
[  244.113858]  [<ffffffff812a090f>] rest_init+0x123/0x12a
[  244.113859]  [<ffffffff812a07ec>] ? csum_partial_copy_generic+0x16c/0x16c
[  244.113862]  [<ffffffff8166bd4f>] start_kernel+0x362/0x36f
[  244.113864]  [<ffffffff8166b7e5>] ? set_init_arg+0x55/0x55
[  244.113866]  [<ffffffff8166b442>] x86_64_start_reservations+0x2a/0x2c
[  244.113868]  [<ffffffff8166b4f5>] x86_64_start_kernel+0xb1/0xb5
[  244.113886] Code: eb c8 48 89 df e8 f4 d6 ff ff 48 83 c4 28 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 ba ce 03 00 00 31 c0 48 89 e5 41 54 41 89 fc 53 ee <e8> bb 2d e5 ff ba cf 03 00 00 ec 88 c3 e8 ae 2d e5 ff 41 0f b6 
[  244.113888] NMI backtrace for cpu 1
[  244.113891] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G           O   3.18.0 #1
[  244.113892] task: ffff880236938f90 ti: ffff880236944000 task.ti: ffff880236944000
[  244.113895] RIP: 0010:[<ffffffff8100912c>]  [<ffffffff8100912c>] default_idle+0xb/0xd
[  244.113896] RSP: 0018:ffff880236947ee8  EFLAGS: 00000246
[  244.113897] RAX: 0000000000000000 RBX: ffff880236947fd8 RCX: 00000000ffffffff
[  244.113898] RDX: 0000000000000046 RSI: 0000000000000000 RDI: ffffffff8100912a
[  244.113898] RBP: ffff880236947ee8 R08: 0000000000000000 R09: 0000000000000001
[  244.113899] R10: ffffffff810415a2 R11: 0000000000000001 R12: ffff880236947fd8
[  244.113899] R13: ffff880236947fd8 R14: 0000000000000000 R15: 0000000000000000
[  244.113900] FS:  0000000000000000(0000) GS:ffff88023fc80000(0000) knlGS:0000000000000000
[  244.113901] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  244.113920] CR2: 00000000008342a8 CR3: 0000000000024000 CR4: 00000000000007a0
[  244.113933] Stack:
[  244.113935]  ffff880236947ef8 ffffffff81009790 ffff880236947f28 ffffffff81050d78
[  244.113936]  ffff880236938f90 0000000000000001 0000000000000000 0000000000000000
[  244.113937]  ffff880236947f48 ffffffff8101aae2 00000000ffff8802 0000000000000000
[  244.113938] Call Trace:
[  244.113940]  [<ffffffff81009790>] arch_cpu_idle+0xa/0xc
[  244.113942]  [<ffffffff81050d78>] cpu_startup_entry+0xcc/0x133
[  244.113944]  [<ffffffff8101aae2>] start_secondary+0x106/0x124
[  244.113966] Code: 81 48 89 e5 e8 00 84 03 00 5d c3 55 48 89 fe 48 c7 c7 40 97 61 81 48 89 e5 e8 2a 84 03 00 5d c3 55 48 89 e5 e8 cb ad 04 00 fb f4 <5d> c3 55 48 89 e5 53 48 89 fb 52 48 8b 37 48 85 f6 74 13 48 8b 
[  244.113968] NMI backtrace for cpu 3
[  244.113970] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G           O   3.18.0 #1
[  244.113971] task: ffff88023693aeb0 ti: ffff88023694c000 task.ti: ffff88023694c000
[  244.113974] RIP: 0010:[<ffffffff8100912c>]  [<ffffffff8100912c>] default_idle+0xb/0xd
[  244.113975] RSP: 0018:ffff88023694fee8  EFLAGS: 00000246
[  244.113976] RAX: 0000000000000000 RBX: ffff88023694ffd8 RCX: 00000000ffffffff
[  244.113977] RDX: 0000000000000046 RSI: 0000000000000000 RDI: ffffffff8100912a
[  244.113977] RBP: ffff88023694fee8 R08: 0000000000000000 R09: 0000000000000001
[  244.113978] R10: ffffffff810415a2 R11: 0000000000000001 R12: ffff88023694ffd8
[  244.113978] R13: ffff88023694ffd8 R14: 0000000000000000 R15: 0000000000000000
[  244.113979] FS:  0000000000000000(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
[  244.113980] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  244.113999] CR2: 000000000069d808 CR3: 0000000001611000 CR4: 00000000000007a0
[  244.114011] Stack:
[  244.114013]  ffff88023694fef8 ffffffff81009790 ffff88023694ff28 ffffffff81050d78
[  244.114014]  ffff88023693aeb0 0000000000000003 0000000000000000 0000000000000000
[  244.114016]  ffff88023694ff48 ffffffff8101aae2 00000000ffff8802 0000000000000000
[  244.114016] Call Trace:
[  244.114018]  [<ffffffff81009790>] arch_cpu_idle+0xa/0xc
[  244.114020]  [<ffffffff81050d78>] cpu_startup_entry+0xcc/0x133
[  244.114021]  [<ffffffff8101aae2>] start_secondary+0x106/0x124
[  244.114034] Code: 81 48 89 e5 e8 00 84 03 00 5d c3 55 48 89 fe 48 c7 c7 40 97 61 81 48 89 e5 e8 2a 84 03 00 5d c3 55 48 89 e5 e8 cb ad 04 00 fb f4 <5d> c3 55 48 89 e5 53 48 89 fb 52 48 8b 37 48 85 f6 74 13 48 8b 
[  244.157462]  2: (1 GPs behind) idle=5c9/140000000000000/0 softirq=479/480 
[  244.173592]  (detected by 0, t=6502 jiffies, g=47, c=46, q=223)
[  244.189763] Task dump for CPU 2:
[  244.189765] rsync           R  running task        0   612    608 0x00000008
[  244.189768]  ffffffff816296e0 ffff88022a8a7860 ffffffff8105217b ffff88022a8a7880
[  244.189771]  ffffffff812aa5ee ffffffff816296e0 00000000ffffffff ffff88022a8a78a0
[  244.189773]  ffffffff8105159b 0000000000000002 ffffffff81059e65 ffff88022a8a78d0
[  244.189775] Call Trace:
[  244.189780]  [<ffffffff8105217b>] ? trace_hardirqs_off+0xd/0xf
[  244.189783]  [<ffffffff812aa5ee>] ? _raw_spin_unlock_irqrestore+0x33/0x41
[  244.189785]  [<ffffffff8105159b>] ? down_trylock+0x27/0x32
[  244.189787]  [<ffffffff81059e65>] ? vprintk_emit+0x3a6/0x420
[  244.189789]  [<ffffffff810595cb>] ? console_trylock+0x1b/0x95
[  244.189791]  [<ffffffff81059ec4>] ? vprintk_emit+0x405/0x420
[  244.189794]  [<ffffffff8101b631>] ? apic_read+0x11/0x13
[  244.189796]  [<ffffffff8105217b>] ? trace_hardirqs_off+0xd/0xf
[  244.189799]  [<ffffffff8101f102>] ? _flat_send_IPI_mask+0x72/0x84
[  244.189802]  [<ffffffff8118ad04>] ? __delay+0xa/0xc
[  244.189804]  [<ffffffff8118ad2a>] ? __const_udelay+0x24/0x26
[  244.189806]  [<ffffffff8101c907>] ? arch_trigger_all_cpu_backtrace+0x9f/0xa9
[  244.189808]  [<ffffffff8105764c>] ? do_raw_spin_lock+0xc6/0x101
[  244.189810]  [<ffffffff812aa507>] ? _raw_spin_lock_irq+0x3a/0x41
[  244.189811]  [<ffffffff812a9b88>] ? rwsem_down_read_failed+0x37/0xdc
[  244.189813]  [<ffffffff812a9b88>] ? rwsem_down_read_failed+0x37/0xdc
[  244.189844]  [<ffffffffa05efb19>] ? dnode_rele+0x81/0xa9 [zfs]
[  244.189848]  [<ffffffff8118bc04>] ? call_rwsem_down_read_failed+0x14/0x30
[  244.189881]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.189884]  [<ffffffff81051703>] ? __down_read+0x12/0x14
[  244.189886]  [<ffffffff812a9764>] ? down_read+0x3c/0x43
[  244.189917]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.189949]  [<ffffffffa062f956>] ? zap_get_leaf_byblk+0x141/0x1a3 [zfs]
[  244.189981]  [<ffffffffa062fa1a>] ? zap_deref_leaf+0x62/0x6d [zfs]
[  244.190039]  [<ffffffffa0632d51>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[  244.190073]  [<ffffffffa06312d2>] ? fzap_cursor_retrieve+0x7e/0x1b0 [zfs]
[  244.190105]  [<ffffffffa0633c9a>] ? zap_cursor_retrieve+0xbf/0x1d0 [zfs]
[  244.190107]  [<ffffffff810517d5>] ? up_read+0x23/0x27
[  244.190126]  [<ffffffffa05e16ee>] ? dmu_prefetch+0x95/0x170 [zfs]
[  244.190166]  [<ffffffffa064e6a4>] ? zfs_readdir+0x1fb/0x3b0 [zfs]
[  244.190170]  [<ffffffff810a4396>] ? handle_mm_fault+0x3be/0x726
[  244.190172]  [<ffffffff812aa582>] ? _raw_spin_unlock+0x23/0x27
[  244.190175]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.190177]  [<ffffffff81053f02>] ? trace_hardirqs_on+0xd/0xf
[  244.190179]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.190180]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[  244.190216]  [<ffffffffa0663cbc>] ? zpl_iterate+0x41/0x58 [zfs]
[  244.190218]  [<ffffffff810cb9e3>] ? iterate_dir+0x73/0x105
[  244.190220]  [<ffffffff810cbc33>] ? SyS_getdents+0x76/0xd2
[  244.190222]  [<ffffffff810cba75>] ? iterate_dir+0x105/0x105
[  244.190224]  [<ffffffff810a4536>] ? handle_mm_fault+0x55e/0x726
[  244.190226]  [<ffffffff812aae92>] ? system_call_fastpath+0x12/0x17

@TioNisla
Copy link
Author

--enable-debug

[    2.219147] SPL: Loaded module v0.6.3-1 (DEBUG mode)
[    2.268462] ZFS: Loaded module v0.6.3-1 (DEBUG mode), ZFS pool version 5000, ZFS filesystem version 5

[    2.527587] =============================================
[    2.527848] [ INFO: possible recursive locking detected ]
[    2.528108] 3.18.0 #1 Tainted: G           O  
[    2.528369] ---------------------------------------------
[    2.528630] vdev_open/0/232 is trying to acquire lock:
[    2.528890]  (&zio->io_lock){+.+...}, at: [<ffffffffa01829e8>] zio_add_child+0x109/0x246 [zfs]
[    2.529151] 
but task is already holding lock:
[    2.529411]  (&zio->io_lock){+.+...}, at: [<ffffffffa0182992>] zio_add_child+0xb3/0x246 [zfs]
[    2.529672] 
other info that might help us debug this:
[    2.529932]  Possible unsafe locking scenario:

[    2.530193]        CPU0
[    2.530330]        ----
[    2.530465]   lock(&zio->io_lock);
[    2.530726]   lock(&zio->io_lock);
[    2.530986] 
 *** DEADLOCK ***

[    2.531178]  May be due to missing lock nesting notation

[    2.531439] 1 lock held by vdev_open/0/232:
[    2.531699]  #0:  (&zio->io_lock){+.+...}, at: [<ffffffffa0182992>] zio_add_child+0xb3/0x246 [zfs]
[    2.534527] 
stack backtrace:
[    2.534788] CPU: 1 PID: 232 Comm: vdev_open/0 Tainted: G           O   3.18.0 #1
[    2.535048]  ffffffff81e88c80 ffff8802322b79d8 ffffffff812a5d40 ffffffff81059a90
[    2.535309]  ffff8802322b8000 ffff8802322b7a98 ffffffff81054dbf ffffffffa00378e8
[    2.535830]  ffffffff00000000 ffff880200000000 ffffffff81e88c80 00000000322b7a38
[    2.536090] Call Trace:
[    2.536621]  [<ffffffff812a5d40>] dump_stack+0x46/0x58
[    2.536895]  [<ffffffff81059a90>] ? console_unlock+0x365/0x394
[    2.537163]  [<ffffffff81054dbf>] __lock_acquire+0xc50/0xe10
[    2.537433]  [<ffffffff81053b02>] ? mark_lock+0x2e/0x226
[    2.537748]  [<ffffffffa01803b4>] ? list_create.constprop.8+0x89/0x89 [zfs]
[    2.538014]  [<ffffffff8105527b>] lock_acquire+0x82/0x98
[    2.538318]  [<ffffffffa01829e8>] ? zio_add_child+0x109/0x246 [zfs]
[    2.538585]  [<ffffffff812a7bd2>] mutex_lock_nested+0x51/0x366
[    2.538890]  [<ffffffffa01829e8>] ? zio_add_child+0x109/0x246 [zfs]
[    2.539202]  [<ffffffffa01829e8>] ? zio_add_child+0x109/0x246 [zfs]
[    2.539473]  [<ffffffffa0029852>] ? spl_kmem_cache_alloc+0x9e2/0x9f4 [spl]
[    2.539779]  [<ffffffffa01829e8>] zio_add_child+0x109/0x246 [zfs]
[    2.540084]  [<ffffffffa01803b4>] ? list_create.constprop.8+0x89/0x89 [zfs]
[    2.540389]  [<ffffffffa0182f9f>] zio_create+0x47a/0x498 [zfs]
[    2.540693]  [<ffffffffa0183ae9>] zio_read_phys+0x139/0x14c [zfs]
[    2.540997]  [<ffffffffa013c35f>] ? vdev_accessible+0x86/0x86 [zfs]
[    2.541296]  [<ffffffffa013a700>] vdev_probe+0x268/0x297 [zfs]
[    2.541598]  [<ffffffffa013c35f>] ? vdev_accessible+0x86/0x86 [zfs]
[    2.541900]  [<ffffffffa013e2ec>] ? vdev_set_state+0x8d/0x1ae [zfs]
[    2.542201]  [<ffffffffa013e8c5>] vdev_open+0x4b8/0x523 [zfs]
[    2.542471]  [<ffffffff812aa5f5>] ? _raw_spin_unlock_irqrestore+0x3a/0x41
[    2.542771]  [<ffffffffa013ea20>] vdev_open_child+0x1e/0x33 [zfs]
[    2.543042]  [<ffffffffa002baa2>] taskq_thread+0x2ba/0x3d0 [spl]
[    2.543312]  [<ffffffff81047104>] ? wake_up_state+0xd/0xd
[    2.543585]  [<ffffffffa002b7e8>] ? taskq_dispatch_delay+0x18b/0x18b [spl]
[    2.543853]  [<ffffffff81040cb8>] kthread+0xb5/0xbd
[    2.544120]  [<ffffffff81040c03>] ? __kthread_parkme+0x5c/0x5c
[    2.544389]  [<ffffffff812aadec>] ret_from_fork+0x7c/0xb0
[    2.544655]  [<ffffffff81040c03>] ? __kthread_parkme+0x5c/0x5c
[    2.704003] Switched to clocksource tsc
[    5.395332] SPL: using hostid 0x00000000
[    5.470726]  zd0: unknown partition table
[    5.545638] EXT4-fs (sda1): re-mounted. Opts: discard
[    5.588552] EXT4-fs (sda1): re-mounted. Opts: discard
[    5.599187] EXT4-fs (sda2): mounting with "discard" option, but the device does not support discard
[    5.615776] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: discard
[    5.681199] EXT4-fs (sda3): mounting with "discard" option, but the device does not support discard
[    5.695661] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts: discard
[    8.818895] ip_tables: (C) 2000-2006 Netfilter Core Team
[    8.827828] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   10.171756] vmxnet3 0000:0b:00.0 eth0: intr type 3, mode 0, 5 vectors allocated
[   10.173728] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
[   30.814776] random: nonblocking pool is initialized

and rsync:

[   55.407720] VERIFY3(zap->zap_u.zap_fat.zap_phys->zap_magic == 0x2F52AB2ABULL) failed (0 == 12703150763)
[   55.432473] PANIC at zap.c:580:zap_deref_leaf()
[   55.444735] Showing stack for process 603
[   55.444738] CPU: 2 PID: 603 Comm: rsync Tainted: G           O   3.18.0 #1
[   55.444740]  0000000000000244 ffff8802334f79d8 ffffffff812a5d40 ffffffff81059a90
[   55.444742]  ffffffffa01d5bc2 ffff8802334f79e8 ffffffffa002da0d ffff8802334f7b68
[   55.444747]  ffffffffa002dab6 ffff8800b8993960 ffff880200000030 ffff8802334f7b78
[   55.444751] Call Trace:
[   55.444757]  [<ffffffff812a5d40>] dump_stack+0x46/0x58
[   55.444762]  [<ffffffff81059a90>] ? console_unlock+0x365/0x394
[   55.444772]  [<ffffffffa002da0d>] spl_dumpstack+0x3d/0x3f [spl]
[   55.444777]  [<ffffffffa002dab6>] spl_panic+0xa7/0xda [spl]
[   55.444825]  [<ffffffffa014f83d>] ? zap_lockdir+0x45c/0x6c0 [zfs]
[   55.444828]  [<ffffffff81051fd1>] ? arch_local_irq_save+0x9/0xc
[   55.444865]  [<ffffffffa014a504>] zap_deref_leaf+0x9a/0x14e [zfs]
[   55.444904]  [<ffffffffa014f83d>] ? zap_lockdir+0x45c/0x6c0 [zfs]
[   55.444942]  [<ffffffffa014e419>] ? spl_rw_clear_owner+0x2d/0x32 [zfs]
[   55.444978]  [<ffffffffa014c01e>] fzap_cursor_retrieve+0x7e/0x242 [zfs]
[   55.445019]  [<ffffffffa014fb91>] zap_cursor_retrieve+0xf0/0x26f [zfs]
[   55.445022]  [<ffffffff810517d5>] ? up_read+0x23/0x27
[   55.445045]  [<ffffffffa00e3846>] ? dmu_prefetch+0xcc/0x1da [zfs]
[   55.445085]  [<ffffffffa016dd18>] zfs_readdir+0x1fb/0x3b0 [zfs]
[   55.445088]  [<ffffffff810a4396>] ? handle_mm_fault+0x3be/0x726
[   55.445091]  [<ffffffff812aa582>] ? _raw_spin_unlock+0x23/0x27
[   55.445094]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[   55.445096]  [<ffffffff81053f02>] ? trace_hardirqs_on+0xd/0xf
[   55.445098]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[   55.445099]  [<ffffffff810cb9ab>] ? iterate_dir+0x3b/0x105
[   55.445141]  [<ffffffffa018b65b>] zpl_iterate+0x40/0x80 [zfs]
[   55.445143]  [<ffffffff810cb9e3>] iterate_dir+0x73/0x105
[   55.445145]  [<ffffffff810cbc33>] SyS_getdents+0x76/0xd2
[   55.445147]  [<ffffffff810cba75>] ? iterate_dir+0x105/0x105
[   55.445149]  [<ffffffff810a4536>] ? handle_mm_fault+0x55e/0x726
[   55.445151]  [<ffffffff812aae92>] system_call_fastpath+0x12/0x17

@ryao
Copy link
Contributor

ryao commented Feb 26, 2015

This appears to be a duplicate of #1445.

@behlendorf
Copy link
Contributor

Closing as a duplicate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@behlendorf @umask @ryao @Boldie @TioNisla