most I/O hang #610

Closed
stephane-chazelas opened this Issue Mar 19, 2012 · 22 comments

Projects

None yet

6 participants

@stephane-chazelas

v0.6.0.53 on debian 3.0.0-16 amd64

First time occurrence of this kind of problem.

Today, I've got a bunch of processes hung in "D" state (though the kernel doesn't report the tasks being hung).

Most I/Os to zfs (raidz1, compress, dedup) hang (system calls don't return) except for a few files (possible cached).

There is disk activity. dstat report heavy read access to all 6 drives in the zpool, though no end-user application seems to be doing any I/O. No write I/O to the disks.

  TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO    COMMAND
  743 be/7 root        145.47 M      0.00 B  0.00 %  0.00 % [z_rd_int/11]
  736 be/7 root        141.00 M      0.00 B  0.00 %  0.00 % [z_rd_int/4]
  742 be/7 root        139.42 M      0.00 B  0.00 %  0.00 % [z_rd_int/10]
  735 be/7 root        139.37 M      0.00 B  0.00 %  0.00 % [z_rd_int/3]
  738 be/7 root        138.70 M      0.00 B  0.00 %  0.00 % [z_rd_int/6]
  740 be/7 root        138.11 M      0.00 B  0.00 %  0.00 % [z_rd_int/8]
  737 be/7 root        138.11 M      0.00 B  0.00 %  0.00 % [z_rd_int/5]
  739 be/7 root        138.03 M      0.00 B  0.00 %  0.00 % [z_rd_int/7]
  741 be/7 root        134.40 M      0.00 B  0.00 %  0.00 % [z_rd_int/9]
  733 be/7 root        133.91 M      0.00 B  0.00 %  0.00 % [z_rd_int/1]
  734 be/7 root        133.38 M      0.00 B  0.00 %  0.00 % [z_rd_int/2]
  732 be/7 root        125.90 M      0.00 B  0.00 %  0.00 % [z_rd_int/0]
  744 be/7 root          5.29 M      0.00 B  0.00 %  0.00 % [z_wr_iss/0]
  752 be/7 root          2.73 M      0.00 B  0.00 %  0.00 % [z_wr_iss/8]
  751 be/7 root          2.44 M      0.00 B  0.00 %  0.00 % [z_wr_iss/7]
  754 be/7 root          2.33 M      0.00 B  0.00 %  0.00 % [z_wr_iss/10]
  748 be/7 root       1822.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/4]
  727 be/7 root       1820.50 K      0.00 B  0.00 %  0.00 % [z_rd_iss/3]
  730 be/7 root       1758.50 K      0.00 B  0.00 %  0.00 % [z_rd_iss/6]
  728 be/7 root       1735.50 K      0.00 B  0.00 %  0.00 % [z_rd_iss/4]
  729 be/7 root       1727.00 K      0.00 B  0.00 %  0.00 % [z_rd_iss/5]
  753 be/7 root       1626.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/9]
  749 be/7 root       1620.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/5]
  726 be/7 root       1579.00 K      0.00 B  0.00 %  0.00 % [z_rd_iss/2]
  731 be/7 root       1555.00 K      0.00 B  0.00 %  0.00 % [z_rd_iss/7]
  725 be/7 root       1521.50 K      0.00 B  0.00 %  0.00 % [z_rd_iss/1]
  724 be/7 root       1401.00 K      0.00 B  0.00 %  0.00 % [z_rd_iss/0]
  745 be/7 root       1088.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/1]
  755 be/7 root        785.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/11]
  747 be/7 root        698.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/3]
  750 be/7 root        669.50 K      0.00 B  0.00 %  0.00 % [z_wr_iss/6]
  776 be/7 root        202.50 K      0.00 B  0.00 %  0.00 % [z_wr_int/15]
30906 be/4 root        200.00 K      0.00 B  0.00 %  0.18 % [kworker/0:0]
27852 be/4 root         94.50 K      0.00 B  0.00 %  0.00 % rsync 

No write I/O spotted.

The above sorted by cumulative read I/O over a period of time. I've been stracing 27852 all along, and no system call ever returned (it's been killed (SIGTERM) long before that but hasn't terminated yet)

There's no special zfs operation like removal of volumes ongoing.

a sysrq-w shows

[246742.383485] z_wr_iss/11     D ffffffff81805120     0   755      2 0x00000000
[246742.383485]  ffff880207511ae0 0000000000000046 00ff880200000000 ffff88041fab0600
[246742.383485]  ffff880207511fd8 ffff880207511fd8 ffff880207511fd8 0000000000012a40
[246742.383485]  ffff880213149720 ffff880207509720 ffff880207511af0 ffff88040fc1f818
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020941f>] space_map_load_wait+0x2f/0x40 [zfs]
[246742.383485]  [<ffffffffa01f2627>] metaslab_activate+0x77/0x160 [zfs]
[246742.383485]  [<ffffffffa01f3ec8>] metaslab_alloc+0x638/0xa40 [zfs]
[246742.383485]  [<ffffffff81153fac>] ? __kmalloc+0x11c/0x160
[246742.383485]  [<ffffffffa0109d8b>] ? kmem_alloc_debug+0x1eb/0x350 [spl]
[246742.383485]  [<ffffffffa024bec2>] zio_dva_allocate+0x82/0x400 [zfs]
[246742.383485]  [<ffffffffa02486c4>] ? zio_buf_free+0x24/0x30 [zfs]
[246742.383485]  [<ffffffffa024e120>] ? zio_checksum_compute+0xd0/0x160 [zfs]
[246742.383485]  [<ffffffffa024953f>] zio_execute+0x9f/0xf0 [zfs]
[246742.383485]  [<ffffffffa010e0fb>] taskq_thread+0x24b/0x5a0 [spl]
[246742.383485]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[246742.383485]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[246742.383485]  [<ffffffffa010deb0>] ? task_done+0x140/0x140 [spl]
[246742.383485]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[246742.383485]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[246742.383485]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[246742.383485]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13

for all the z_wr_iss threads.

[246742.383485] txg_sync        D 0000000000000003     0   822      2 0x00000000
[246742.383485]  ffff88020f5e1bd0 0000000000000046 ffff88020f5e1b70 ffffffff810574c2
[246742.383485]  ffff88020f5e1fd8 ffff88020f5e1fd8 ffff88020f5e1fd8 0000000000012a40
[246742.383485]  ffff8802074e4560 ffff8802075e4560 ffff88020f5e1be0 ffff880194834368
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff810574c2>] ? default_wake_function+0x12/0x20
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa024bc9b>] zio_wait+0xfb/0x170 [zfs]
[246742.383485]  [<ffffffffa01e724a>] dsl_pool_sync+0xca/0x450 [zfs]
[246742.383485]  [<ffffffffa01faf3e>] spa_sync+0x38e/0xa00 [zfs]
[246742.383485]  [<ffffffffa020b006>] txg_sync_thread+0x286/0x450 [zfs]
[246742.383485]  [<ffffffffa020ad80>] ? txg_init+0x260/0x260 [zfs]
[246742.383485]  [<ffffffffa010cf88>] thread_generic_wrapper+0x78/0x90 [spl]
[246742.383485]  [<ffffffffa010cf10>] ? __thread_create+0x310/0x310 [spl]
[246742.383485]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[246742.383485]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[246742.383485]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[246742.383485]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13

and for blocked userspace application:

[246742.383485] rsync           D ffffffff81805120     0 20292  20288 0x00000000
[246742.383485]  ffff8801f28adb58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff8801f28adfd8 ffff8801f28adfd8 ffff8801f28adfd8 0000000000012a40
[246742.383485]  ffff8802130b9720 ffff8801dd6a9720 ffff8801f28adb68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8104cb7e>] ? perf_event_task_sched_out+0x2e/0xa0
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27794  27790 0x00000004
[246742.383485]  ffff88038f5f3b58 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff88038f5f3fd8 ffff88038f5f3fd8 ffff88038f5f3fd8 0000000000012a40
[246742.383485]  ffff8802130ddc80 ffff88038b031720 ffff88038f5f3b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8117bc7b>] ? core_sys_select+0x2db/0x330
[246742.383485]  [<ffffffff814db9dd>] ? sock_aio_read+0x2d/0x40
[246742.383485]  [<ffffffff81167ec2>] ? do_sync_read+0xd2/0x110
[246742.383485]  [<ffffffff811ff61c>] ? free_rb_tree_fname+0x5c/0xb0
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27828  27823 0x00000000
[246742.383485]  ffff880390877b58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff880390877fd8 ffff880390877fd8 ffff880390877fd8 0000000000012a40
[246742.383485]  ffffffff81c0b020 ffff8803b3e08000 ffff880390877b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8103e7a8>] ? flush_tlb_page+0x48/0xb0
[246742.383485]  [<ffffffff8103d47c>] ? ptep_set_access_flags+0x6c/0x70
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27847  27842 0x00000004
[246742.383485]  ffff88011eef3b58 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff88011eef3fd8 ffff88011eef3fd8 ffff88011eef3fd8 0000000000012a40
[246742.383485]  ffff88021310dc80 ffff880206401720 ffff88011eef3b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffffa01099cb>] ? kmem_free_debug+0x4b/0x150 [spl]
[246742.383485]  [<ffffffffa0220170>] ? zfs_acl_node_free+0x30/0x40 [zfs]
[246742.383485]  [<ffffffff8160772e>] ? _raw_spin_lock+0xe/0x20
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27851  27847 0x00000000
[246742.383485]  ffff8803ff1e3b78 0000000000000082 0000000000000010 000000d01fffcc08
[246742.383485]  ffff8803ff1e3fd8 ffff8803ff1e3fd8 ffff8803ff1e3fd8 0000000000012a40
[246742.383485]  ffff8802130f4560 ffff880401bc2e40 ffff8803ff1e3b88 ffff880318c5e638
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa0233764>] zfs_range_lock+0x294/0x5c0 [zfs]
[246742.383485]  [<ffffffffa023dc2f>] zfs_write+0x5af/0xc50 [zfs]
[246742.383485]  [<ffffffff8115f340>] ? __mem_cgroup_commit_charge+0x70/0xc0
[246742.383485]  [<ffffffff81161d7e>] ? mem_cgroup_charge_common+0xae/0xe0
[246742.383485]  [<ffffffff811167ed>] ? lru_cache_add_lru+0x2d/0x50
[246742.383485]  [<ffffffff8113956d>] ? page_add_new_anon_rmap+0x9d/0xb0
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27852  27828 0x00000004
[246742.383485]  ffff8801c6b15b58 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff8801c6b15fd8 ffff8801c6b15fd8 ffff8801c6b15fd8 0000000000012a40
[246742.383485]  ffff8802130d8000 ffff88020ccb5c80 ffff8801c6b15b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8104cb7e>] ? perf_event_task_sched_out+0x2e/0xa0
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D 0000000000000007     0 27858  27853 0x00000000
[246742.383485]  ffff8801287bfb78 0000000000000086 0000000000000010 000000d0811c2a8c
[246742.383485]  ffff8801287bffd8 ffff8801287bffd8 ffff8801287bffd8 0000000000012a40
[246742.383485]  ffff8801de0a5c80 ffff8801ef95dc80 ffff8801287bfb88 ffff8801056f2238
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa0233764>] zfs_range_lock+0x294/0x5c0 [zfs]
[246742.383485]  [<ffffffffa023dc2f>] zfs_write+0x5af/0xc50 [zfs]
[246742.383485]  [<ffffffff8110bfb7>] ? filemap_fault+0x87/0x3c0
[246742.383485]  [<ffffffff810720ce>] ? complete_signal+0xee/0x220
[246742.383485]  [<ffffffff81073296>] ? do_send_sig_info+0x66/0x90
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff81608a6e>] ? do_device_not_available+0xe/0x10
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27861  27858 0x00000004
[246742.383485]  ffff880199427b58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff880199427fd8 ffff880199427fd8 ffff880199427fd8 0000000000012a40
[246742.383485]  ffff88021310dc80 ffff88020eb21720 ffff880199427b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff81167db2>] ? do_sync_write+0xd2/0x110
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff81608a6e>] ? do_device_not_available+0xe/0x10
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 27973  27968 0x00000000
[246742.383485]  ffff88011a503b58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff88011a503fd8 ffff88011a503fd8 ffff88011a503fd8 0000000000012a40
[246742.383485]  ffffffff81c0b020 ffff8801de0a5c80 ffff88011a503b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8103e7a8>] ? flush_tlb_page+0x48/0xb0
[246742.383485]  [<ffffffff8103d47c>] ? ptep_set_access_flags+0x6c/0x70
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 28009      1 0x00000004
[246742.383485]  ffff88012bad3a88 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff88012bad3fd8 ffff88012bad3fd8 ffff88012bad3fd8 0000000000012a40
[246742.383485]  ffff88021310c560 ffff880206619720 ffff88012bad3a98 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023f102>] zfs_rename+0x412/0xde0 [zfs]
[246742.383485]  [<ffffffffa0115da0>] ? tsd_exit+0x1a0/0x1d0 [spl]
[246742.383485]  [<ffffffffa02506be>] zpl_rename+0x5e/0x90 [zfs]
[246742.383485]  [<ffffffff811745a6>] vfs_rename_other+0xd6/0x120
[246742.383485]  [<ffffffff8117547b>] vfs_rename+0xbb/0x240
[246742.383485]  [<ffffffff81178798>] sys_renameat+0x218/0x240
[246742.383485]  [<ffffffff8160772e>] ? _raw_spin_lock+0xe/0x20
[246742.383485]  [<ffffffff8118576e>] ? vfsmount_lock_local_unlock+0x1e/0x30
[246742.383485]  [<ffffffff811875b0>] ? mntput_no_expire+0x30/0xf0
[246742.383485]  [<ffffffff81172b72>] ? path_put+0x22/0x30
[246742.383485]  [<ffffffff81167474>] ? sys_fchmodat+0x64/0x100
[246742.383485]  [<ffffffff811787db>] sys_rename+0x1b/0x20
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 28010  27973 0x00000004
[246742.383485]  ffff88031febfb58 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff88031febffd8 ffff88031febffd8 ffff88031febffd8 0000000000012a40
[246742.383485]  ffff8802130b9720 ffff8803254a9720 ffff88031febfb68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 28955  28950 0x00000000
[246742.383485]  ffff88031fc73b78 0000000000000086 0000000000000010 000000d0811c2a8c
[246742.383485]  ffff88031fc73fd8 ffff88031fc73fd8 ffff88031fc73fd8 0000000000012a40
[246742.383485]  ffff8802130b9720 ffff8803647e4560 ffff88031fc73b88 ffff88031df06338
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa0233764>] zfs_range_lock+0x294/0x5c0 [zfs]
[246742.383485]  [<ffffffffa023dc2f>] zfs_write+0x5af/0xc50 [zfs]
[246742.383485]  [<ffffffff8110bfb7>] ? filemap_fault+0x87/0x3c0
[246742.383485]  [<ffffffff811098fa>] ? unlock_page+0x2a/0x40
[246742.383485]  [<ffffffff8112c4d9>] ? __do_fault+0x439/0x510
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff81608a6e>] ? do_device_not_available+0xe/0x10
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 28966  28955 0x00000004
[246742.383485]  ffff88011978bb58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff88011978bfd8 ffff88011978bfd8 ffff88011978bfd8 0000000000012a40
[246742.383485]  ffff88021314dc80 ffff880203ac0000 ffff88011978bb68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8110bfb7>] ? filemap_fault+0x87/0x3c0
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] sb-backup       D ffffffff81805120     0 30169  28976 0x00000000
[246742.383485]  ffff8803a16ebb58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff8803a16ebfd8 ffff8803a16ebfd8 ffff8803a16ebfd8 0000000000012a40
[246742.383485]  ffffffff81c0b020 ffff88038329dc80 ffff8803a16ebb68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8110bfb7>] ? filemap_fault+0x87/0x3c0
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D 0000000000000007     0 30195  30190 0x00000004
[246742.383485]  ffff880387855b58 0000000000000086 0000000000000000 0000000000000000
[246742.383485]  ffff880387855fd8 ffff880387855fd8 ffff880387855fd8 0000000000012a40
[246742.383485]  ffff8802074c0000 ffff880383298000 ffff880387855b68 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023d9f7>] zfs_write+0x377/0xc50 [zfs]
[246742.383485]  [<ffffffff8115f340>] ? __mem_cgroup_commit_charge+0x70/0xc0
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff8118295c>] ? destroy_inode+0x3c/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] rsync           D ffffffff81805120     0 30198  30195 0x00000000
[246742.383485]  ffff8802ffc9db78 0000000000000086 0000000000000010 000000d01fffbc08
[246742.383485]  ffff8802ffc9dfd8 ffff8802ffc9dfd8 ffff8802ffc9dfd8 0000000000012a40
[246742.383485]  ffff8802130ac560 ffff88040b2c2e40 ffff8802ffc9db88 ffff88040e0b8c38
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa0233764>] zfs_range_lock+0x294/0x5c0 [zfs]
[246742.383485]  [<ffffffffa023dc2f>] zfs_write+0x5af/0xc50 [zfs]
[246742.383485]  [<ffffffff8110bfb7>] ? filemap_fault+0x87/0x3c0
[246742.383485]  [<ffffffff811098fa>] ? unlock_page+0x2a/0x40
[246742.383485]  [<ffffffff8112c4d9>] ? __do_fault+0x439/0x510
[246742.383485]  [<ffffffff8112fcc8>] ? handle_mm_fault+0x1f8/0x350
[246742.383485]  [<ffffffff810734a6>] ? group_send_sig_info+0x56/0x70
[246742.383485]  [<ffffffffa02501c2>] zpl_write_common+0x52/0x80 [zfs]
[246742.383485]  [<ffffffffa0250258>] zpl_write+0x68/0xa0 [zfs]
[246742.383485]  [<ffffffff81168563>] vfs_write+0xb3/0x180
[246742.383485]  [<ffffffff8116888a>] sys_write+0x4a/0x90
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
[246742.383485] cp              D ffffffff81805120     0 30999  27556 0x00000000
[246742.383485]  ffff88039ed5fa78 0000000000000082 0000000000000000 0000000000000000
[246742.383485]  ffff88039ed5ffd8 ffff88039ed5ffd8 ffff88039ed5ffd8 0000000000012a40
[246742.383485]  ffff8802130ddc80 ffff8803b109c560 ffff88039ed5fa88 ffff880205ea0b18
[246742.383485] Call Trace:
[246742.383485]  [<ffffffff816058cf>] schedule+0x3f/0x60
[246742.383485]  [<ffffffffa01146b8>] cv_wait_common+0x98/0x190 [spl]
[246742.383485]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[246742.383485]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[246742.383485]  [<ffffffffa01147e3>] __cv_wait+0x13/0x20 [spl]
[246742.383485]  [<ffffffffa020a1cb>] txg_wait_open+0x8b/0x110 [zfs]
[246742.383485]  [<ffffffffa01cff8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[246742.383485]  [<ffffffffa023c688>] zfs_create+0x2a8/0x6d0 [zfs]
[246742.383485]  [<ffffffffa0250b87>] zpl_create+0xa7/0xe0 [zfs]
[246742.383485]  [<ffffffff811734b3>] ? generic_permission+0x23/0xd0
[246742.383485]  [<ffffffff811747c4>] vfs_create+0xb4/0x120
[246742.383485]  [<ffffffff81176639>] do_last+0x649/0x730
[246742.383485]  [<ffffffff811778aa>] path_openat+0xca/0x3f0
[246742.383485]  [<ffffffff81177cd2>] do_filp_open+0x42/0xa0
[246742.383485]  [<ffffffff812f4cc1>] ? strncpy_from_user+0x31/0x40
[246742.383485]  [<ffffffff8117307a>] ? do_getname+0x10a/0x180
[246742.383485]  [<ffffffff8160772e>] ? _raw_spin_lock+0xe/0x20
[246742.383485]  [<ffffffff81184f87>] ? alloc_fd+0xf7/0x150
[246742.383485]  [<ffffffff8116795d>] do_sys_open+0xed/0x220
[246742.383485]  [<ffffffff81167ab0>] sys_open+0x20/0x30
[246742.383485]  [<ffffffff8160fb82>] system_call_fastpath+0x16/0x1b
@behlendorf
Member

Thanks for the additional debugging. It appears the following is happening (why I'm not sure)

  • The rsync process is blocked waiting for free space in the next transaction group.
  • The current transaction group is in the process of being sync'ed out and has issued multiple writes.
  • One of those writes required that a new space map be loaded from disk and it's waiting for it to be read.
  • For some reason that space map can't be read from disk yet isn't returning an error code. Everything is blocked waiting for this I/O to complete.

It would be interested to see both zpool events and zpool status to see if there are any issues with the drives.

@stephane-chazelas

Thanks Brian,

Could be a regression introduced by a recent change (I'm on the daily ppa) because it never happened before, and after a reboot, the system is back into that state.

The other change (appart from the upgrade from ppa 51 to ppa 53 (now 54) is me changing zfs_arc_max from 2GiB to 4GiB (on a system with 16GiB of RAM).

cetautomatix# zpool events
TIME CLASS
Mar 19 2012 15:18:37.978286000 resource.fs.zfs.statechange
Mar 19 2012 15:18:37.978318000 resource.fs.zfs.statechange
Mar 19 2012 15:18:37.978321000 resource.fs.zfs.statechange
Mar 19 2012 15:18:37.978504000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.057657000 resource.fs.zfs.statechange
Mar 19 2012 15:18:37.978218000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262498000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262595000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262634000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262639000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262642000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.262397000 resource.fs.zfs.statechange
Mar 19 2012 15:18:38.445541000 resource.fs.zfs.statechange
Mar 19 2012 15:20:54.149555000 ereport.fs.zfs.vdev.unknown
Mar 19 2012 15:20:54.198074000 ereport.fs.zfs.zpool
cetautomatix:# date
Tue Mar 20 08:59:16 GMT 2012
cetautomatix:
# uptime
09:00:20 up 17:42, 3 users, load average: 73.23, 73.07, 72.99
cetautomatix:# ps -el | grep -wc D
73
cetautomatix:
# zpool status
pool: backup
state: ONLINE
scan: none requested
config:

    NAME                                    STATE     READ WRITE CKSUM
    backup                                  ONLINE       0     0     0
      raidz1-0                              ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1BZ702845  ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1BZ702848  ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1MZ102084  ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1MZ102094  ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1MZ102099  ONLINE       0     0     0
        ata-SAMSUNG_HD203WI_S1UYJ1MZ102100  ONLINE       0     0     0
    spares
      ata-SAMSUNG_HD203WI_S1UYJDWZ710112    AVAIL

errors: No known data errors


I'll try and revert to ppa 51 for now.
@stephane-chazelas

This may be different this time, the zvol tasks are in D state as well, so is
irqbalance (even though it has not business with zfs) and there are some "hung
tasks notices" and there's not been any I/O to the disks since around 6am:

S WIDE-WCHAN-COLUMN COMMAND
D cv_wait_common    kswapd0
D cv_wait_common    kswapd1
D cv_wait_common    zvol/0
D cv_wait_common    zvol/1
D cv_wait_common    zvol/2
D cv_wait_common    zvol/3
D cv_wait_common    zvol/4
D cv_wait_common    zvol/5
D cv_wait_common    zvol/6
D cv_wait_common    zvol/7
D cv_wait_common    zvol/8
D cv_wait_common    zvol/9
D cv_wait_common    zvol/10
D cv_wait_common    zvol/11
D cv_wait_common    zvol/12
D cv_wait_common    zvol/13
D cv_wait_common    zvol/14
D cv_wait_common    zvol/15
D cv_wait_common    zvol/16
D cv_wait_common    zvol/17
D cv_wait_common    zvol/18
D cv_wait_common    zvol/19
D cv_wait_common    zvol/20
D cv_wait_common    zvol/21
D cv_wait_common    zvol/22
D cv_wait_common    zvol/23
D cv_wait_common    zvol/24
D cv_wait_common    zvol/25
D cv_wait_common    zvol/26
D cv_wait_common    zvol/27
D cv_wait_common    zvol/28
D cv_wait_common    zvol/29
D cv_wait_common    zvol/30
D cv_wait_common    zvol/31
D cv_wait_common    zvol/32
D cv_wait_common    zvol/33
D cv_wait_common    zvol/34
D cv_wait_common    zvol/35
D metaslab_alloc    z_wr_iss/0
D metaslab_alloc    z_wr_iss/1
D metaslab_alloc    z_wr_iss/2
D metaslab_alloc    z_wr_iss/3
D metaslab_alloc    z_wr_iss/4
D metaslab_alloc    z_wr_iss/5
D metaslab_alloc    z_wr_iss/6
D metaslab_alloc    z_wr_iss/7
D metaslab_alloc    z_wr_iss/8
D metaslab_alloc    z_wr_iss/9
D metaslab_alloc    z_wr_iss/10
D metaslab_alloc    z_wr_iss/11
D cv_wait_common    txg_sync
D cv_wait_common    irqbalance
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    ntfsclone
D cv_wait_common    ntfsclone
D cv_wait_common    ntfsclone
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    flush-zfs-134
D cv_wait_common    ssh
D cv_wait_common    rsync
D cv_wait_common    flush-zfs-137
D sleep_on_buffer   jbd2/zd4432-8
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    rsync
D get_request_wait  flush-230:4432
D cv_wait_common    rsync
D cv_wait_common    rsync
D cv_wait_common    flush-zfs-126

(811 is zfs_iput_taskq)

cetautomatix login: [22682.376094] INFO: task zfs_iput_taskq/:811 blocked for more than 120 seconds.
[22682.383306] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.391729] INFO: task txg_sync:815 blocked for more than 120 seconds.
[22682.398300] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.406512] INFO: task rsync:24093 blocked for more than 120 seconds.
[22682.413000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.421141] INFO: task rsync:24157 blocked for more than 120 seconds.
[22682.427633] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.435776] INFO: task rsync:24176 blocked for more than 120 seconds.
[22682.442264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.450394] INFO: task rsync:24223 blocked for more than 120 seconds.
[22682.456881] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.465007] INFO: task rsync:24242 blocked for more than 120 seconds.
[22682.471491] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.479618] INFO: task ntfsclone:24592 blocked for more than 120 seconds.
[22682.486450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.494583] INFO: task ntfsclone:24712 blocked for more than 120 seconds.
[22682.501413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[22682.509554] INFO: task rsync:24726 blocked for more than 120 seconds.
[22682.516039] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[63835.051794] SysRq : Changing Loglevel
[63835.055498] Loglevel set to 9
[64594.333388] SysRq : Show Blocked State
[64594.337065]   task                        PC stack   pid father
[64594.337065] kswapd0         D 0000000000000000     0    80      2 0x00000000
[64594.337065]  ffff88020fa3f8d0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88020fa3ffd8 ffff88020fa3ffd8 ffff88020fa3ffd8 0000000000012a40
[64594.337065]  ffff88040d2a0000 ffff8804134e4560 ffff88020fa3f8e0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa01c0fc4>] dmu_free_long_range_impl+0x184/0x270 [zfs]
[64594.337065]  [<ffffffffa01c2dbb>] dmu_free_long_range+0x4b/0x70 [zfs]
[64594.337065]  [<ffffffffa0221790>] zfs_rmnode+0x60/0x340 [zfs]
[64594.337065]  [<ffffffffa0240ac7>] zfs_zinactive+0x87/0xd0 [zfs]
[64594.337065]  [<ffffffffa023bb8e>] zfs_inactive+0x5e/0x1c0 [zfs]
[64594.337065]  [<ffffffff811176b9>] ? truncate_pagecache+0x59/0x70
[64594.337065]  [<ffffffffa024def8>] zpl_evict_inode+0x28/0x30 [zfs]
[64594.337065]  [<ffffffff81182a21>] evict+0x91/0x170
[64594.337065]  [<ffffffff81182b3e>] dispose_list+0x3e/0x50
[64594.337065]  [<ffffffff81182eaf>] prune_icache+0x16f/0x320
[64594.337065]  [<ffffffff81183081>] shrink_icache_memory+0x21/0x50
[64594.337065]  [<ffffffff8111b1dd>] shrink_slab+0x11d/0x190
[64594.337065]  [<ffffffff8111e42a>] balance_pgdat+0x52a/0x6f0
[64594.337065]  [<ffffffff8111e6da>] kswapd+0xea/0x1f0
[64594.337065]  [<ffffffff8111e5f0>] ? balance_pgdat+0x6f0/0x6f0
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] kswapd1         D ffffffff81805120     0    81      2 0x00000000
[64594.337065]  ffff88020f9d58d0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88020f9d5fd8 ffff88020f9d5fd8 ffff88020f9d5fd8 0000000000012a40
[64594.337065]  ffff88021316dc80 ffff88021324dc80 ffff88020f9d58e0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa01c0fc4>] dmu_free_long_range_impl+0x184/0x270 [zfs]
[64594.337065]  [<ffffffffa01c2dbb>] dmu_free_long_range+0x4b/0x70 [zfs]
[64594.337065]  [<ffffffffa0221790>] zfs_rmnode+0x60/0x340 [zfs]
[64594.337065]  [<ffffffffa0240ac7>] zfs_zinactive+0x87/0xd0 [zfs]
[64594.337065]  [<ffffffffa023bb8e>] zfs_inactive+0x5e/0x1c0 [zfs]
[64594.337065]  [<ffffffff811176b9>] ? truncate_pagecache+0x59/0x70
[64594.337065]  [<ffffffffa024def8>] zpl_evict_inode+0x28/0x30 [zfs]
[64594.337065]  [<ffffffff81182a21>] evict+0x91/0x170
[64594.337065]  [<ffffffff81182b3e>] dispose_list+0x3e/0x50
[64594.337065]  [<ffffffff81182eaf>] prune_icache+0x16f/0x320
[64594.337065]  [<ffffffff81183081>] shrink_icache_memory+0x21/0x50
[64594.337065]  [<ffffffff8111b1dd>] shrink_slab+0x11d/0x190
[64594.337065]  [<ffffffff8111e42a>] balance_pgdat+0x52a/0x6f0
[64594.337065]  [<ffffffff8111e6da>] kswapd+0xea/0x1f0
[64594.337065]  [<ffffffff8111e5f0>] ? balance_pgdat+0x6f0/0x6f0
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/0          D 0000000000000000     0   582      2 0x00000000
[64594.337065]  ffff8804121fdbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804121fdfd8 ffff8804121fdfd8 ffff8804121fdfd8 0000000000012a40
[64594.337065]  ffff88040d351720 ffff880412271720 ffff8804121fdbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/1          D ffffffff81805120     0   583      2 0x00000000
[64594.337065]  ffff8804121ffbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804121fffd8 ffff8804121fffd8 ffff8804121fffd8 0000000000012a40
[64594.337065]  ffff8802130bc560 ffff880412272e40 ffff8804121ffbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/2          D 0000000000000002     0   584      2 0x00000000
[64594.337065]  ffff88041063dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041063dfd8 ffff88041063dfd8 ffff88041063dfd8 0000000000012a40
[64594.337065]  ffff88040d36ae40 ffff880412274560 ffff88041063dbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/3          D 0000000000000003     0   585      2 0x00000000
[64594.337065]  ffff88041063fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041063ffd8 ffff88041063ffd8 ffff88041063ffd8 0000000000012a40
[64594.337065]  ffff88040d322e40 ffff880412275c80 ffff88041063fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/4          D ffffffff81805120     0   586      2 0x00000000
[64594.337065]  ffff880410051bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410051fd8 ffff880410051fd8 ffff880410051fd8 0000000000012a40
[64594.337065]  ffff880213115c80 ffff880412278000 ffff880410051bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/5          D 0000000000000005     0   587      2 0x00000000
[64594.337065]  ffff880410053bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410053fd8 ffff880410053fd8 ffff880410053fd8 0000000000012a40
[64594.337065]  ffff880410270000 ffff880412279720 ffff880410053bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/6          D 0000000000000006     0   588      2 0x00000000
[64594.337065]  ffff880410105bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410105fd8 ffff880410105fd8 ffff880410105fd8 0000000000012a40
[64594.337065]  ffff88040d338000 ffff88041227ae40 ffff880410105bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/7          D 0000000000000007     0   589      2 0x00000000
[64594.337065]  ffff880410107bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410107fd8 ffff880410107fd8 ffff880410107fd8 0000000000012a40
[64594.337065]  ffff880410272e40 ffff88041227c560 ffff880410107bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/8          D 0000000000000008     0   590      2 0x00000000
[64594.337065]  ffff88041026dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041026dfd8 ffff88041026dfd8 ffff88041026dfd8 0000000000012a40
[64594.337065]  ffff88040d33ae40 ffff88041227dc80 ffff88041026dba0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/9          D 0000000000000009     0   591      2 0x00000000
[64594.337065]  ffff88041026fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041026ffd8 ffff88041026ffd8 ffff88041026ffd8 0000000000012a40
[64594.337065]  ffff88040d2b5c80 ffff8804106e0000 ffff88041026fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/10         D 000000000000000a     0   592      2 0x00000000
[64594.337065]  ffff880410305bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410305fd8 ffff880410305fd8 ffff880410305fd8 0000000000012a40
[64594.337065]  ffff88040d33dc80 ffff8804106e1720 ffff880410305bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/11         D 000000000000000b     0   593      2 0x00000000
[64594.337065]  ffff880410307bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410307fd8 ffff880410307fd8 ffff880410307fd8 0000000000012a40
[64594.337065]  ffff880410311720 ffff8804106e2e40 ffff880410307bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/12         D ffffffff81805120     0   594      2 0x00000000
[64594.337065]  ffff88040fc81bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88040fc81fd8 ffff88040fc81fd8 ffff88040fc81fd8 0000000000012a40
[64594.337065]  ffffffff81c0b020 ffff8804106e4560 ffff88040fc81bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/13         D 0000000000000001     0   595      2 0x00000000
[64594.337065]  ffff88040fc83bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88040fc83fd8 ffff88040fc83fd8 ffff88040fc83fd8 0000000000012a40
[64594.337065]  ffff88040d320000 ffff8804106e5c80 ffff88040fc83bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/14         D 0000000000000002     0   596      2 0x00000000
[64594.337065]  ffff8804126e5bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804126e5fd8 ffff8804126e5fd8 ffff8804126e5fd8 0000000000012a40
[64594.337065]  ffff88040d321720 ffff880410028000 ffff8804126e5bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/15         D 0000000000000003     0   597      2 0x00000000
[64594.337065]  ffff8804126e7bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804126e7fd8 ffff8804126e7fd8 ffff8804126e7fd8 0000000000012a40
[64594.337065]  ffff88040d2a4560 ffff880410029720 ffff8804126e7bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/16         D 0000000000000004     0   598      2 0x00000000
[64594.337065]  ffff88040fc8dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88040fc8dfd8 ffff88040fc8dfd8 ffff88040fc8dfd8 0000000000012a40
[64594.337065]  ffff880412278000 ffff88041002ae40 ffff88040fc8dbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/17         D 0000000000000005     0   599      2 0x00000000
[64594.337065]  ffff88040fc8fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88040fc8ffd8 ffff88040fc8ffd8 ffff88040fc8ffd8 0000000000012a40
[64594.337065]  ffff88040d325c80 ffff88041002c560 ffff88040fc8fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/18         D 0000000000000006     0   600      2 0x00000000
[64594.337065]  ffff880410031bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410031fd8 ffff880410031fd8 ffff880410031fd8 0000000000012a40
[64594.337065]  ffff88040d2b1720 ffff88041002dc80 ffff880410031bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/19         D 0000000000000007     0   601      2 0x00000000
[64594.337065]  ffff880410033bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410033fd8 ffff880410033fd8 ffff880410033fd8 0000000000012a40
[64594.337065]  ffff88041227c560 ffff880410040000 ffff880410033bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/20         D 0000000000000008     0   602      2 0x00000000
[64594.337065]  ffff880410049bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410049fd8 ffff880410049fd8 ffff880410049fd8 0000000000012a40
[64594.337065]  ffff88040d33ae40 ffff880410041720 ffff880410049bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/21         D 0000000000000009     0   603      2 0x00000000
[64594.337065]  ffff88041004bbc0 0000000000000046 ffff8802938ac580 0000000000000044
[64594.337065]  ffff88041004bfd8 ffff88041004bfd8 ffff88041004bfd8 0000000000012a40
[64594.337065]  ffff88040d2f9720 ffff880410042e40 ffff88041004bba0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/22         D 000000000000000a     0   604      2 0x00000000
[64594.337065]  ffff88041004dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041004dfd8 ffff88041004dfd8 ffff88041004dfd8 0000000000012a40
[64594.337065]  ffff88040d33dc80 ffff880410044560 ffff88041004dbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/23         D ffffffff81805120     0   605      2 0x00000000
[64594.337065]  ffff88041004fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041004ffd8 ffff88041004ffd8 ffff88041004ffd8 0000000000012a40
[64594.337065]  ffff880213169720 ffff880410045c80 ffff88041004fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/24         D 0000000000000000     0   606      2 0x00000000
[64594.337065]  ffff880410119bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410119fd8 ffff880410119fd8 ffff880410119fd8 0000000000012a40
[64594.337065]  ffff88040d3f1720 ffff880410110000 ffff880410119bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/25         D 0000000000000001     0   607      2 0x00000000
[64594.337065]  ffff88041011bbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041011bfd8 ffff88041011bfd8 ffff88041011bfd8 0000000000012a40
[64594.337065]  ffff88040d352e40 ffff880410111720 ffff88041011bbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/26         D 0000000000000002     0   608      2 0x00000000
[64594.337065]  ffff88041011dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041011dfd8 ffff88041011dfd8 ffff88041011dfd8 0000000000012a40
[64594.337065]  ffff88040d321720 ffff880410112e40 ffff88041011dbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/27         D 0000000000000003     0   609      2 0x00000000
[64594.337065]  ffff88041011fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041011ffd8 ffff88041011ffd8 ffff88041011ffd8 0000000000012a40
[64594.337065]  ffff88040d2a4560 ffff880410114560 ffff88041011fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/28         D 0000000000000004     0   610      2 0x00000000
[64594.337065]  ffff8804101a9bc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804101a9fd8 ffff8804101a9fd8 ffff8804101a9fd8 0000000000012a40
[64594.337065]  ffff88040d2e9720 ffff880410115c80 ffff8804101a9bd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/29         D ffffffff81805120     0   611      2 0x00000000
[64594.337065]  ffff8804101abbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804101abfd8 ffff8804101abfd8 ffff8804101abfd8 0000000000012a40
[64594.337065]  ffff88021312c560 ffff880410270000 ffff8804101abbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/30         D ffffffff81805120     0   612      2 0x00000000
[64594.337065]  ffff8804101adbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804101adfd8 ffff8804101adfd8 ffff8804101adfd8 0000000000012a40
[64594.337065]  ffff880213142e40 ffff880410271720 ffff8804101adbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/31         D 0000000000000007     0   613      2 0x00000000
[64594.337065]  ffff8804101afbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff8804101affd8 ffff8804101affd8 ffff8804101affd8 0000000000012a40
[64594.337065]  ffff88040d339720 ffff880410272e40 ffff8804101afbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/32         D ffffffff81805120     0   614      2 0x00000000
[64594.337065]  ffff880410279060 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff880410279fd8 ffff880410279fd8 ffff880410279fd8 0000000000012a40
[64594.337065]  ffff880213144560 ffff880410274560 ffff880410279070 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa01c0fc4>] dmu_free_long_range_impl+0x184/0x270 [zfs]
[64594.337065]  [<ffffffffa01c2dbb>] dmu_free_long_range+0x4b/0x70 [zfs]
[64594.337065]  [<ffffffffa0221790>] zfs_rmnode+0x60/0x340 [zfs]
[64594.337065]  [<ffffffffa0240ac7>] zfs_zinactive+0x87/0xd0 [zfs]
[64594.337065]  [<ffffffffa023bb8e>] zfs_inactive+0x5e/0x1c0 [zfs]
[64594.337065]  [<ffffffff811176b9>] ? truncate_pagecache+0x59/0x70
[64594.337065]  [<ffffffffa024def8>] zpl_evict_inode+0x28/0x30 [zfs]
[64594.337065]  [<ffffffff81182a21>] evict+0x91/0x170
[64594.337065]  [<ffffffff81182b3e>] dispose_list+0x3e/0x50
[64594.337065]  [<ffffffff81182eaf>] prune_icache+0x16f/0x320
[64594.337065]  [<ffffffff81183081>] shrink_icache_memory+0x21/0x50
[64594.337065]  [<ffffffff8111b1dd>] shrink_slab+0x11d/0x190
[64594.337065]  [<ffffffff8111d978>] do_try_to_free_pages+0x288/0x2e0
[64594.337065]  [<ffffffff8111dcd4>] try_to_free_pages+0xb4/0x180
[64594.337065]  [<ffffffff81113260>] __alloc_pages_nodemask+0x4e0/0x800
[64594.337065]  [<ffffffff81149273>] alloc_pages_current+0xa3/0x110
[64594.337065]  [<ffffffff81151f5d>] allocate_slab+0xfd/0x160
[64594.337065]  [<ffffffffa01cf2aa>] ? dmu_zfetch+0x2da/0xde0 [zfs]
[64594.337065]  [<ffffffff81151ff0>] new_slab+0x30/0x160
[64594.337065]  [<ffffffff815f5299>] __slab_alloc+0x127/0x222
[64594.337065]  [<ffffffffa0106e03>] ? kmem_alloc_debug+0x263/0x350 [spl]
[64594.337065]  [<ffffffff81153fc2>] __kmalloc+0x132/0x160
[64594.337065]  [<ffffffffa0106e03>] ? kmem_alloc_debug+0x263/0x350 [spl]
[64594.337065]  [<ffffffffa0106e03>] kmem_alloc_debug+0x263/0x350 [spl]
[64594.337065]  [<ffffffffa012d18b>] ? avl_find+0x5b/0xb0 [zavl]
[64594.337065]  [<ffffffffa0205a75>] space_map_remove+0x185/0x340 [zfs]
[64594.337065]  [<ffffffffa01c253c>] ? dmu_read+0x12c/0x180 [zfs]
[64594.337065]  [<ffffffffa02065d7>] space_map_load+0x1a7/0x390 [zfs]
[64594.337065]  [<ffffffffa01ef68b>] metaslab_activate+0xdb/0x160 [zfs]
[64594.337065]  [<ffffffffa01f0ec8>] metaslab_alloc+0x638/0xa40 [zfs]
[64594.337065]  [<ffffffffa01b0f18>] ? remove_reference.isra.8+0xa8/0xd0 [zfs]
[64594.337065]  [<ffffffff811536df>] ? kfree+0x3f/0x150
[64594.337065]  [<ffffffff811536df>] ? kfree+0x3f/0x150
[64594.337065]  [<ffffffff81153fac>] ? __kmalloc+0x11c/0x160
[64594.337065]  [<ffffffffa0249ea3>] zio_alloc_zil+0x63/0x180 [zfs]
[64594.337065]  [<ffffffffa024378a>] zil_lwb_write_start+0x1ba/0x350 [zfs]
[64594.337065]  [<ffffffffa02440ab>] zil_commit+0x1fb/0x6a0 [zfs]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffffa0250152>] zvol_write+0x172/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/33         D 0000000000000009     0   615      2 0x00000000
[64594.337065]  ffff88041027bbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041027bfd8 ffff88041027bfd8 ffff88041027bfd8 0000000000012a40
[64594.337065]  ffff88040d2f9720 ffff880410275c80 ffff88041027bbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/34         D 000000000000000a     0   616      2 0x00000000
[64594.337065]  ffff88041027dbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041027dfd8 ffff88041027dfd8 ffff88041027dfd8 0000000000012a40
[64594.337065]  ffff88040d2fae40 ffff880410310000 ffff88041027dbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64594.337065]  [<ffffffff8104e8c9>] ? finish_task_switch+0x49/0xf0
[64594.337065]  [<ffffffff810574b0>] ? try_to_wake_up+0x200/0x200
[64594.337065]  [<ffffffffa010aeb0>] ? task_done+0x140/0x140 [spl]
[64594.337065]  [<ffffffff81080dac>] kthread+0x8c/0xa0
[64594.337065]  [<ffffffff81610ca4>] kernel_thread_helper+0x4/0x10
[64594.337065]  [<ffffffff81080d20>] ? flush_kthread_worker+0xa0/0xa0
[64594.337065]  [<ffffffff81610ca0>] ? gs_change+0x13/0x13
[64594.337065] zvol/35         D 000000000000000b     0   617      2 0x00000000
[64594.337065]  ffff88041027fbc0 0000000000000046 0000000000000000 0000000000000000
[64594.337065]  ffff88041027ffd8 ffff88041027ffd8 ffff88041027ffd8 0000000000012a40
[64594.337065]  ffff88040d350000 ffff880410311720 ffff88041027fbd0 ffff88041278ab18
[64594.337065] Call Trace:
[64594.337065]  [<ffffffff816058cf>] schedule+0x3f/0x60
[64594.337065]  [<ffffffffa01116b8>] cv_wait_common+0x98/0x190 [spl]
[64594.337065]  [<ffffffff810466f3>] ? __wake_up+0x53/0x70
[64594.337065]  [<ffffffff81081850>] ? add_wait_queue+0x60/0x60
[64594.337065]  [<ffffffffa01117e3>] __cv_wait+0x13/0x20 [spl]
[64594.337065]  [<ffffffffa02071cb>] txg_wait_open+0x8b/0x110 [zfs]
[64594.337065]  [<ffffffffa01ccf8d>] dmu_tx_wait+0xed/0xf0 [zfs]
[64594.337065]  [<ffffffffa01cd016>] dmu_tx_assign+0x86/0x460 [zfs]
[64594.337065]  [<ffffffffa0250082>] zvol_write+0xa2/0x400 [zfs]
[64594.337065]  [<ffffffff81032a69>] ? default_spin_lock_flags+0x9/0x10
[64594.337065]  [<ffffffffa010b0fb>] taskq_thread+0x24b/0x5a0 [spl]
[64
@stephane-chazelas

Sorry, I'm all confused now. There were disk events. I'm not sure how I missed those from a number of disks. Things like:

Mar 18 15:59:06 cetautomatix kernel: [166490.736046] sd 0:0:9:0: attempting task abort! scmd(ffff880151f84a00)
Mar 18 15:59:06 cetautomatix kernel: [166490.736051] sd 0:0:9:0: [sdj] CDB: Read(10): 28 00 d8 07 7d d3 00 00 43 00
Mar 18 15:59:06 cetautomatix kernel: [166490.736060] scsi target0:0:9: handle(0x0014), sas_address(0x500065b36789abf1), phy(17)
Mar 18 15:59:06 cetautomatix kernel: [166490.736062] scsi target0:0:9: enclosure_logical_id(0x500065b37689abff), slot(9)

(same for sdc, sdd, sdh at different times)

Followed by

Mar 18 15:59:28 cetautomatix kernel: [166512.689756] ereport.fs.zfs.delay ena= 0xf4745afb4593bd9c pool_context=0x0 pool_failmode="wait" vdev_guid=
Mar 18 15:59:28 cetautomatix kernel: [166512.689769]  0x5365c0760896fcaa vdev_type="disk" vdev_path=
Mar 18 15:59:28 cetautomatix kernel: [166512.689777] 0x77141b8d5c502c01 detector=[ version= "/dev/disk/by-id/ata-SAMSUNG_HD203WI_S1UYJ1MZ102100-part1" parent_guid=
Mar 18 15:59:28 cetautomatix kernel: [166512.689787]  0x16e8fae68cf8d787 parent_type="raidz"0x0 scheme="zfs"
Mar 18 15:59:28 cetautomatix kernel: [166512.689796]  pool=0xf4745afb4593bd9c vdev=0x5365c0760896fcaa zio_err=0x0 zio_flags=0x604c0 ] pool="backup" pool_guid=
Mar 18 15:59:28 cetautomatix kernel: [166512.689811]  0xf4745afb4593bd9c pool_context=0x0 pool_failmode="wait" vdev_guid=
Mar 18 15:59:28 cetautomatix kernel: [166512.689819]  0x5365c0760896fcaa vdev_type="disk" vdev_path=
Mar 18 15:59:28 cetautomatix kernel: [166512.689831]

Even though the disks don't match (SAMSUNG_HD203WI_S1UYJ1MZ102100 is sdg, not sdj

Today, for that new occurrence, there's no disk error though.

@behlendorf
Member

The first batch of errors are clearly read errors which would go a long way to explain why zfs was unable to read back the required space maps. Or perhaps why it was just taking so long since the second zfs ereport.fs.zfs.delay indicates that the read I/Os were completing successfully (zio_err=0x0) after at least 30 seconds.

Today's occurrence may be more of the same, the zvol threads are all stuck of a txg again. You'll want to check what is blocking the txg_sync_thread.

As for a regression there's nothing which has recently been merged in to the zfs code which I can see would cause this sort of problem. I would strongly suspect you may be having some hardware trouble.

@stephane-chazelas

For today's occurrence, no disk error. I've reverted to 51. We'll see if I still get the issue. If yes, I'll revert the zfs_arc_max and see how it goes.

@stephane-chazelas

Reverted to 51. Now, those nightly backup went a lot further (this is a backup server backing up data from about 15 other machines, and 80% went through, while the last two days we got stuck after none got through). We're stuck again, but not completely. No disk error.

Most applications (mostly rsync) are "stuck", there's a lot of reading from the disks (5MB/s) mostly from z_rd_int's. Some applications occasionaly manage to read some data. Only one application (ntfsclone onto a zvol, patched to only write modified clusters) manages to occasionally write data). It should be noted that it's the one application that is writing to some area that is not dedupped (zvol with volblocksize=4k).

Here's a iotop -a ouput run over a few minutes, sorted by accumulated read:

Total DISK READ: 763.05 K/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO    COMMAND
  754 be/7 root        212.20 M      0.00 B  0.00 %  0.00 % [z_rd_int/3]
  753 be/7 root        211.36 M      0.00 B  0.00 %  0.00 % [z_rd_int/2]
  759 be/7 root        211.33 M      0.00 B  0.00 %  0.00 % [z_rd_int/8]
  762 be/7 root        210.98 M      0.00 B  0.00 %  0.00 % [z_rd_int/11]
  761 be/7 root        210.12 M      0.00 B  0.00 %  0.00 % [z_rd_int/10]
  760 be/7 root        210.02 M      0.00 B  0.00 %  0.00 % [z_rd_int/9]
  755 be/7 root        206.56 M      0.00 B  0.00 %  0.00 % [z_rd_int/4]
  756 be/7 root        205.95 M      0.00 B  0.00 %  0.00 % [z_rd_int/5]
  751 be/7 root        202.77 M      0.00 B  0.00 %  0.00 % [z_rd_int/0]
  758 be/7 root        201.47 M      0.00 B  0.00 %  0.00 % [z_rd_int/7]
  752 be/7 root        198.28 M      0.00 B  0.00 %  0.00 % [z_rd_int/1]
  757 be/7 root        196.27 M      0.00 B  0.00 %  0.00 % [z_rd_int/6]
39208 be/4 root         46.45 M     15.29 M  0.00 %  4.95 % ntfsclone -O zvol /dev/nbd11
  770 be/7 root          9.55 M      0.00 B  0.00 %  0.00 % [z_wr_iss/7]
  769 be/7 root          8.89 M      0.00 B  0.00 %  0.00 % [z_wr_iss/6]
  773 be/7 root          6.85 M      0.00 B  0.00 %  0.00 % [z_wr_iss/10]
  774 be/7 root          6.84 M      0.00 B  0.00 %  0.00 % [z_wr_iss/11]
  763 be/7 root          6.72 M      0.00 B  0.00 %  0.00 % [z_wr_iss/0]
  772 be/7 root          4.71 M      0.00 B  0.00 %  0.00 % [z_wr_iss/9]
51864 be/4 root          4.36 M      0.00 B  0.00 %  7.33 % mdadm --examine --scan --config=partitions
  764 be/7 root          2.54 M      0.00 B  0.00 %  0.00 % [z_wr_iss/1]
  748 be/7 root          2.26 M      0.00 B  0.00 %  0.00 % [z_rd_iss/5]
  747 be/7 root          2.21 M      0.00 B  0.00 %  0.00 % [z_rd_iss/4]
  746 be/7 root          2.16 M      0.00 B  0.00 %  0.00 % [z_rd_iss/3]
  750 be/7 root          2.15 M      0.00 B  0.00 %  0.00 % [z_rd_iss/7]
  768 be/7 root          2.10 M      0.00 B  0.00 %  0.00 % [z_wr_iss/5]
  749 be/7 root          2.08 M      0.00 B  0.00 %  0.00 % [z_rd_iss/6]
  745 be/7 root          2.05 M      0.00 B  0.00 %  0.00 % [z_rd_iss/2]
  744 be/7 root          2.00 M      0.00 B  0.00 %  0.00 % [z_rd_iss/1]
  743 be/7 root       1742.00 K      0.00 B  0.00 %  0.00 % [z_rd_iss/0]
  630 be/7 root       1452.50 K      0.00 B  0.00 %  0.00 % [zvol/22]
  782 be/7 root        412.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/2]
  790 be/7 root        368.50 K      0.00 B  0.00 %  0.00 % [z_wr_int/10]
40950 be/4 root        295.50 K      0.00 B  0.00 %  0.00 % rsync --archive --acls --xattrs --hard-links --delete --numeric-i~se --stats eti-server-r415:/tmp/sb-backup-tmp-Z21pMK/xvda1/ data/
  793 be/7 root        272.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/13]
  788 be/7 root        247.50 K      0.00 B  0.00 %  0.00 % [z_wr_int/8]
  787 be/7 root        202.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/7]
  766 be/7 root        191.50 K      0.00 B  0.00 %  0.00 % [z_wr_iss/3]
  765 be/7 root        179.00 K      0.00 B  0.00 %  0.00 % [z_wr_iss/2]
  785 be/7 root        158.50 K      0.00 B  0.00 %  0.00 % [z_wr_int/5]
  792 be/7 root        153.50 K      0.00 B  0.00 %  0.00 % [z_wr_int/12]
  636 be/7 root        139.00 K      0.00 B  0.00 %  0.00 % [zvol/28]
  789 be/7 root        135.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/9]
  637 be/7 root        127.00 K      0.00 B  0.00 %  0.00 % [zvol/29]
  781 be/7 root        121.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/1]
  795 be/7 root        115.00 K      0.00 B  0.00 %  0.00 % [z_wr_int/15]
  608 be/7 root        111.00 K      0.00 B  0.00 %  0.00 % [zvol/0]

The zpool is getting pretty full, maybe that's part of the problem.

@behlendorf
Member

Can you try 0.6.0.54 with the following patch applied, behlendorf/zfs@1813cc0 . This fixes a slight defect in the zio handling which was accidentally introduced in 0.6.0.53. I expect to be merging it in to master shortly and it may fix your issue.

Oh, and it would be more likely to cause trouble on a full pool when logical IOs need to be constructed due to a lack of free space.

@stephane-chazelas

Sorry Brian, I'm just back from vacation. I've updated to ppa-daily 55 which seems to have that patch and will keep you posted.

@stephane-chazelas

Nope, no luck with 55 either. All rsync processes blocked, only ntfsclone ones (writing on non-dedup zvols) occasionaly manage to get anything written to the pool.

One processor (not always the same) has some iowait time:

top - 11:13:36 up 18:01, 10 users,  load average: 40.10, 40.50, 37.63
Tasks: 601 total,   2 running, 595 sleeping,   3 stopped,   1 zombie
Cpu0  :  0.7%us,  2.3%sy,  0.0%ni, 93.4%id,  3.3%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.7%us,  0.3%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  1.3%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  7.6%sy,  0.0%ni, 92.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.2%sy,  0.0%ni, 24.9%id, 74.8%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.7%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16451520k total, 12627496k used,  3824024k free,  6660024k buffers
Swap: 33554428k total,        0k used, 33554428k free,   292980k cached

Another thing I notice (but it's the same with 51) is that there are a lot of zil_clean processes:

ps -eo state,wchan:30,comm | sed 's,/.*,,' | sort | uniq -c | sort -n | tail
     12 S run_ksoftirqd                  ksoftirqd
     12 S taskq_thread                   spl_system_task
     12 S taskq_thread                   tx_commit_cb
     12 S taskq_thread                   z_fr_iss
     14 D cv_wait_common                 rsync
     16 S taskq_thread                   z_wr_int
     28 S worker_thread                  kworker
     34 S taskq_thread                   zvol
     84 S wait                           sb-backup
    163 S taskq_thread                   zil_clean

Which is not far from the number of mounted zfs datasets:

mount | grep -c 'type zfs'
158

There still is a lot of reading from the disks.

----system---- -net/total- ---load-avg--- --dsk/sdb-----dsk/sdc-----dsk/sdd-----dsk/sde-----dsk/sdh-----dsk/sdi-----dsk/sdj----dsk/total-
     time     | recv  send| 1m   5m  15m | read  writ: read  writ: read  writ: read  writ: read  writ: read  writ: read  writ: read  writ
27-03 09:52:29|   0     0 |37.2 40.1 38.7|  25B    0 : 454k   35k: 538k   37k: 536k   37k: 528k   37k: 450k   35k: 447k   35k:3158k  220k
27-03 10:02:29|  45k 2966B|38.8 41.4 39.9|   0     0 : 436k   20k: 521k   22k: 522k   22k: 514k   22k: 431k   20k: 423k   20k:2871k  141k
27-03 10:12:29|  12k 2140B|36.9 35.7 37.5|   0     0 : 397k 8753B: 478k 9813B: 474k 9786B: 466k 9847B: 389k 8804B: 384k 8738B:2593k   55k
27-03 10:22:29|  20k 3386B|31.6 31.6 34.3|   0     0 : 449k 8639B: 576k 9481B: 564k 9496B: 558k 9357B: 442k 8546B: 439k 8708B:3043k   58k
27-03 10:32:29|  12k 4998B|37.8 37.2 35.7|   0     0 : 461k 6990B: 597k 7793B: 589k 7834B: 582k 7792B: 459k 6970B: 454k 7070B:3147k   45k
27-03 10:42:29|  56k 4052B|31.4 33.5 34.4|   0     0 : 452k   19k: 565k   20k: 556k   20k: 551k   20k: 447k   19k: 444k   19k:3048k  127k
27-03 10:52:29|  11k 2218B|33.6 33.9 34.4|   0     0 : 455k 7451B: 559k 8035B: 556k 8081B: 546k 8132B: 447k 7469B: 442k 7399B:3011k   50k
27-03 11:02:29|  18k 5231B|36.6 34.0 33.9|   0     0 : 451k   11k: 554k   12k: 546k   12k: 538k   12k: 441k   12k: 440k   12k:2983k   80k
27-03 11:12:29|6205B 2215B|41.0 40.7 37.5|   0     0 : 477k 6692B: 608k 7245B: 596k 7219B: 587k 7179B: 471k 6629B: 471k 6653B:3219k   46k

I'll boot back on 51.

@stephane-chazelas

Forget about the "regression" aspect. Same happened this morning with 51.

@stephane-chazelas

Good it have something to do with the size of the dedup table or something like that?

@stephane-chazelas

Now, I've removed 2 old snapshots (of a 6TB sparse compressed but not dedupped zvol) which freed 600GB (going from 330GB free to 988GB on that 10TB zpool) and it seems to have kicked it out of that stuck state.

@behlendorf
Member

Thanks for the update, perhaps it's simply related to available free space. Once in that state you will be using different aspects on the code base more, such as gang blocks, which are no doubt less tested. I haven't made any changes in this regard for the Linux port, but certainly even the upstream code is known to behave badly when your pool is very full.

@ryao
Member
ryao commented Apr 19, 2012

@stephane-chazelas It looks like this might be a preemption glitch. Is your kernel compiled with CONFIG_PREEMPT_VOLUNTARY=y?

@stephane-chazelas
$ grep CONFIG_PREEMPT_VOLUNTARY /boot/config-$(uname -r)
# CONFIG_PREEMPT_VOLUNTARY is not set

Nope. Note that it only happened when the zpool usage was over 90%.

@durin42
durin42 commented Jul 27, 2012

I'm also seeing this on the stock Ubuntu kernel. CONFIG_PREEMPT_VOLUNTARY is set to y, and I do have a pool over 90% full. I'll try to move some data around and see if the problem stops happening if I can unload the pool a bit.

@dswartz
Contributor
dswartz commented Oct 18, 2012

I just had a lockup with 12.04 stock ubuntu kernel with what looks like this issue.

CONFIG_PREEMPT_VOLUNTARY=y

I had to back out to using my old opensolaris VM (this is a virtualized SAN/NAS) to avoid hassles. Trace is:

Oct 18 09:30:07 ubuntusan1 kernel: [124637.468979] INFO: task txg_sync:2241 blo
cked for more than 120 seconds.
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469053] "echo 0 > /proc/sys/kernel/h
ung_task_timeout_secs" disables this message.
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469130] txg_sync D 0000000000
000000 0 2241 2 0x00000000
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469132] ffff88050dd19bd0 0000000000
000046 ffff88050dd19b70 ffffffff8105f692
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469135] ffff88050dd19fd8 ffff88050d
d19fd8 ffff88050dd19fd8 00000000000137c0
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469137] ffff880512fb2e00 ffff880512
feae00 ffff88050dd19be0 ffff8802d54f2648
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469139] Call Trace:
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469145] [] ? defa
ult_wake_function+0x12/0x20
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469149] [] schedu
le+0x3f/0x60
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469160] [] cv_wai
t_common+0x98/0x190 [spl]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469163] [] ? add_
wait_queue+0x60/0x60
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469168] [] _cv_w
ait+0x13/0x20 [spl]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469189] [] zio_wa
it+0xfb/0x170 [zfs]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469204] [] dsl_po
ol_sync+0xca/0x450 [zfs]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469219] [] spa_sy
nc+0x38e/0xa00 [zfs]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469235] [] txg_sy
nc_thread+0x286/0x450 [zfs]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469250] [] ? txg

init+0x250/0x250 [zfs]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469254] [] thread
_generic_wrapper+0x78/0x90 [spl]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469258] [] ? __th
read_create+0x310/0x310 [spl]
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469260] [] kthrea
d+0x8c/0xa0
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469262] [] kernel
_thread_helper+0x4/0x10
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469264] [] ? flus
h_kthread_worker+0xa0/0xa0
Oct 18 09:30:07 ubuntusan1 kernel: [124637.469266] [] ? gs_c
hange+0x13/0x13

@behlendorf
Member

Closing as stale. This hasn't been observed is quite a while and there have been numerous fixes since 0.6.0.53 which could have been the root cause for this. If your able to reproduce in the issue with the latest code we'll take another hard look at this.

@behlendorf behlendorf closed this Apr 11, 2013
@akorn
akorn commented May 22, 2013

I think I'm seeing something very similar with 0.6.1-1~wheezy (kernel 3.7.2).

I have a hardware RAID1 array with LUKS (dm-crypt) on it, LVM in that and zfs in that (yes, I know this is bad). The pool has had dedup enabled for some filesystems but I turned it off a while ago. Performance used to be fair if not stellar, but since yesterday I'm seeing huge load spikes (100+).

I once had a pool corrupted on this box; my suspect was the RAID controller, so I disabled write-back caching on it a few months ago.

I tried both the cfq and the deadline i/o scheduler on the RAID to see if it made a difference, but it didn't really (currently it's set to deadline).

The box only has 4G RAM, but the pool is tiny (only 80G, with less than 60 used).

iostat -x reports:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.09    0.00   98.31    1.60    0.00    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00   10.36    0.19   407.91     0.75    77.50     0.27   29.39   29.93    0.00   3.45   3.63

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22.53    0.00   17.75   59.73    0.00    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               4.33     0.00  195.67  135.67   871.83  9486.67    62.53    11.26   33.83   36.03   30.66   2.90  96.13

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.36    0.00   22.92   66.72    0.00    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00  249.67  280.33  1775.00  5228.50    26.43    14.47   25.64   29.75   21.98   1.89 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.42    0.00   22.26   74.32    0.00    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00  349.67  279.00  4813.33  1116.00    18.86    16.31   26.96   30.26   22.83   1.59 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.64    0.00   15.67   77.68    0.00    0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               3.00     0.00  390.33    7.33  2207.67    33.83    11.27    10.20   25.54   25.80   11.68   2.51 100.00

iotop (In accumulated mode, sorted by reads):

Total DISK READ:       6.09 M/s | Total DISK WRITE:      91.23 K/s
  TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO    COMMAND                                                                                                
 9469 be/7 root          7.34 G      0.00 B  0.00 %  0.02 % [z_rd_int/1]
 9468 be/7 root          6.32 G      0.00 B  0.00 %  0.03 % [z_rd_int/0]
 7156 be/3 openvpn      13.96 M    244.00 K  2.54 %  0.06 % openvpn --config /etc/openvpn/vpnusers-udp.conf
13040 be/4 root         12.92 M      0.00 B  0.00 %  0.01 % apache2 -k start

I also have exactly the same amount of zil_clean threads as the number of mounted zfs filesystems (39), fwiw.

At first I thought the problem was maybe that I had too many snapshots (I noticed that the zfs list command zfs-auto-snapshot issued would take more than 30 minutes to complete during these load spikes), so I reduced the number of snapshots from 3200 to about 650, but this didn't really help.

At first the pool was pretty full (around 3G free of 60G), but I gave it an additional 20G and it didn't seem to have any effect.

top, fwiw:

top - 18:07:18 up  4:58,  8 users,  load average: 61.07, 63.20, 64.53
Threads: 919 total,   3 running, 914 sleeping,   0 stopped,   2 zombie
%Cpu0  : 24.8 us, 19.1 sy,  0.0 ni,  0.0 id, 55.7 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  : 21.6 us, 26.5 sy,  0.0 ni,  0.0 id, 48.1 wa,  0.0 hi,  3.9 si,  0.0 st
KiB Mem:   4037688 total,  3768244 used,   269444 free,        0 buffers
KiB Swap:  6291452 total,   499740 used,  5791712 free,    79896 cached

  PID  PR  NI  VIRT  RES CODE  SHR DATA SWAP S   UID nTH %MEM %CPU     TIME+ WCHAN      COMMAND                                                                       
 9508   0 -20     0    0    0    0    0    0 D     0   1  0.0  0.0   0:35.44 cv_wait_c+ [txg_sync]                                                                    
15774  20   0  6692  640 1268  376  768  400 D  1067   1  0.0  0.0   0:00.29 vfs_readd+ /usr/bin/imapd Maildir                                                        
17202  20   0 29336  708   88  232  560    0 D     0   1  0.0  0.0   0:00.19 cv_wait_c+ zfs list -H -t snapshot -S creation -o name                                   
17607  20   0  8312 2464 1268  620 2388  340 D  1092   1  0.1  0.0   0:00.84 cv_wait_c+ /usr/bin/imapd Maildir                                                        
17609  20   0  6744  364 1268  256  820  384 D  1094   1  0.0  0.3   0:11.61 cv_wait_c+ /usr/bin/imapd Maildir                                                        
17850  20   0  6744  368 1268  256  820  380 D  1094   1  0.0  0.0   0:11.37 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18056  20   0  6744  364 1268  256  820  384 D  1094   1  0.0  0.0   0:08.52 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18111  20   0  6744  368 1268  256  820  380 D  1094   1  0.0  0.0   0:07.95 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18264  20   0  7412  568 1268  256 1488  952 D  1094   1  0.0  0.0   0:12.35 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18285  20   0  6744  372 1268  256  820  376 D  1094   1  0.0  0.0   0:07.62 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18411  20   0  6744  744 1268  256  820    0 D  1094   1  0.0  0.0   0:07.28 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18501  20   0  7412 1532 1268  336 1488   72 D  1094   1  0.0  0.0   0:12.46 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18567  20   0  6744  748 1268  256  820    0 D  1094   1  0.0  0.0   0:07.38 cv_wait_c+ /usr/bin/imapd Maildir                                                        
18847  20   0  6744 2032 1268 1540  820    0 D  1094   1  0.1  0.0   0:07.45 cv_wait_c+ /usr/bin/imapd Maildir                                                        
19402  20   0  3304  724 1268  588  488    0 D  1067   1  0.0  0.0   0:00.06 vfs_readd+ /usr/bin/imapd Maildir                                                        
19564  20   0  7384 2812 1268 1528 1460   20 D  1060   1  0.1  0.0   0:00.36 cv_wait_c+ /usr/bin/imapd Maildir                                                        
19588  20   0  3304  724 1268  588  488    0 D  1067   1  0.0  0.0   0:00.08 cv_wait_c+ /usr/bin/imapd Maildir                                                        
19820  20   0  6744 1748 1268 1540  820  284 D  1094   1  0.0  0.0   0:04.37 cv_wait_c+ /usr/bin/imapd Maildir                                                        
19881  20   0  3304  728 1268  588  488    0 D  1067   1  0.0  0.0   0:00.07 cv_wait_c+ /usr/bin/imapd Maildir                                                        
19918  20   0  3304  716 1268  588  488    0 D  1059   1  0.0  0.0   0:00.06 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20051  20   0  6600 2076 1268 1588  676    0 D  1052   1  0.1  0.0   0:00.19 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20059  20   0  6744 1960 1268 1540  820   68 D  1094   1  0.0  0.0   0:04.02 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20073  20   0  3304  664 1268  592  488   56 D  1016   1  0.0  0.0   0:00.02 last       /usr/bin/imapd Maildir                                                        
20079  20   0  3304  664 1268  592  488   56 D  1016   1  0.0  0.0   0:00.03 last       /usr/bin/imapd Maildir                                                        
20116  20   0  3304  732 1268  592  488    0 D  1016   1  0.0  0.0   0:00.02 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20295  20   0  6744 2032 1268 1540  820    0 D  1094   1  0.1  0.3   0:03.87 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20338  20   0  7724 1880   76  304 2188  604 D  1059   1  0.0  0.0   0:23.30 cv_wait_c+ /usr/lib/gamin/gam_server                                                     
20390  20   0  3304  680 1268  584  488   36 D  1016   1  0.0  0.0   0:00.02 last       /usr/bin/imapd Maildir                                                        
20392  20   0  7412 2804 1268 1544 1488    0 D  1094   1  0.1 12.5   0:07.15 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20481  20   0  6744 1644 1268 1540  820  388 D  1094   1  0.0  0.0   0:04.22 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20487  20   0  6780 1944 1268 1584  856  308 D  1067   1  0.0  0.0   0:00.27 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20640  20   0  3304  716 1268  588  488    0 D  1059   1  0.0  0.0   0:00.04 vfs_readd+ /usr/bin/imapd Maildir                                                        
20645  20   0  3304  720 1268  584  488    0 D  1016   1  0.0  0.0   0:00.02 last       /usr/bin/imapd Maildir                                                        
20673  20   0  7412 2812 1268 1544 1488    0 D  1094   1  0.1 12.2   0:07.10 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20738  20   0  6744 2028 1268 1540  820    0 D  1094   1  0.1  0.0   0:03.57 cv_wait_c+ /usr/bin/imapd Maildir                                                        
20769  20   0  3304  708 1268  584  488    0 D  1016   1  0.0  0.0   0:00.01 last       /usr/bin/imapd Maildir                                                        
20777  20   0  3304  720 1268  584  488    0 D  1016   1  0.0  0.0   0:00.02 last       /usr/bin/imapd Maildir                                                        
20791  20   0  3304  708 1268  584  488    0 D  1016   1  0.0  0.0   0:00.01 last       /usr/bin/imapd Maildir                                                        
20913  20   0  6744 2028 1268 1540  820    0 D  1094   1  0.1  0.0   0:03.87 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21046  20   0  6744 2032 1268 1540  820    0 D  1094   1  0.1  0.3   0:03.44 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21184  20   0  3304  720 1268  584  488    0 D  1016   1  0.0  0.0   0:00.01 last       /usr/bin/imapd Maildir                                                        
21201  20   0  3304  712 1268  588  488    0 D  1104   1  0.0  0.0   0:00.01 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21347  20   0  6744 2028 1268 1540  820    0 D  1094   1  0.1  0.3   0:03.40 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21349  20   0  3304  720 1268  592  488    0 D  1096   1  0.0  0.0   0:00.01 vfs_readd+ /usr/bin/imapd Maildir                                                        
21449  20   0  3304  716 1268  588  488    0 D  1067   1  0.0  0.0   0:00.03 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21495  20   0  3304  712 1268  584  488    0 D  1016   1  0.0  0.0   0:00.00 last       /usr/bin/imapd Maildir                                                        
21561  20   0  3304  720 1268  592  488    0 D  1096   1  0.0  0.0   0:00.01 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21599  20   0 66196  20m 1224 2076  51m  34m D   224   1  0.5  0.0   0:00.06 sleep_on_+ amavisd (21599-01)                                                            
21601  20   0 66196  20m 1224 2076  51m  34m D   224   1  0.5  0.0   0:00.07 sleep_on_+ amavisd (21601-01)                                                            
21602  20   0 66196  20m 1224 2080  51m  34m D   224   1  0.5  0.0   0:00.06 sleep_on_+ amavisd (21602-01)                                                            
21603  20   0 66196  20m 1224 2084  51m  34m D   224   1  0.5  0.3   0:00.07 sleep_on_+ amavisd (21603-01)                                                            
21637  20   0  6488 1948 1268 1496  564    0 D  1013   1  0.0  0.0   0:00.02 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21650  20   0  7412 2808 1268 1544 1488    0 D  1094   1  0.1  0.3   0:01.28 cv_wait_c+ /usr/bin/imapd Maildir                                                        
21656  20   0 66196  20m 1224 2084  51m  34m D   224   1  0.5  0.6   0:00.06 sleep_on_+ amavisd (21656-01)                                                            

zpool list:

NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zenc  79.5G  55.3G  24.2G    69%  1.05x  ONLINE  -

zpool status:

zpool status -v
  pool: zenc
 state: ONLINE
  scan: scrub repaired 0 in 1h42m with 0 errors on Sun Dec 23 17:57:51 2012
config: 

        NAME                    STATE     READ WRITE CKSUM
        zenc                    ONLINE       0     0     0
          encrypted-zpool_zenc  ONLINE       0     0     0

errors: No known data errors

zpool events:

IME                           CLASS
May 22 2013 13:08:56.380995000 ereport.fs.zfs.vdev.unknown
May 22 2013 13:08:56.458327000 ereport.fs.zfs.zpool
May 22 2013 13:10:56.147383000 resource.fs.zfs.statechange
May 22 2013 13:10:56.490735000 resource.fs.zfs.statechange
May 22 2013 13:11:02.695009000 ereport.fs.zfs.zpool.destroy
May 22 2013 13:11:02.927138000 ereport.fs.zfs.config.sync
May 22 2013 13:11:06.820314000 resource.fs.zfs.statechange
May 22 2013 13:11:07.544298000 ereport.fs.zfs.config.sync
May 22 2013 13:11:08.026223000 ereport.fs.zfs.config.sync
May 22 2013 13:17:03.416163000 ereport.fs.zfs.delay
May 22 2013 13:17:03.442031000 ereport.fs.zfs.delay
May 22 2013 13:17:03.975039000 ereport.fs.zfs.delay
May 22 2013 13:36:48.631889000 ereport.fs.zfs.delay
May 22 2013 13:36:48.652769000 ereport.fs.zfs.delay
May 22 2013 13:36:50.082867000 ereport.fs.zfs.delay
May 22 2013 13:38:14.376030000 ereport.fs.zfs.delay
May 22 2013 13:38:14.376201000 ereport.fs.zfs.delay
May 22 2013 13:38:14.376251000 ereport.fs.zfs.delay
May 22 2013 13:38:14.376427000 ereport.fs.zfs.delay
May 22 2013 13:38:14.376518000 ereport.fs.zfs.delay
May 22 2013 13:38:14.599854000 ereport.fs.zfs.delay
May 22 2013 13:39:08.510433000 ereport.fs.zfs.delay
May 22 2013 14:09:47.335881000 ereport.fs.zfs.delay
May 22 2013 14:09:47.336076000 ereport.fs.zfs.delay
May 22 2013 14:09:47.336303000 ereport.fs.zfs.delay
May 22 2013 14:09:47.341940000 ereport.fs.zfs.delay
May 22 2013 14:09:47.342333000 ereport.fs.zfs.delay
May 22 2013 14:09:47.347767000 ereport.fs.zfs.delay
May 22 2013 14:09:47.347957000 ereport.fs.zfs.delay
May 22 2013 14:36:42.129504000 ereport.fs.zfs.delay
May 22 2013 14:36:42.146217000 ereport.fs.zfs.delay
May 22 2013 14:36:42.300040000 ereport.fs.zfs.delay
May 22 2013 15:00:20.669609000 ereport.fs.zfs.delay
May 22 2013 15:00:20.669798000 ereport.fs.zfs.delay
May 22 2013 16:14:33.508637000 ereport.fs.zfs.delay
May 22 2013 16:14:33.508928000 ereport.fs.zfs.delay
May 22 2013 16:14:33.517692000 ereport.fs.zfs.delay
May 22 2013 16:14:33.537611000 ereport.fs.zfs.delay
May 22 2013 16:28:36.036940000 ereport.fs.zfs.delay
May 22 2013 16:28:36.037313000 ereport.fs.zfs.delay
May 22 2013 16:30:44.475241000 ereport.fs.zfs.delay
May 22 2013 16:30:44.475498000 ereport.fs.zfs.delay
May 22 2013 16:30:46.812055000 ereport.fs.zfs.delay
May 22 2013 16:34:36.842930000 ereport.fs.zfs.delay
May 22 2013 16:34:36.843120000 ereport.fs.zfs.delay
May 22 2013 16:34:36.843284000 ereport.fs.zfs.delay
May 22 2013 16:34:36.843439000 ereport.fs.zfs.delay
May 22 2013 16:34:36.843501000 ereport.fs.zfs.delay
May 22 2013 16:34:36.843863000 ereport.fs.zfs.delay
May 22 2013 16:41:28.826673000 ereport.fs.zfs.delay
May 22 2013 16:41:28.826927000 ereport.fs.zfs.delay
May 22 2013 17:24:47.232516000 ereport.fs.zfs.config.sync
May 22 2013 17:26:36.406364000 ereport.fs.zfs.config.sync

A typical zfs.delay event looks like this:

May 22 2013 16:41:28.826927000 ereport.fs.zfs.delay
        class = "ereport.fs.zfs.delay"
        ena = 0xb9aa71accc200001
        detector = (embedded nvlist)
                version = 0x0
                scheme = "zfs"
                pool = 0x51c9a01baf965292
                vdev = 0x46852815a6f7ec44
        (end detector)
        pool = "zenc"
        pool_guid = 0x51c9a01baf965292
        pool_context = 0x0
        pool_failmode = "wait"
        vdev_guid = 0x46852815a6f7ec44
        vdev_type = "disk"
        vdev_path = "/dev/mapper/encrypted-zpool_zenc"
        vdev_ashift = 0x9
        parent_guid = 0x51c9a01baf965292
        parent_type = "root"
        zio_err = 0x0
        zio_flags = 0x60440
        zio_stage = 0x100000
        zio_pipeline = 0x170000
        zio_delay = 0xb637
        zio_offset = 0xa01075400
        zio_size = 0x1000
        zio_objset = 0x0
        zio_object = 0x55b3
        zio_level = 0x0
        zio_blkid = 0x2
        time = 0x519cd918 0x3149e798 
@akorn
akorn commented May 22, 2013

sysrq-w, part 1:

SysRq : Show Blocked State
  task                        PC stack   pid father
txg_sync        D ffff88011fd11580     0  9508      2 0x00000000
 ffff88010329fb38 0000000000000046 ffff88010734b760 ffff88010329ffd8
 ffff88010329ffd8 ffff88010329ffd8 ffff88010b96b8e0 ffff88010734b760
 ffff88010329fb38 ffff88010734b760 ffff88011fd11e18 ffff8800499a7530
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffffa0164f60>] ? dmu_objset_sync+0x320/0x320 [zfs]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa01850d9>] dsl_pool_sync+0xd9/0x520 [zfs]
 [<ffffffffa0199b6f>] spa_sync+0x38f/0xa40 [zfs]
 [<ffffffff810771e7>] ? ktime_get_ts+0x47/0xe0
 [<ffffffffa01a9bcf>] txg_sync_thread+0x2df/0x540 [zfs]
 [<ffffffffa01a98f0>] ? txg_thread_wait.isra.2+0x40/0x40 [zfs]
 [<ffffffffa00a5673>] thread_generic_wrapper+0x73/0x90 [spl]
 [<ffffffffa00a5600>] ? __thread_create+0x300/0x300 [spl]
 [<ffffffff81052a8b>] kthread+0xbb/0xc0
 [<ffffffff810529d0>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff814b926c>] ret_from_fork+0x7c/0xb0
 [<ffffffff810529d0>] ? kthread_create_on_node+0x120/0x120
clamd           D ffff88011fc11580     0 20771  11885 0x20020000
 ffff88002c12bc08 0000000000000082 ffff88010696ca60 ffff88002c12bfd8
 ffff88002c12bfd8 ffff88002c12bfd8 ffffffff81814440 ffff88010696ca60
 ffff88002c12bc08 ffff88010696ca60 ffff88011fc11e18 0000000000000082
Call Trace:
 [<ffffffff810bb890>] ? sleep_on_page+0x10/0x10
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffff810bb899>] sleep_on_page_killable+0x9/0x40
 [<ffffffff814b6768>] __wait_on_bit+0x58/0x90
 [<ffffffff810bd9c8>] wait_on_page_bit_killable+0x78/0x80
 [<ffffffff81053150>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff810bda5d>] __lock_page_or_retry+0x8d/0xb0
 [<ffffffff810daf3d>] handle_pte_fault+0x26d/0xab0
 [<ffffffff810dbf77>] handle_mm_fault+0x227/0x2e0
 [<ffffffff8102a669>] __do_page_fault+0x139/0x470
 [<ffffffff8106227a>] ? __dequeue_entity+0x2a/0x50
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff8102a9c9>] do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] page_fault+0x22/0x30
clamd           D ffff88011fc11580     0 20772  11885 0x20020000
 ffff8800882adc08 0000000000000082 ffff8800d9a0d1c0 ffff8800882adfd8
 ffff8800882adfd8 ffff8800882adfd8 ffff88006fffcf40 ffff8800d9a0d1c0
 ffff8800882adc08 ffff8800d9a0d1c0 ffff88011fc11e18 0000000000000082
Call Trace:
 [<ffffffff810bb890>] ? sleep_on_page+0x10/0x10
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffff810bb899>] sleep_on_page_killable+0x9/0x40
 [<ffffffff814b6768>] __wait_on_bit+0x58/0x90
 [<ffffffff810bd9c8>] wait_on_page_bit_killable+0x78/0x80
 [<ffffffff81053150>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff810bda5d>] __lock_page_or_retry+0x8d/0xb0
 [<ffffffff810daf3d>] handle_pte_fault+0x26d/0xab0
 [<ffffffff810dbf77>] handle_mm_fault+0x227/0x2e0
 [<ffffffff8102a669>] __do_page_fault+0x139/0x470
 [<ffffffff8106227a>] ? __dequeue_entity+0x2a/0x50
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff8102a9c9>] do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] page_fault+0x22/0x30
clamd           D ffff88011fc11580     0 20773  11885 0x20020000
 ffff8800bc08fc08 0000000000000082 ffff88006fffcf40 ffff8800bc08ffd8
 ffff8800bc08ffd8 ffff8800bc08ffd8 ffff88010696ca60 ffff88006fffcf40
 ffff8800bc08fc08 ffff88006fffcf40 ffff88011fc11e18 0000000000000082
Call Trace:
 [<ffffffff810bb890>] ? sleep_on_page+0x10/0x10
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffff810bb899>] sleep_on_page_killable+0x9/0x40
 [<ffffffff814b6768>] __wait_on_bit+0x58/0x90
 [<ffffffff810bd9c8>] wait_on_page_bit_killable+0x78/0x80
 [<ffffffff81053150>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff810bda5d>] __lock_page_or_retry+0x8d/0xb0
 [<ffffffff810daf3d>] handle_pte_fault+0x26d/0xab0
 [<ffffffff810dbf77>] handle_mm_fault+0x227/0x2e0
 [<ffffffff8102a669>] __do_page_fault+0x139/0x470
 [<ffffffff8106227a>] ? __dequeue_entity+0x2a/0x50
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff8102a9c9>] do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] page_fault+0x22/0x30
gam_server      D ffff88011fd11580     0 13984  10850 0x20020000
 ffff8800942418e8 0000000000000082 ffff8800ac466f00 ffff880094241fd8
 ffff880094241fd8 ffff880094241fd8 ffff8801070570c0 ffff8800ac466f00
 ffff88007cca6468 ffff8800db7e4918 ffff8800db7e491c ffff8800ac466f00
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffffa01e56bd>] zfs_zget+0xad/0x1c0 [zfs]
 [<ffffffffa00a3118>] ? kmem_alloc_debug+0x128/0x390 [spl]
 [<ffffffffa01c5b9f>] zfs_dirent_lock+0x49f/0x5a0 [zfs]
 [<ffffffffa01c5d26>] zfs_dirlook+0x86/0x2e0 [zfs]
 [<ffffffffa01c2af8>] ? zfs_zaccess+0x98/0x3c0 [zfs]
 [<ffffffffa00add3f>] ? tsd_exit+0x5f/0x2a0 [spl]
 [<ffffffffa01dee91>] zfs_lookup+0x2f1/0x340 [zfs]
 [<ffffffffa01f2f90>] zpl_lookup+0x70/0xd0 [zfs]
 [<ffffffff8110b748>] lookup_real+0x18/0x50
 [<ffffffff8110b993>] __lookup_hash+0x33/0x40
 [<ffffffff814b26ff>] lookup_slow+0x42/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff81148ff0>] ? compat_filldir+0x110/0x110
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffffa01dcba0>] ? zfs_getattr_fast+0xd0/0x150 [zfs]
 [<ffffffff8111fb54>] ? mntput_no_expire+0x44/0x160
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff81107169>] vfs_lstat+0x19/0x20
 [<ffffffff8102fbd5>] sys32_lstat64+0x15/0x30
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff814b7c24>] ? schedule+0x24/0x60
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
gam_server      D ffff88011fc11580     0 15704  10850 0x20020000
 ffff8800691ed6b8 0000000000000082 ffff8800694eb720 ffff8800691edfd8
 ffff8800691edfd8 ffff8800691edfd8 ffff88011a1fb5e0 ffff8800694eb720
 ffff8800691ed6b8 ffff8800694eb720 ffff88011fc11e18 ffff88005f0070f8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0170fd4>] dnode_hold_impl+0x1b4/0x590 [zfs]
 [<ffffffffa0150f50>] ? remove_reference.isra.9+0xa0/0xc0 [zfs]
 [<ffffffffa01713c4>] dnode_hold+0x14/0x20 [zfs]
 [<ffffffffa016163f>] dmu_bonus_hold+0x2f/0x2e0 [zfs]
 [<ffffffffa0193f89>] sa_buf_hold+0x9/0x10 [zfs]
 [<ffffffffa01e56d2>] zfs_zget+0xc2/0x1c0 [zfs]
 [<ffffffffa00a3118>] ? kmem_alloc_debug+0x128/0x390 [spl]
 [<ffffffffa01c5b9f>] zfs_dirent_lock+0x49f/0x5a0 [zfs]
 [<ffffffffa01c5d26>] zfs_dirlook+0x86/0x2e0 [zfs]
 [<ffffffffa01c2af8>] ? zfs_zaccess+0x98/0x3c0 [zfs]
 [<ffffffffa00add3f>] ? tsd_exit+0x5f/0x2a0 [spl]
 [<ffffffffa01dee91>] zfs_lookup+0x2f1/0x340 [zfs]
 [<ffffffffa01f2f90>] zpl_lookup+0x70/0xd0 [zfs]
 [<ffffffff8110b748>] lookup_real+0x18/0x50
 [<ffffffff8110b993>] __lookup_hash+0x33/0x40
 [<ffffffff814b26ff>] lookup_slow+0x42/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff81148ff0>] ? compat_filldir+0x110/0x110
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffffa01dcba0>] ? zfs_getattr_fast+0xd0/0x150 [zfs]
 [<ffffffff8111fb54>] ? mntput_no_expire+0x44/0x160
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff81107169>] vfs_lstat+0x19/0x20
 [<ffffffff8102fbd5>] sys32_lstat64+0x15/0x30
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
gam_server      D ffff88011fd11580     0 20338  10850 0x20020000
 ffff88002780fbe8 0000000000000086 ffff880091f3a080 ffff88002780ffd8
 ffff88002780ffd8 ffff88002780ffd8 ffff8800d8ebf560 ffff880091f3a080
 ffff880090697b00 ffff8800cd13a2a8 ffff8800cd13a2ac ffff880091f3a080
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffffa01dcba0>] ? zfs_getattr_fast+0xd0/0x150 [zfs]
 [<ffffffff8111fb54>] ? mntput_no_expire+0x44/0x160
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff81107169>] vfs_lstat+0x19/0x20
 [<ffffffff8102fbd5>] sys32_lstat64+0x15/0x30
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff814b7c24>] ? schedule+0x24/0x60
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 15774  15773 0x20020000
 ffff88010913bb58 0000000000000082 ffff8800d8e35760 ffff88010913bfd8
 ffff88010913bfd8 ffff88010913bfd8 ffff8800881aaf40 ffff8800d8e35760
 ffff88010913bb58 ffff8800d8e35760 ffff88011fc11e18 ffff8800bd27ecc0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
zfs             D ffff88011fd11580     0 17202  16815 0x00000000
 ffff8800a333b7e8 0000000000000082 ffff880010a9d8e0 ffff8800a333bfd8
 ffff8800a333bfd8 ffff8800a333bfd8 ffff88010b8778e0 ffff880010a9d8e0
 ffff8800a333b7e8 ffff880010a9d8e0 ffff88011fd11e18 ffff88001c7f3e58
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa015a105>] ? dmu_buf_rele+0x25/0x30 [zfs]
 [<ffffffffa015b484>] __dbuf_hold_impl+0x354/0x470 [zfs]
 [<ffffffffa015b61f>] dbuf_hold_impl+0x7f/0xb0 [zfs]
 [<ffffffffa015c4db>] dbuf_hold+0x1b/0x30 [zfs]
 [<ffffffffa0170f6f>] dnode_hold_impl+0x14f/0x590 [zfs]
 [<ffffffffa015ac77>] ? dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa01713c4>] dnode_hold+0x14/0x20 [zfs]
 [<ffffffffa016163f>] dmu_bonus_hold+0x2f/0x2e0 [zfs]
 [<ffffffff810f2769>] ? __kmalloc+0x119/0x120
 [<ffffffffa017e684>] dsl_deadlist_open+0x54/0x160 [zfs]
 [<ffffffffa0161359>] ? dmu_object_info_from_dnode+0x129/0x190 [zfs]
 [<ffffffffa017798c>] dsl_dataset_get_ref+0x29c/0xa00 [zfs]
 [<ffffffffa00a19f6>] ? kmem_free_debug+0x46/0x140 [spl]
 [<ffffffffa0000001>] ? hwmon_device_unregister+0x1/0x60 [hwmon]
 [<ffffffffa017882b>] dsl_dataset_hold_obj+0x1b/0x40 [zfs]
 [<ffffffffa0179d09>] dsl_dataset_stats+0x1d9/0x2f0 [zfs]
 [<ffffffffa0178702>] ? dsl_dataset_fast_stat+0xf2/0x200 [zfs]
 [<ffffffffa0164a8a>] dmu_objset_stats+0x1a/0x50 [zfs]
 [<ffffffffa01cbfd6>] zfs_ioc_objset_stats_impl+0x66/0xf0 [zfs]
 [<ffffffffa01cdfb7>] zfs_ioc_snapshot_list_next+0x307/0x340 [zfs]
 [<ffffffffa01d15f7>] zfsdev_ioctl+0xd7/0x1b0 [zfs]
 [<ffffffff81113a5b>] do_vfs_ioctl+0x8b/0x500
 [<ffffffff810df3de>] ? remove_vma+0x4e/0x60
 [<ffffffff810e0b98>] ? do_munmap+0x318/0x3f0
 [<ffffffff81113f61>] sys_ioctl+0x91/0xb0
 [<ffffffff814b9316>] system_call_fastpath+0x1a/0x1f
imapd           D ffff88011fc11580     0 17213  10850 0x20020000
 ffff88000b8d1838 0000000000000086 ffff880071252100 ffff88000b8d1fd8
 ffff88000b8d1fd8 ffff88000b8d1fd8 ffff880103233820 ffff880071252100
 ffff88000b8d1838 ffff880071252100 ffff88011fc11e18 ffff88010ff79860
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa015c5ea>] dbuf_prefetch+0xfa/0x280 [zfs]
 [<ffffffffa016ed88>] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
 [<ffffffffa015a310>] ? dbuf_do_evict+0x70/0x70 [zfs]
 [<ffffffffa016f80c>] dmu_zfetch+0x77c/0xea0 [zfs]
 [<ffffffffa015ad9a>] dbuf_read+0x45a/0x7f0 [zfs]
 [<ffffffffa0161a4a>] dmu_buf_hold_array_by_dnode+0x15a/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 17580  10850 0x20020000
 ffff88002eaa7a38 0000000000000082 ffff8800a8ea3200 ffff88002eaa7fd8
 ffff88002eaa7fd8 ffff88002eaa7fd8 ffff8800da3f4280 ffff8800a8ea3200
 ffffffffa015a310 ffff8800a13c22f0 ffff8800a13c22f4 ffff8800a8ea3200
Call Trace:
 [<ffffffffa015a310>] ? dbuf_do_evict+0x70/0x70 [zfs]
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffffa015a310>] ? dbuf_do_evict+0x70/0x70 [zfs]
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffffa016f423>] dmu_zfetch+0x393/0xea0 [zfs]
 [<ffffffffa015ad9a>] dbuf_read+0x45a/0x7f0 [zfs]
 [<ffffffffa0161a4a>] dmu_buf_hold_array_by_dnode+0x15a/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 17607  10850 0x20020000
 ffff88005f0b78d8 0000000000000082 ffff880117cab820 ffff88005f0b7fd8
 ffff88005f0b7fd8 ffff88005f0b7fd8 ffffffff81814440 ffff880117cab820
 ffff88005f0b78d8 ffff880117cab820 ffff88011fc11e18 ffff880052986450
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0161554>] dmu_buf_hold+0xf4/0x1b0 [zfs]
 [<ffffffffa01b870a>] zap_get_leaf_byblk+0x4a/0x2b0 [zfs]
 [<ffffffffa0159cc1>] ? dbuf_rele_and_unlock+0x151/0x200 [zfs]
 [<ffffffffa01b89d8>] zap_deref_leaf+0x68/0x80 [zfs]
 [<ffffffffa01ba4bf>] fzap_cursor_retrieve+0xdf/0x270 [zfs]
 [<ffffffffa01bdc36>] zap_cursor_retrieve+0x66/0x2f0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa0196d0e>] ? sa_lookup_impl+0xe/0x10 [zfs]
 [<ffffffffa0196da1>] ? sa_lookup+0x41/0x50 [zfs]
 [<ffffffff81148f73>] ? compat_filldir+0x93/0x110
 [<ffffffffa01dd6f7>] zfs_readdir+0x147/0x4c0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffff810dbf77>] ? handle_mm_fault+0x227/0x2e0
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa01f2954>] zpl_readdir+0x54/0x80 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffff811142f8>] vfs_readdir+0xb8/0xe0
 [<ffffffff8114ab2a>] compat_sys_getdents+0x8a/0x100
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 17609  10850 0x20020000
 ffff880029fd1bb8 0000000000000086 ffff8800ac4b3620 ffff880029fd1fd8
 ffff880029fd1fd8 ffff880029fd1fd8 ffff8801031160c0 ffff8800ac4b3620
 ffff880029fd1bc8 ffff880080621890 0000000000000002 ffff880080621858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 17850  10850 0x20020000
 ffff88010ec83b58 0000000000000086 ffff8800ae5c2e00 ffff88010ec83fd8
 ffff88010ec83fd8 ffff88010ec83fd8 ffffffff81814440 ffff8800ae5c2e00
 ffff88010ec83b58 ffff8800ae5c2e00 ffff88011fc11e18 ffff880050000780
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18056  10850 0x20020000
 ffff880089119bb8 0000000000000082 ffff88006b7cb660 ffff880089119fd8
 ffff880089119fd8 ffff880089119fd8 ffff8800d3dd8720 ffff88006b7cb660
 ffff880089119bc8 ffff880091807890 0000000000000002 ffff880091807858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18058  18057 0x20020000
 ffff880029f13b58 0000000000000086 ffff88003e7b4180 ffff880029f13fd8
 ffff880029f13fd8 ffff880029f13fd8 ffff88011a09a0c0 ffff88003e7b4180
 ffff880029f13b58 ffff88003e7b4180 ffff88011fd11e18 ffff88002533e018
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18111  10850 0x20020000
 ffff880078527bb8 0000000000000082 ffff88011a2de180 ffff880078527fd8
 ffff880078527fd8 ffff880078527fd8 ffff8800d022cf00 ffff88011a2de180
 ffff880078527bc8 ffff880091807890 0000000000000002 ffff880091807858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 18264  10850 0x20020000
 ffff88002c4f7b58 0000000000000086 ffff88007e97b520 ffff88002c4f7fd8
 ffff88002c4f7fd8 ffff88002c4f7fd8 ffff88008e33a7a0 ffff88007e97b520
 ffff88002c4f7b58 ffff88007e97b520 ffff88011fc11e18 ffff88003ff46018
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18285  10850 0x20020000
 ffff880011347bb8 0000000000000086 ffff8800d3dd8720 ffff880011347fd8
 ffff880011347fd8 ffff880011347fd8 ffff88011a09a0c0 ffff8800d3dd8720
 ffff880011347bc8 ffff880091807890 0000000000000002 ffff880091807858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 18411  10850 0x20020000
 ffff88000e719bb8 0000000000000082 ffff88003553f7a0 ffff88000e719fd8
 ffff88000e719fd8 ffff88000e719fd8 ffff8800d3dd8720 ffff88003553f7a0
 ffff88000e719bc8 ffff880091807890 0000000000000002 ffff880091807858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 18501  10850 0x20020000
 ffff880037545bb8 0000000000000086 ffff88008e33a7a0 ffff880037545fd8
 ffff880037545fd8 ffff880037545fd8 ffff8800db40a9e0 ffff88008e33a7a0
 ffff880037545bc8 ffff88008ee82210 0000000000000002 ffff88008ee821d8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18567  10850 0x20020000
 ffff8800d2fa7bb8 0000000000000086 ffff8800a866b000 ffff8800d2fa7fd8
 ffff8800d2fa7fd8 ffff8800d2fa7fd8 ffff88006b7cb660 ffff8800a866b000
 ffff8800d2fa7bc8 ffff880091807890 0000000000000002 ffff880091807858
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 18797  10850 0x20020000
 ffff88008c2dd8d8 0000000000000086 ffff8800d8ebf560 ffff88008c2ddfd8
 ffff88008c2ddfd8 ffff88008c2ddfd8 ffff88010b8778e0 ffff8800d8ebf560
 ffff88008c2dd8d8 ffff8800d8ebf560 ffff88011fd11e18 ffff880066656888
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0161554>] dmu_buf_hold+0xf4/0x1b0 [zfs]
 [<ffffffffa01b870a>] zap_get_leaf_byblk+0x4a/0x2b0 [zfs]
 [<ffffffffa0159cc1>] ? dbuf_rele_and_unlock+0x151/0x200 [zfs]
 [<ffffffffa01b89d8>] zap_deref_leaf+0x68/0x80 [zfs]
 [<ffffffffa01ba4bf>] fzap_cursor_retrieve+0xdf/0x270 [zfs]
 [<ffffffffa01bdc36>] zap_cursor_retrieve+0x66/0x2f0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa0196d0e>] ? sa_lookup_impl+0xe/0x10 [zfs]
 [<ffffffffa0196da1>] ? sa_lookup+0x41/0x50 [zfs]
 [<ffffffff81148f73>] ? compat_filldir+0x93/0x110
 [<ffffffffa01dd6f7>] zfs_readdir+0x147/0x4c0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffff8105d215>] ? check_preempt_curr+0x85/0xa0
 [<ffffffff8105d242>] ? ttwu_do_wakeup+0x12/0x90
 [<ffffffff8105f662>] ? try_to_wake_up+0x1f2/0x270
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa01f2954>] zpl_readdir+0x54/0x80 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffff811142f8>] vfs_readdir+0xb8/0xe0
 [<ffffffff814b77e0>] ? __schedule+0x2a0/0x630
 [<ffffffff8114ab2a>] compat_sys_getdents+0x8a/0x100
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 18847  10850 0x20020000
 ffff8800050e9b58 0000000000000086 ffff880044e27040 ffff8800050e9fd8
 ffff8800050e9fd8 ffff8800050e9fd8 ffff88003553f7a0 ffff880044e27040
 ffff8800050e9b58 ffff880044e27040 ffff88011fc11e18 ffff880056348ff0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
@akorn
akorn commented May 22, 2013

sysrq-w, part 2:

imapd           D ffff88011fc11580     0 19131  10850 0x20020000
 ffff88001d341838 0000000000000086 ffff88010b8778e0 ffff88001d341fd8
 ffff88001d341fd8 ffff88001d341fd8 ffffffff81814440 ffff88010b8778e0
 ffff88001d341838 ffff88010b8778e0 ffff88011fc11e18 ffff8800639035e8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa015c5ea>] dbuf_prefetch+0xfa/0x280 [zfs]
 [<ffffffffa016ed88>] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
 [<ffffffffa015a310>] ? dbuf_do_evict+0x70/0x70 [zfs]
 [<ffffffff814b8659>] ? _raw_spin_unlock_irqrestore+0x9/0x10
 [<ffffffffa016f80c>] dmu_zfetch+0x77c/0xea0 [zfs]
 [<ffffffffa015ad9a>] dbuf_read+0x45a/0x7f0 [zfs]
 [<ffffffffa0161a4a>] dmu_buf_hold_array_by_dnode+0x15a/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 19402  10850 0x20020000
 ffff8800b8871bb8 0000000000000082 ffff8800881aaf40 ffff8800b8871fd8
 ffff8800b8871fd8 ffff8800b8871fd8 ffff8800cf2428a0 ffff8800881aaf40
 ffff8800b8871bc8 ffff88002f80d110 0000000000000002 ffff88002f80d0d8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 19438  10850 0x20020000
 ffff8800db5f1838 0000000000000086 ffff880109290920 ffff8800db5f1fd8
 ffff8800db5f1fd8 ffff8800db5f1fd8 ffff880117caa380 ffff880109290920
 ffff8800db5f1838 ffff880109290920 ffff88011fd11e18 ffff88001b1dcf38
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa015c5ea>] dbuf_prefetch+0xfa/0x280 [zfs]
 [<ffffffffa016ed88>] dmu_zfetch_dofetch.isra.4+0xe8/0x160 [zfs]
 [<ffffffffa015a310>] ? dbuf_do_evict+0x70/0x70 [zfs]
 [<ffffffffa016f80c>] dmu_zfetch+0x77c/0xea0 [zfs]
 [<ffffffffa015ad9a>] dbuf_read+0x45a/0x7f0 [zfs]
 [<ffffffffa0161a4a>] dmu_buf_hold_array_by_dnode+0x15a/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 19579  19578 0x20020000
 ffff88008047f8d8 0000000000000086 ffff88005e6d69a0 ffff88008047ffd8
 ffff88008047ffd8 ffff88008047ffd8 ffffffff81814440 ffff88005e6d69a0
 ffff88008047f8d8 ffff88005e6d69a0 ffff88011fc11e18 ffff88007d089c98
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0161554>] dmu_buf_hold+0xf4/0x1b0 [zfs]
 [<ffffffffa01b870a>] zap_get_leaf_byblk+0x4a/0x2b0 [zfs]
 [<ffffffffa0159cc1>] ? dbuf_rele_and_unlock+0x151/0x200 [zfs]
 [<ffffffffa01b89d8>] zap_deref_leaf+0x68/0x80 [zfs]
 [<ffffffffa01ba4bf>] fzap_cursor_retrieve+0xdf/0x270 [zfs]
 [<ffffffffa01bdc36>] zap_cursor_retrieve+0x66/0x2f0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa0196d0e>] ? sa_lookup_impl+0xe/0x10 [zfs]
 [<ffffffffa0196da1>] ? sa_lookup+0x41/0x50 [zfs]
 [<ffffffff81148f73>] ? compat_filldir+0x93/0x110
 [<ffffffffa01dd6f7>] zfs_readdir+0x147/0x4c0 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa01dcba0>] ? zfs_getattr_fast+0xd0/0x150 [zfs]
 [<ffffffff8103bfd1>] ? current_fs_time+0x11/0x60
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffffa01f2954>] zpl_readdir+0x54/0x80 [zfs]
 [<ffffffff81148ee0>] ? compat_set_fd_set+0x80/0x80
 [<ffffffff811142f8>] vfs_readdir+0xb8/0xe0
 [<ffffffff8114ab2a>] compat_sys_getdents+0x8a/0x100
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 19588  10850 0x20020000
 ffff880091c61b58 0000000000000086 ffff88005a18f1c0 ffff880091c61fd8
 ffff880091c61fd8 ffff880091c61fd8 ffffffff81814440 ffff88005a18f1c0
 ffff880091c61b58 ffff88005a18f1c0 ffff88011fc11e18 ffff8800730be450
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 19820  10850 0x20020000
 ffff880049d8dbb8 0000000000000082 ffff88008ec1d7e0 ffff880049d8dfd8
 ffff880049d8dfd8 ffff880049d8dfd8 ffff8800a8e96400 ffff88008ec1d7e0
 ffff880049d8dbc8 ffff88010b3e8350 0000000000000002 ffff88010b3e8318
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 19881  10850 0x20020000
 ffff88008e7ddbb8 0000000000000082 ffff8800cf2428a0 ffff88008e7ddfd8
 ffff88008e7ddfd8 ffff88008e7ddfd8 ffffffff81814440 ffff8800cf2428a0
 ffff88008e7ddbc8 ffff88002f80d110 0000000000000002 ffff88002f80d0d8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 19918  10850 0x20020000
 ffff88011176db58 0000000000000082 ffff88008ec1d100 ffff88011176dfd8
 ffff88011176dfd8 ffff88011176dfd8 ffff88011a09a0c0 ffff88008ec1d100
 ffff88011176db58 ffff88008ec1d100 ffff88011fd11e18 ffff8800b8826018
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20029  20027 0x20020000
 ffff880067a71b58 0000000000000086 ffff88011a0c6100 ffff880067a71fd8
 ffff880067a71fd8 ffff880067a71fd8 ffff88010644aa20 ffff88011a0c6100
 ffff880067a71b58 ffff88011a0c6100 ffff88011fc11e18 ffff88001a6d8780
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20059  10850 0x20020000
 ffff8800836c1b58 0000000000000082 ffff8800d3d91200 ffff8800836c1fd8
 ffff8800836c1fd8 ffff8800836c1fd8 ffff8800a8e96400 ffff8800d3d91200
 ffff8800836c1b58 ffff8800d3d91200 ffff88011fd11e18 ffff88005d70a0d0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20073  10850 0x20020000
 ffff8801092f9c08 0000000000000082 ffff88006381c280 ffff8801092f9fd8
 ffff8801092f9fd8 ffff8801092f9fd8 ffff88011a1101c0 ffff88006381c280
 ffff880117fb6000 ffff8800a3594ea8 ffff8800a3594eac ffff88006381c280
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20079  10850 0x20020000
 ffff8801093adc08 0000000000000082 ffff88005e6d62c0 ffff8801093adfd8
 ffff8801093adfd8 ffff8801093adfd8 ffff88011f07a040 ffff88005e6d62c0
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff88005e6d62c0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20116  10850 0x20020000
 ffff880015755c08 0000000000000082 ffff880008aa97a0 ffff880015755fd8
 ffff880015755fd8 ffff880015755fd8 ffff88004780ef40 ffff880008aa97a0
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff880008aa97a0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20178  10850 0x20020000
 ffff88010595fb58 0000000000000086 ffff8800079ca240 ffff88010595ffd8
 ffff88010595ffd8 ffff88010595ffd8 ffff88011a09a0c0 ffff8800079ca240
 ffff88010595fb58 ffff8800079ca240 ffff88011fd11e18 ffff88010861f0f8
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20295  10850 0x20020000
 ffff88004df19bb8 0000000000000082 ffff8800483215e0 ffff88004df19fd8
 ffff88004df19fd8 ffff88004df19fd8 ffff88008ec1d7e0 ffff8800483215e0
 ffff88004df19bc8 ffff88010b3e8350 0000000000000002 ffff88010b3e8318
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20390  10850 0x20020000
 ffff88010b8a5c08 0000000000000082 ffff88004780ef40 ffff88010b8a5fd8
 ffff88010b8a5fd8 ffff88010b8a5fd8 ffff8800694eb040 ffff88004780ef40
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff88004780ef40
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20392  10850 0x20020000
 ffff88011616db58 0000000000000082 ffff8801059af8e0 ffff88011616dfd8
 ffff88011616dfd8 ffff88011616dfd8 ffff88010644aa20 ffff8801059af8e0
 ffff88011616db58 ffff8801059af8e0 ffff88011fc11e18 ffff880016be2940
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20481  10850 0x20020000
 ffff880110c79bb8 0000000000000082 ffff8800a8e96400 ffff880110c79fd8
 ffff880110c79fd8 ffff880110c79fd8 ffff8800da3f4960 ffff8800a8e96400
 ffff880110c79bc8 ffff88010b3e8350 0000000000000002 ffff88010b3e8318
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20640  20639 0x20020000
 ffff88000fda9b58 0000000000000086 ffff88006fffd620 ffff88000fda9fd8
 ffff88000fda9fd8 ffff88000fda9fd8 ffffffff81814440 ffff88006fffd620
 ffff88000fda9b58 ffff88006fffd620 ffff88011fc11e18 ffff88007e66d370
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0161ab5>] dmu_buf_hold_array_by_dnode+0x1c5/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa00356b3>] ? avl_add+0x33/0x50 [zavl]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffff810df63f>] ? __vm_enough_memory+0x2f/0x150
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20645  10850 0x20020000
 ffff880079ec16d8 0000000000000086 ffff880109c3b140 ffff880079ec1fd8
 ffff880079ec1fd8 ffff880079ec1fd8 ffff88005e6d62c0 ffff880109c3b140
 ffff880079ec16d8 ffff880109c3b140 ffff88011fc11e18 ffff8800840f0ff0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0170fd4>] dnode_hold_impl+0x1b4/0x590 [zfs]
 [<ffffffffa0150f50>] ? remove_reference.isra.9+0xa0/0xc0 [zfs]
 [<ffffffffa01713c4>] dnode_hold+0x14/0x20 [zfs]
 [<ffffffffa016163f>] dmu_bonus_hold+0x2f/0x2e0 [zfs]
 [<ffffffffa0193f89>] sa_buf_hold+0x9/0x10 [zfs]
 [<ffffffffa01e56d2>] zfs_zget+0xc2/0x1c0 [zfs]
 [<ffffffffa00a3118>] ? kmem_alloc_debug+0x128/0x390 [spl]
 [<ffffffffa01c5b9f>] zfs_dirent_lock+0x49f/0x5a0 [zfs]
 [<ffffffffa01c5d26>] zfs_dirlook+0x86/0x2e0 [zfs]
 [<ffffffffa01c2af8>] ? zfs_zaccess+0x98/0x3c0 [zfs]
 [<ffffffffa01dee91>] zfs_lookup+0x2f1/0x340 [zfs]
 [<ffffffffa01f2f90>] zpl_lookup+0x70/0xd0 [zfs]
 [<ffffffff8110b748>] lookup_real+0x18/0x50
 [<ffffffff8110b993>] __lookup_hash+0x33/0x40
 [<ffffffff814b26ff>] lookup_slow+0x42/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20673  10850 0x20020000
 ffff88011b2e1bb8 0000000000000086 ffff8800ae4dcb20 ffff88011b2e1fd8
 ffff88011b2e1fd8 ffff88011b2e1fd8 ffff88011a09a0c0 ffff8800ae4dcb20
 ffff88011b2e1bc8 ffff88008c061750 0000000000000002 ffff88008c061718
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20738  20737 0x20020000
 ffff88008aa2bbb8 0000000000000082 ffff8800ac528f80 ffff88008aa2bfd8
 ffff88008aa2bfd8 ffff88008aa2bfd8 ffff8800483215e0 ffff8800ac528f80
 ffff88008aa2bbc8 ffff88010b3e8350 0000000000000002 ffff88010b3e8318
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffffa00acac5>] cv_wait_common+0xf5/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acbc0>] __cv_wait+0x10/0x20 [spl]
 [<ffffffffa0161b2b>] dmu_buf_hold_array_by_dnode+0x23b/0x510 [zfs]
 [<ffffffffa0161f90>] dmu_buf_hold_array+0x60/0x90 [zfs]
 [<ffffffffa01d50b0>] ? zfs_range_proxify.part.1+0x80/0x90 [zfs]
 [<ffffffffa016351c>] dmu_read_uio+0x3c/0xd0 [zfs]
 [<ffffffffa01df118>] zfs_read+0x168/0x480 [zfs]
 [<ffffffffa01f2acd>] zpl_read_common+0x4d/0x80 [zfs]
 [<ffffffffa01f2b5f>] zpl_read+0x5f/0x90 [zfs]
 [<ffffffff81102914>] vfs_read+0xa4/0x180
 [<ffffffff81102a3d>] sys_read+0x4d/0x90
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
amavisd-new     D ffff88011fc11580     0 20741  12124 0x20020000
 ffff88010f393c08 0000000000200082 ffff88006381d720 ffff88010f393fd8
 ffff88010f393fd8 ffff88010f393fd8 ffffffff81814440 ffff88006381d720
 ffff88010f393c08 ffff88006381d720 ffff88011fc11e18 0000000000000082
Call Trace:
 [<ffffffff810bb890>] ? sleep_on_page+0x10/0x10
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffff810bb899>] sleep_on_page_killable+0x9/0x40
 [<ffffffff814b6768>] __wait_on_bit+0x58/0x90
 [<ffffffff810bd9c8>] wait_on_page_bit_killable+0x78/0x80
 [<ffffffff81053150>] ? autoremove_wake_function+0x40/0x40
 [<ffffffff810bda5d>] __lock_page_or_retry+0x8d/0xb0
 [<ffffffff810daf3d>] handle_pte_fault+0x26d/0xab0
 [<ffffffff810dbf77>] handle_mm_fault+0x227/0x2e0
 [<ffffffff8102a669>] __do_page_fault+0x139/0x470
 [<ffffffff81076e87>] ? getnstimeofday+0x47/0xd0
 [<ffffffff8102a9c9>] do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] page_fault+0x22/0x30
imapd           D ffff88011fd11580     0 20769  10850 0x20020000
 ffff880079293c08 0000000000000086 ffff88005f6755a0 ffff880079293fd8
 ffff880079293fd8 ffff880079293fd8 ffff88011a09ae80 ffff88005f6755a0
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff88005f6755a0
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20777  20776 0x20020000
 ffff880078439c08 0000000000000082 ffff8800694eb040 ffff880078439fd8
 ffff880078439fd8 ffff880078439fd8 ffff88005f6755a0 ffff8800694eb040
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff8800694eb040
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20788  10850 0x20020000
 ffff880065595638 0000000000000082 ffff8800d3d25180 ffff880065595fd8
 ffff880065595fd8 ffff880065595fd8 ffff88010b8778e0 ffff8800d3d25180
 ffff880065595638 ffff8800d3d25180 ffff88011fc11e18 ffff8800415f1428
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0161554>] dmu_buf_hold+0xf4/0x1b0 [zfs]
 [<ffffffffa01b870a>] zap_get_leaf_byblk+0x4a/0x2b0 [zfs]
 [<ffffffffa01bca70>] ? zap_match.part.5+0x70/0x70 [zfs]
 [<ffffffffa01b89d8>] zap_deref_leaf+0x68/0x80 [zfs]
 [<ffffffffa01b951b>] fzap_lookup+0x5b/0x120 [zfs]
 [<ffffffffa01bee49>] zap_lookup_norm+0xe9/0x190 [zfs]
 [<ffffffffa00a3118>] ? kmem_alloc_debug+0x128/0x390 [spl]
 [<ffffffffa01c00e5>] ? zfs_acl_next_ace+0x105/0x170 [zfs]
 [<ffffffffa01bef6e>] zap_lookup+0x2e/0x30 [zfs]
 [<ffffffffa01c5bd7>] zfs_dirent_lock+0x4d7/0x5a0 [zfs]
 [<ffffffffa01c5d26>] zfs_dirlook+0x86/0x2e0 [zfs]
 [<ffffffffa01c2af8>] ? zfs_zaccess+0x98/0x3c0 [zfs]
 [<ffffffff81118cb6>] ? d_splice_alias+0xe6/0xf0
 [<ffffffffa01dee91>] zfs_lookup+0x2f1/0x340 [zfs]
 [<ffffffffa01f2f90>] zpl_lookup+0x70/0xd0 [zfs]
 [<ffffffff8110b748>] lookup_real+0x18/0x50
 [<ffffffff8110b993>] __lookup_hash+0x33/0x40
 [<ffffffff814b26ff>] lookup_slow+0x42/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff810dad5d>] ? handle_pte_fault+0x8d/0xab0
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffff8102a6dc>] ? __do_page_fault+0x1ac/0x470
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff81107186>] vfs_stat+0x16/0x20
 [<ffffffff8102fba5>] sys32_stat64+0x15/0x30
 [<ffffffff8102a9c9>] ? do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] ? page_fault+0x22/0x30
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fc11580     0 20791  10850 0x20020000
 ffff8800813a1c08 0000000000000086 ffff88011f07a040 ffff8800813a1fd8
 ffff8800813a1fd8 ffff8800813a1fd8 ffff88006381c280 ffff88011f07a040
 0000000000000000 ffff8800a3594ea8 ffff8800a3594eac ffff88011f07a040
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e176>] link_path_walk+0x806/0x870
 [<ffffffff810c69c8>] ? release_pages+0x1b8/0x1f0
 [<ffffffff81112159>] path_openat+0xa9/0x570
 [<ffffffff810d87fc>] ? tlb_flush_mmu+0x5c/0xa0
 [<ffffffff810df2b7>] ? unmap_region+0xd7/0x110
 [<ffffffff8111274c>] do_filp_open+0x3c/0x90
 [<ffffffff8111df12>] ? __alloc_fd+0x42/0x130
 [<ffffffff81101e1f>] do_sys_open+0xef/0x1d0
 [<ffffffff8114b086>] compat_sys_open+0x16/0x20
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20795  20793 0x20020000
 ffff880083acd6b8 0000000000000082 ffff8800a8e509e0 ffff880083acdfd8
 ffff880083acdfd8 ffff880083acdfd8 ffff88011a09a0c0 ffff8800a8e509e0
 ffff880083acd6b8 ffff8800a8e509e0 ffff88011fd11e18 ffff88001d0ce450
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa015ac77>] dbuf_read+0x337/0x7f0 [zfs]
 [<ffffffffa0161554>] dmu_buf_hold+0xf4/0x1b0 [zfs]
 [<ffffffffa01bd2b2>] zap_lockdir+0x52/0x710 [zfs]
 [<ffffffffa0196da1>] ? sa_lookup+0x41/0x50 [zfs]
 [<ffffffffa01c0e87>] ? zfs_acl_node_read.constprop.13+0x2e7/0x300 [zfs]
 [<ffffffffa01beda5>] zap_lookup_norm+0x45/0x190 [zfs]
 [<ffffffffa00a3118>] ? kmem_alloc_debug+0x128/0x390 [spl]
 [<ffffffffa01c00e5>] ? zfs_acl_next_ace+0x105/0x170 [zfs]
 [<ffffffffa01bef6e>] zap_lookup+0x2e/0x30 [zfs]
 [<ffffffffa01c5bd7>] zfs_dirent_lock+0x4d7/0x5a0 [zfs]
 [<ffffffffa01c5d26>] zfs_dirlook+0x86/0x2e0 [zfs]
 [<ffffffffa01c2af8>] ? zfs_zaccess+0x98/0x3c0 [zfs]
 [<ffffffff81118cb6>] ? d_splice_alias+0xe6/0xf0
 [<ffffffffa01dee91>] zfs_lookup+0x2f1/0x340 [zfs]
 [<ffffffffa01f2f90>] zpl_lookup+0x70/0xd0 [zfs]
 [<ffffffff8110b748>] lookup_real+0x18/0x50
 [<ffffffff8110b993>] __lookup_hash+0x33/0x40
 [<ffffffff814b26ff>] lookup_slow+0x42/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff810dad5d>] ? handle_pte_fault+0x8d/0xab0
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffff8102a6dc>] ? __do_page_fault+0x1ac/0x470
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff810df3de>] ? remove_vma+0x4e/0x60
 [<ffffffff81107186>] vfs_stat+0x16/0x20
 [<ffffffff8102fba5>] sys32_stat64+0x15/0x30
 [<ffffffff8102a9c9>] ? do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] ? page_fault+0x22/0x30
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
imapd           D ffff88011fd11580     0 20802  20801 0x20020000
 ffff88000812dbe8 0000000000000082 ffff8800354ca380 ffff88000812dfd8
 ffff88000812dfd8 ffff88000812dfd8 ffff88006fffcf40 ffff8800354ca380
 0000000000000002 ffff88005e0b4ea8 ffff88005e0b4eac ffff8800354ca380
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7e89>] schedule_preempt_disabled+0x9/0x10
 [<ffffffff814b6d12>] __mutex_lock_slowpath+0xc2/0x130
 [<ffffffff814b684d>] mutex_lock+0x1d/0x40
 [<ffffffff814b26f1>] lookup_slow+0x34/0xa6
 [<ffffffff8110e9af>] path_lookupat+0x6ef/0x750
 [<ffffffff810dad5d>] ? handle_pte_fault+0x8d/0xab0
 [<ffffffff8110ea3f>] filename_lookup+0x2f/0xc0
 [<ffffffff811103b7>] user_path_at_empty+0x87/0x100
 [<ffffffff8102a6dc>] ? __do_page_fault+0x1ac/0x470
 [<ffffffff8111043c>] user_path_at+0xc/0x10
 [<ffffffff81107110>] vfs_fstatat+0x30/0x70
 [<ffffffff81000f2f>] ? __switch_to+0x37f/0x430
 [<ffffffff81107186>] vfs_stat+0x16/0x20
 [<ffffffff8102fba5>] sys32_stat64+0x15/0x30
 [<ffffffff8102a9c9>] ? do_page_fault+0x9/0x10
 [<ffffffff814b8df2>] ? page_fault+0x22/0x30
 [<ffffffff814ba9b9>] ia32_do_call+0x13/0x13
zpool           D ffff88011fd11580     0 20803  10895 0x00000000
 ffff88005aef7b28 0000000000000086 ffff8801016182c0 ffff88005aef7fd8
 ffff88005aef7fd8 ffff88005aef7fd8 ffff88011a09a0c0 ffff8801016182c0
 ffff88005aef7b28 ffff8801016182c0 ffff88011fd11e18 ffff8800129cf530
Call Trace:
 [<ffffffff814b7c24>] schedule+0x24/0x60
 [<ffffffff814b7cea>] io_schedule+0x8a/0xd0
 [<ffffffffa00aca73>] cv_wait_common+0xa3/0x1a0 [spl]
 [<ffffffff81053110>] ? finish_wait+0x80/0x80
 [<ffffffffa00acb83>] __cv_wait_io+0x13/0x20 [spl]
 [<ffffffffa01ed8b3>] zio_wait+0x103/0x1a0 [zfs]
 [<ffffffffa0155338>] arc_read_nolock+0x508/0x750 [zfs]
 [<ffffffff810f2769>] ? __kmalloc+0x119/0x120
 [<ffffffffa01545e0>] ? arc_buf_remove_ref+0x100/0x100 [zfs]
 [<ffffffffa018b065>] dsl_read_nolock+0x25/0x30 [zfs]
 [<ffffffffa0165767>] dmu_objset_open_impl+0xd7/0x680 [zfs]
 [<ffffffffa0165d94>] dmu_objset_from_ds+0x84/0x90 [zfs]
 [<ffffffffa0165eac>] dmu_objset_hold+0x3c/0x60 [zfs]
 [<ffffffffa01cc084>] zfs_ioc_objset_stats+0x24/0x60 [zfs]
 [<ffffffffa01d14d9>] ? pool_status_check+0x59/0xa0 [zfs]
 [<ffffffffa01d15f7>] zfsdev_ioctl+0xd7/0x1b0 [zfs]
 [<ffffffff81113a5b>] do_vfs_ioctl+0x8b/0x500
 [<ffffffff810df3de>] ? remove_vma+0x4e/0x60
 [<ffffffff810e0b98>] ? do_munmap+0x318/0x3f0
 [<ffffffff81113f61>] sys_ioctl+0x91/0xb0
 [<ffffffff814b9316>] system_call_fastpath+0x1a/0x1f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment