Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZFS gets stuck when copying large files #7241

Closed
tuomari opened this issue Feb 27, 2018 · 5 comments
Closed

ZFS gets stuck when copying large files #7241

tuomari opened this issue Feb 27, 2018 · 5 comments

Comments

@tuomari
Copy link

tuomari commented Feb 27, 2018

System information

Type Version/Name
Distribution Name Debian
Distribution Version Jessie (8.10)
Linux Kernel 4.9.49
Architecture amd64
ZFS Version 0.7.0-338_g41532e5
SPL Version v0.7.0-29_g378c6ed

Describe the problem you're observing

I am running multiple kvm virtuals from zvols. When moving large amounts of data from one zvol to another, zfs stops writing data to all disks. There are no errors in zpool events, syslog, or anywhere I could find in the host system. I was able to repeat the problem with the latest master ( ZFS 0.7.0-338_g41532e5 and SPL 0.7.0-29_g378c6ed.

(Maybe) noteworthy points

  • Destination had sync=disabled ( yes, I know this is dangerous and stupid... )
  • There was a lot of free memory ( > 30G )
  • There was a lot of free space on the pool
  • Source and destination were both in the SSD tank
  • There was zfs send running from ssdtank to tank on the background, limited with pv to 10M/s
  • After reboot log-disks on the tank pool was completely full
  • Data rates: ~constant 200M/s read from zvol A, and chunky ~500-600M/s writes to B. After the 5-10G file is moved, delete ( with trim ) the original file from zvol A. Repeat.

My disk layout is as follows:

  pool: ssdtank
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 2 days 20:26:09 with 0 errors on Thu Feb  8 11:06:08 2018
config:

	NAME                                                   STATE     READ WRITE CKSUM
	ssdtank                                                ONLINE       0     0     0
	  mirror-0                                             ONLINE       0     0     0
	    ata-Samsung_SSD_850_PRO_2TB_S2KMNWAG801975L-part4  ONLINE       0     0     0
	    ata-Samsung_SSD_850_PRO_2TB_S2KMNX0H701810E-part4  ONLINE       0     0     0
	  mirror-1                                             ONLINE       0     0     0
	    ata-Samsung_SSD_850_PRO_2TB_S2KMNCAGB03862T-part4  ONLINE       0     0     0
	    ata-Samsung_SSD_850_PRO_2TB_S2KMNX0H701818D-part4  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: resilvered 2.27T in 8 days 05:14:41 with 0 errors on Fri Jul 14 14:36:45 2017
config:

	NAME                                                                STATE     READ WRITE CKSUM
	tank                                                                ONLINE       0     0     0
	  mirror-0                                                          ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N2YF4XDX                        ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7DYXSU7                        ONLINE       0     0     0
	  mirror-1                                                          ONLINE       0     0     0
	    ata-ST3000DM001-1CH166_W1F3P4FA                                 ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7DPPL8F                        ONLINE       0     0     0
	  mirror-2                                                          ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5XALU50                        ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N0RVP95P                        ONLINE       0     0     0
	  mirror-4                                                          ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5ECAKDR                        ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N2VL6ST1                        ONLINE       0     0     0
	  mirror-5                                                          ONLINE       0     0     0
	    ata-ST3000DM001-1CH166_Z1F4DJQZ                                 ONLINE       0     0     0
	    ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N2KNSX2D                        ONLINE       0     0     0
	logs
	  mirror-6                                                          ONLINE       0     0     0
	    scsi-INTEL-SSD_910_200GB-XVV11VNB-part1                         ONLINE       0     0     0
	    ata-OWC_Mercury_Enterprise_Pro_6G_SSD_MEP6G05012E26L6601-part1  ONLINE       0     0     0
	    scsi-INTEL-SSD_910_200GB-XVV11VPB-part1                         ONLINE       0     0     0
	cache
	  scsi-INTEL-SSD_910_200GB-XVV11VNB-part2                           ONLINE       0     0     0
	  scsi-INTEL-SSD_910_200GB-XVV11VPB-part2                           ONLINE       0     0     0
	spares
	  ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7JX81L3                          AVAIL   
	  ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N1PJNVX0                          AVAIL   

errors: No known data errors

Include any warning/errors/backtraces from the system logs

result from echo w > /proc/sysrq-trigger


Feb 24 04:44:44 genetti kernel: [14121947.139231] sysrq: SysRq : Show Blocked State
Feb 24 04:44:44 genetti kernel: [14121947.139261]   task                        PC stack   pid father
Feb 24 04:44:44 genetti kernel: [14121947.139730] systemd-udevd   D    0   798      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.139759]  ffff881febcd52d8 ffff883fee480800 ffff881febcd4e00 ffff881fffbd8080
Feb 24 04:44:44 genetti kernel: [14121947.139809]  ffff881ff295a700 ffffc9001b4ffae8 ffffffff816d85e3 ffffc9001b4ffac8
Feb 24 04:44:44 genetti kernel: [14121947.139860]  0000000000000246 ffffc9001b4ffaf8 ffff881febcd4e00 ffff881fede63420
Feb 24 04:44:44 genetti kernel: [14121947.139910] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.139936]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.139964]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.139990]  [<ffffffff816d8d59>] schedule_preempt_disabled+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.140032]  [<ffffffff816db3c2>] __mutex_lock_slowpath+0xf2/0x2c0
Feb 24 04:44:44 genetti kernel: [14121947.140062]  [<ffffffff8130fea0>] ? disk_map_sector_rcu+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.140092]  [<ffffffff816db5a5>] mutex_lock+0x15/0x30
Feb 24 04:44:44 genetti kernel: [14121947.140120]  [<ffffffff81222b9a>] __blkdev_get+0x5a/0x450
Feb 24 04:44:44 genetti kernel: [14121947.140147]  [<ffffffff81222fc8>] blkdev_get+0x38/0x340
Feb 24 04:44:44 genetti kernel: [14121947.140175]  [<ffffffff812db1e3>] ? security_capable+0x43/0x60
Feb 24 04:44:44 genetti kernel: [14121947.140202]  [<ffffffff816ddb99>] ? _raw_spin_unlock+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.140229]  [<ffffffff81223376>] blkdev_open+0x56/0x70
Feb 24 04:44:44 genetti kernel: [14121947.140257]  [<ffffffff811e57df>] do_dentry_open+0x1df/0x2f0
Feb 24 04:44:44 genetti kernel: [14121947.140284]  [<ffffffff81223320>] ? blkdev_get_by_dev+0x50/0x50
Feb 24 04:44:44 genetti kernel: [14121947.140312]  [<ffffffff811e6afa>] vfs_open+0x4a/0x70
Feb 24 04:44:44 genetti kernel: [14121947.140338]  [<ffffffff811f2cd6>] ? may_open+0x96/0x100
Feb 24 04:44:44 genetti kernel: [14121947.140364]  [<ffffffff811f68a0>] path_openat+0x2f0/0x1560
Feb 24 04:44:44 genetti kernel: [14121947.140393]  [<ffffffff81182f9c>] ? release_pages+0x29c/0x330
Feb 24 04:44:44 genetti kernel: [14121947.140420]  [<ffffffff811f8c09>] do_filp_open+0x79/0xd0
Feb 24 04:44:44 genetti kernel: [14121947.140448]  [<ffffffff811cb7d6>] ? kmem_cache_alloc+0xd6/0x1a0
Feb 24 04:44:44 genetti kernel: [14121947.140476]  [<ffffffff811f7cd1>] ? getname_flags+0x51/0x1f0
Feb 24 04:44:44 genetti kernel: [14121947.140502]  [<ffffffff816ddb99>] ? _raw_spin_unlock+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.140530]  [<ffffffff81206979>] ? __alloc_fd+0xa9/0x160
Feb 24 04:44:44 genetti kernel: [14121947.140557]  [<ffffffff811e6e91>] do_sys_open+0x111/0x1f0
Feb 24 04:44:44 genetti kernel: [14121947.140584]  [<ffffffff811e6f89>] SyS_open+0x19/0x20
Feb 24 04:44:44 genetti kernel: [14121947.140610]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.140654] zvol            D    0   958      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.140682]  ffff881fee27abd8 ffff8829972d8000 ffff881fee27a700 ffff883fff498080
Feb 24 04:44:44 genetti kernel: [14121947.140731]  ffff883ff2b41a00 ffffc9001ba63b90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.140781]  ffffc9001ba63b98 ffffffff816db199 ffff881fee27a700 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.143499] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.143522]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.143550]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.143579]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.143610]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.143639]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.143669]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.143750]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.143808]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.143867]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.143897]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.143926]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.143955]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.143983]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.144009]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.144036]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.144242] z_zvol          D    0  4056      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.144270]  ffff881fe5112bd8 ffff883f7f298000 ffff881fe5112700 ffff881fffa98080
Feb 24 04:44:44 genetti kernel: [14121947.144320]  ffff881ff295e800 ffffc9001ce8fb98 ffffffff816d85e3 ffffc9001ce8fb68
Feb 24 04:44:44 genetti kernel: [14121947.144370]  ffffffffa0458cd5 ffffc9001ce8fb88 ffff881fe5112700 ffff881fd1c85620
Feb 24 04:44:44 genetti kernel: [14121947.144419] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.144442]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.144470]  [<ffffffffa0458cd5>] ? spl_kmem_free+0x25/0x30 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.144499]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.144525]  [<ffffffff816d8d59>] schedule_preempt_disabled+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.144554]  [<ffffffff816db3c2>] __mutex_lock_slowpath+0xf2/0x2c0
Feb 24 04:44:44 genetti kernel: [14121947.144582]  [<ffffffff816db5a5>] mutex_lock+0x15/0x30
Feb 24 04:44:44 genetti kernel: [14121947.144637]  [<ffffffffa0755432>] zvol_remove_minors_impl+0x152/0x3b0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.144709]  [<ffffffffa07564b9>] zvol_task_cb+0x289/0x5b0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.144738]  [<ffffffff816d85eb>] ? __schedule+0x17b/0x610
Feb 24 04:44:44 genetti kernel: [14121947.144767]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.144795]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.144824]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.144851]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.144877]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.144909]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.145145] txg_sync        D    0 11819      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.145173]  ffff881fe4e711d8 ffff881fec8cd800 ffff881fe4e70d00 ffff883fff2d8080
Feb 24 04:44:44 genetti kernel: [14121947.145223]  ffff883ff2b1db00 ffffc9005ba2fae8 ffffffff816d85e3 ffffc9005ba2faf0
Feb 24 04:44:44 genetti kernel: [14121947.145273]  ffffffffa06e077f ffffc9005ba2fad0 ffff881fe4e70d00 ffff883fff2d8080
Feb 24 04:44:44 genetti kernel: [14121947.145322] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.145345]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.145401]  [<ffffffffa06e077f>] ? spa_taskq_dispatch_ent+0x6f/0x90 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.145434]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.145464]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.145523]  [<ffffffffa0740d1d>] ? zio_issue_async+0xd/0x20 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.145554]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.145583]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.145610]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.145643]  [<ffffffffa0460386>] cv_wait_common+0xa6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.145671]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.145701]  [<ffffffffa0460453>] __cv_wait_io+0x13/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.145757]  [<ffffffffa0745632>] zio_wait+0xf2/0x190 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.145811]  [<ffffffffa06c26d4>] dsl_pool_sync+0xa4/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.145867]  [<ffffffffa06dece5>] spa_sync+0x3e5/0xd30 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.145895]  [<ffffffff810d131d>] ? default_wake_function+0xd/0x10
Feb 24 04:44:44 genetti kernel: [14121947.145953]  [<ffffffffa06f1d64>] txg_sync_thread+0x2d4/0x470 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.146010]  [<ffffffffa06f1a90>] ? txg_init+0x260/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.146040]  [<ffffffffa045b356>] thread_generic_wrapper+0x76/0x90 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.146071]  [<ffffffffa045b2e0>] ? __thread_exit+0x20/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.146099]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.146125]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.146152]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.146601] qemu:servaali.i D    0 19528      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.146629]  ffff881f6da084d8 ffff883ff22d0000 ffff881f6da08000 ffff881fff8d8080
Feb 24 04:44:44 genetti kernel: [14121947.146679]  ffff881ff2932700 ffffc90051e1fbe8 ffffffff816d85e3 00000000efbfe130
Feb 24 04:44:44 genetti kernel: [14121947.146728]  0000000000000000 ffff883feea66ed0 ffff881f6da08000 ffff881fff8d8080
Feb 24 04:44:44 genetti kernel: [14121947.146778] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.146800]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.146827]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.146853]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.146880]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.146909]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.146937]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.146965]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.146993]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.147020]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.147048]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.147076]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.147103]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.147132]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.147159]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.147185]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.147212]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.147238]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.147301] qemu:grisoni.iu D    0  7658      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.147329]  ffff883f459891d8 ffff883fa0b00000 ffff883f45988d00 ffff883fff218080
Feb 24 04:44:44 genetti kernel: [14121947.147379]  ffff883ff2b18d00 ffffc90054a73be8 ffffffff816d85e3 00000000d1a13530
Feb 24 04:44:44 genetti kernel: [14121947.147428]  0000000000000000 ffff883feea66ed0 ffff883f45988d00 ffff883fff218080
Feb 24 04:44:44 genetti kernel: [14121947.147486] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.147508]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.147535]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.147561]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.147588]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.147615]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.147643]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.147671]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.147697]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.147724]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.147752]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.147779]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.147806]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.147841]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.147867]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.147893]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.147920]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.147946]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.147994] qemu:jaguarundi D    0 33036      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.148022]  ffff8819a8a46cd8 ffff883f7f298000 ffff8819a8a46800 ffff881fffa18080
Feb 24 04:44:44 genetti kernel: [14121947.148075]  ffff881ff2936800 ffffc900573bbbe8 ffffffff816d85e3 00000000efbfed30
Feb 24 04:44:44 genetti kernel: [14121947.148125]  0000000000000000 ffff883feea66ed0 ffff8819a8a46800 ffff881fffa18080
Feb 24 04:44:44 genetti kernel: [14121947.148174] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.148196]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.148223]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.148249]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.148280]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.148307]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.148344]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.148372]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.148398]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.148425]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.148455]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.148483]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.148509]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.148536]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.148563]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.148589]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.148616]  [<ffffffff8116f4d6>] ? fire_user_return_notifiers+0x36/0x40
Feb 24 04:44:44 genetti kernel: [14121947.148645]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.148671]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.148717] qemu:orion.sens D    0 21647      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.148745]  ffff8836294091d8 ffff883f4b408000 ffff883629408d00 ffff883fff158080
Feb 24 04:44:44 genetti kernel: [14121947.148795]  ffff883ff29ba700 ffffc9005bef7be8 ffffffff816d85e3 00000000a32bf930
Feb 24 04:44:44 genetti kernel: [14121947.148844]  0000000000000000 ffff883feea66ed0 ffff883629408d00 ffff883fff158080
Feb 24 04:44:44 genetti kernel: [14121947.148894] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.148916]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.148943]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.148968]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.148996]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.149022]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.149050]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.149078]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.149105]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.149131]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.149159]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.149187]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.149213]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.149240]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.149267]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.149293]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.149319]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.149345]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.149390] qemu:malbolge.c D    0  6208      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.149418]  ffff881907db84d8 ffff881f105e0000 ffff881907db8000 ffff881fff918080
Feb 24 04:44:44 genetti kernel: [14121947.149472]  ffff881ff2933400 ffffc90066f03be8 ffffffff816d85e3 00000000a32bf530
Feb 24 04:44:44 genetti kernel: [14121947.149533]  0000000000000000 ffff883feea66ed0 ffff881907db8000 ffff881fff918080
Feb 24 04:44:44 genetti kernel: [14121947.149587] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.152199]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.152226]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.152252]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.152279]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.152312]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.152340]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.152368]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.152395]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.152422]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.152450]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.152477]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.152504]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.152531]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.152558]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.152584]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.152611]  [<ffffffff8116f4d6>] ? fire_user_return_notifiers+0x36/0x40
Feb 24 04:44:44 genetti kernel: [14121947.152644]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.152670]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.152741] qemu:www1.xxxxx D    0 34296      1 0x00000004
Feb 24 04:44:44 genetti kernel: [14121947.152769]  ffff881fd4dd84d8 ffff881edd570000 ffff881fd4dd8000 ffff881fffb18080
Feb 24 04:44:44 genetti kernel: [14121947.152818]  ffff881ff295ce00 ffffc9005197fc60 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.152868]  ffffc9005197fc68 ffffffff816db199 ffff881fd4dd8000 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.152918] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.152939]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.152967]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.152995]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.153033]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.153067]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.153096]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.153151]  [<ffffffffa06f1484>] txg_wait_synced+0xe4/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.153207]  [<ffffffffa07402ca>] zil_close+0x9a/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.153262]  [<ffffffffa0753854>] zvol_shutdown_zv+0x14/0xa0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.153331]  [<ffffffffa0756e5a>] zvol_release+0xfa/0x1e0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.153359]  [<ffffffff81222ae9>] __blkdev_put+0x219/0x270
Feb 24 04:44:44 genetti kernel: [14121947.153387]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.153416]  [<ffffffff812233d7>] blkdev_put+0x47/0x140
Feb 24 04:44:44 genetti kernel: [14121947.153442]  [<ffffffff81223580>] blkdev_close+0x20/0x30
Feb 24 04:44:44 genetti kernel: [14121947.153475]  [<ffffffff811ea06a>] __fput+0xda/0x1e0
Feb 24 04:44:44 genetti kernel: [14121947.153504]  [<ffffffff811ea1a9>] ____fput+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.153532]  [<ffffffff810c5919>] task_work_run+0x79/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.153559]  [<ffffffff8100253a>] exit_to_usermode_loop+0x9a/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.153591]  [<ffffffff810029b8>] syscall_return_slowpath+0x98/0x110
Feb 24 04:44:44 genetti kernel: [14121947.153619]  [<ffffffff816dde9f>] entry_SYSCALL_64_fastpath+0x92/0x94
Feb 24 04:44:44 genetti kernel: [14121947.153662] qemu:kinkaju.iu D    0 25001      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.153690]  ffff881c6f6fecd8 ffff881fa2728000 ffff881c6f6fe800 ffff881fffb58080
Feb 24 04:44:44 genetti kernel: [14121947.153740]  ffff881ff295c100 ffffc9001dda3be8 ffffffff816d85e3 00000000d1c80530
Feb 24 04:44:44 genetti kernel: [14121947.153789]  0000000000000000 ffff883feea66ed0 ffff881c6f6fe800 ffff881fffb58080
Feb 24 04:44:44 genetti kernel: [14121947.153839] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.153861]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.153888]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.153914]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.153941]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.153968]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.153996]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.154024]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.154052]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.154079]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.154107]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.154134]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.154162]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.154189]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.154215]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.154242]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.154268]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.154294]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.154360] qemu:opossumi.q D    0 21309      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.154388]  ffff881c3544abd8 ffff881edfe58000 ffff881c3544a700 ffff881fff998080
Feb 24 04:44:44 genetti kernel: [14121947.154438]  ffff881fb9958d00 ffffc9005da1fbe8 ffffffff816d85e3 00000000ec8db930
Feb 24 04:44:44 genetti kernel: [14121947.154488]  0000000000000000 ffff883feea66ed0 ffff881c3544a700 ffff881fff998080
Feb 24 04:44:44 genetti kernel: [14121947.154537] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.154559]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.154586]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.154612]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.154639]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.154666]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.154694]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.154722]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.154749]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.154776]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.154804]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.154831]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.154858]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.154885]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.154912]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.154938]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.154964]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.154990]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.155069] qemu:hellkitty. D    0 35278      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.155097]  ffff881e64bd11d8 ffff883dc8418000 ffff881e64bd0d00 ffff881fff858080
Feb 24 04:44:44 genetti kernel: [14121947.155147]  ffff881e64bd0000 ffffc900549c7be8 ffffffff816d85e3 00000000ca6c8d30
Feb 24 04:44:44 genetti kernel: [14121947.155196]  0000000000000000 ffff883feea66ed0 ffff881e64bd0d00 ffff881fff858080
Feb 24 04:44:44 genetti kernel: [14121947.155246] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.155268]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.155295]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.155320]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.155348]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.155374]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.155402]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.155431]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.155458]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.155485]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.155513]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.155540]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.155567]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.155594]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.155621]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.155647]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.155674]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.155700]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.155791] qemu:nena.insom D    0  6110      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.155819]  ffff88105e381ed8 ffff881f30158000 ffff88105e381a00 ffff881fffa98080
Feb 24 04:44:44 genetti kernel: [14121947.155869]  ffff881ff295e800 ffffc9005334fbe8 ffffffff816d85e3 00000000cfb7c130
Feb 24 04:44:44 genetti kernel: [14121947.155919]  0000000000000000 ffff883feea66ed0 ffff88105e381a00 ffff881fffa98080
Feb 24 04:44:44 genetti kernel: [14121947.155968] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.155990]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.156017]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.156042]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.156070]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.156096]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.156124]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.156152]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.156178]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.156205]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.156233]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.156260]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.156287]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.156314]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.156340]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.156367]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.156393]  [<ffffffff8116f4d6>] ? fire_user_return_notifiers+0x36/0x40
Feb 24 04:44:44 genetti kernel: [14121947.156422]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.156448]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.156553] qemu:ikneumoni. D    0 36595      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.156581]  ffff8810492b1ed8 ffff880179ef0000 ffff8810492b1a00 ffff883fff198080
Feb 24 04:44:44 genetti kernel: [14121947.156631]  ffff883ff112b400 ffffc9005ea3fbe8 ffffffff816d85e3 00000000ef9db930
Feb 24 04:44:44 genetti kernel: [14121947.156680]  0000000000000000 ffff883feea66ed0 ffff8810492b1a00 ffff883fff198080
Feb 24 04:44:44 genetti kernel: [14121947.156730] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.156752]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.156779]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.156804]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.156844]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.156871]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.156898]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.156926]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.156953]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.156980]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.157008]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.157035]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.157062]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.157089]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.157115]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.157141]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.157177]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.157203]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.157251] qemu:atlas.sens D    0  8691      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.157278]  ffff8821d64d9ed8 ffff8829972d8000 ffff8821d64d9a00 ffff883fff398080
Feb 24 04:44:44 genetti kernel: [14121947.157328]  ffff883ff2b1b400 ffffc9005fe6fbe8 ffffffff816d85e3 00000000e6589d30
Feb 24 04:44:44 genetti kernel: [14121947.157378]  0000000000000000 ffff883feea66ed0 ffff8821d64d9a00 ffff883fff398080
Feb 24 04:44:44 genetti kernel: [14121947.157427] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.160061]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.160104]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.160132]  [<ffffffff816dca3b>] schedule_timeout+0x1fb/0x460
Feb 24 04:44:44 genetti kernel: [14121947.160163]  [<ffffffff81108e66>] ? ktime_get+0x36/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.160190]  [<ffffffff816d83f6>] io_schedule_timeout+0xb6/0x130
Feb 24 04:44:44 genetti kernel: [14121947.160219]  [<ffffffff816d93c0>] wait_for_completion_io+0xa0/0x100
Feb 24 04:44:44 genetti kernel: [14121947.160247]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.160273]  [<ffffffff812f82d1>] submit_bio_wait+0x51/0x60
Feb 24 04:44:44 genetti kernel: [14121947.160300]  [<ffffffff81309655>] blkdev_issue_discard+0x65/0xa0
Feb 24 04:44:44 genetti kernel: [14121947.160328]  [<ffffffff8130f0d8>] blk_ioctl_discard+0x78/0x90
Feb 24 04:44:44 genetti kernel: [14121947.160356]  [<ffffffff8130fc4b>] blkdev_ioctl+0x71b/0x8c0
Feb 24 04:44:44 genetti kernel: [14121947.160383]  [<ffffffff811e7773>] ? __vfs_write+0x23/0x120
Feb 24 04:44:44 genetti kernel: [14121947.160411]  [<ffffffff812f9869>] ? bio_free+0x49/0x60
Feb 24 04:44:44 genetti kernel: [14121947.160439]  [<ffffffff81221da8>] block_ioctl+0x38/0x40
Feb 24 04:44:44 genetti kernel: [14121947.160468]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.160496]  [<ffffffff8116f4d6>] ? fire_user_return_notifiers+0x36/0x40
Feb 24 04:44:44 genetti kernel: [14121947.160525]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.160551]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.160801] zfs             D    0 22834  22007 0x00000004
Feb 24 04:44:44 genetti kernel: [14121947.160829]  ffff88147f6d2bd8 ffff881fed25fc00 ffff88147f6d2700 ffff881fffb58080
Feb 24 04:44:44 genetti kernel: [14121947.160879]  ffff881ff295c100 ffffc9005bd5f840 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.160928]  ffffc9005bd5f848 ffffffff816db199 ffff88147f6d2700 ffff88050f8c1030
Feb 24 04:44:44 genetti kernel: [14121947.160978] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.161000]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.161028]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.161056]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.161086]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.161114]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.161144]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.161193]  [<ffffffffa068de10>] bqueue_enqueue+0x60/0xd0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161245]  [<ffffffffa06a1b3e>] dmu_recv_stream+0x66e/0x1110 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161300]  [<ffffffffa06cac08>] ? dsl_sync_task+0x228/0x240 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161352]  [<ffffffffa069b220>] ? receive_cksum+0x30/0x30 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161410]  [<ffffffffa0722af4>] zfs_ioc_recv_impl+0xda4/0x10f0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161460]  [<ffffffffa0687c19>] ? dmu_buf_rele+0x9/0x10 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161522]  [<ffffffffa0708044>] ? zap_put_leaf+0x34/0x40 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161551]  [<ffffffff810d6997>] ? __enqueue_entity+0x67/0x70
Feb 24 04:44:44 genetti kernel: [14121947.161579]  [<ffffffff810d7bc3>] ? select_task_rq_fair+0x723/0xa70
Feb 24 04:44:44 genetti kernel: [14121947.161637]  [<ffffffffa0722ffa>] zfs_ioc_recv+0x1ba/0x2a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161666]  [<ffffffff811cc68c>] ? __slab_free+0x9c/0x2a0
Feb 24 04:44:44 genetti kernel: [14121947.161693]  [<ffffffff816ddb99>] ? _raw_spin_unlock+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.161721]  [<ffffffff8117ad67>] ? get_page_from_freelist+0x397/0xa40
Feb 24 04:44:44 genetti kernel: [14121947.161749]  [<ffffffff8117ad67>] ? get_page_from_freelist+0x397/0xa40
Feb 24 04:44:44 genetti kernel: [14121947.161808]  [<ffffffffa0720672>] zfsdev_ioctl+0x5a2/0x640 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.161836]  [<ffffffff811a75f2>] ? handle_mm_fault+0x342/0xfd0
Feb 24 04:44:44 genetti kernel: [14121947.161864]  [<ffffffff811acdf2>] ? vma_merge+0x202/0x340
Feb 24 04:44:44 genetti kernel: [14121947.161891]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.161918]  [<ffffffff81099720>] ? __do_page_fault+0x220/0x430
Feb 24 04:44:44 genetti kernel: [14121947.161946]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.161972]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.162004] receive_writer  D    0 28521      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.162032]  ffff881febcd11d8 ffff883fee514c00 ffff881febcd0d00 ffff881fff818080
Feb 24 04:44:44 genetti kernel: [14121947.162082]  ffffffff81c0d500 ffffc9001d297c20 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.162132]  ffffc9001d297c28 ffffffff816db199 ffff881febcd0d00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.162182] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.162204]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.162231]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.162260]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.162288]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.162317]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.162346]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.162403]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.162453]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.162505]  [<ffffffffa069fc61>] receive_free.isra.11+0x91/0xa0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.162557]  [<ffffffffa06a0053>] receive_writer_thread+0x313/0xce0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.162588]  [<ffffffffa0458cd5>] ? spl_kmem_free+0x25/0x30 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.162617]  [<ffffffff811ccf2d>] ? kfree+0x14d/0x160
Feb 24 04:44:44 genetti kernel: [14121947.162673]  [<ffffffffa069fd40>] ? receive_freeobjects.isra.12+0xd0/0xd0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.162722]  [<ffffffffa045b356>] thread_generic_wrapper+0x76/0x90 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.162754]  [<ffffffffa045b2e0>] ? __thread_exit+0x20/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.162782]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.162808]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.162835]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.162862]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.162907] zvol            D    0  1042      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.162935]  ffff88375c971ed8 ffff883e36c60000 ffff88375c971a00 ffff883fff118080
Feb 24 04:44:44 genetti kernel: [14121947.162985]  ffff883ff29bb400 ffffc9008402bb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.163035]  ffffc9008402bb98 ffffffff816db199 ffff88375c971a00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.163084] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.163106]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.163134]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.163163]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.163192]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.163220]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.163250]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.163306]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.163355]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.163413]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.163443]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.163471]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.163499]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.163535]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.163561]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.163596]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.163622]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.163651] zvol            D    0  1044      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.163679]  ffff88375c972bd8 ffff883f4b408000 ffff88375c972700 ffff883fff458080
Feb 24 04:44:44 genetti kernel: [14121947.163729]  ffff883ff2b40d00 ffffc9008403bb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.163778]  ffffc9008403bb98 ffffffff816db199 ffff88375c972700 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.163828] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.163850]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.163877]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.163913]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.163941]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.163969]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.163998]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164066]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164114]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164171]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164201]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164229]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.164257]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164284]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.164309]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.164336]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.164368] zvol            D    0  1051      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.164396]  ffff881ff08f1ed8 ffff883dc8418000 ffff881ff08f1a00 ffff881fff858080
Feb 24 04:44:44 genetti kernel: [14121947.164445]  ffff881ff2930d00 ffffc90084053b90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.164505]  ffffc90084053b98 ffffffff816db199 ffff881ff08f1a00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.164554] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.164576]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.164603]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.164632]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.164660]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164688]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.164717]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164773]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164835]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164891]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.164921]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.164948]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.164977]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.165004]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.165029]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.165056]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.165083] zvol            D    0  1052      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.165111]  ffff881ff08f52d8 ffff883fed72f400 ffff881ff08f4e00 ffff881fffa98080
Feb 24 04:44:44 genetti kernel: [14121947.165160]  ffff881ff295e800 ffffc9008409bb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.165210]  ffffc9008409bb98 ffffffff816db199 ffff881ff08f4e00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.165259] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.165281]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.165308]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.165337]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.165365]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.165393]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.165430]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.165486]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.165546]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.165606]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.165641]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.165669]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.165698]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.165725]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.165751]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.165780]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.165808]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.165836] zvol            D    0  1053      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.165863]  ffff881ff08f04d8 ffff883fa0b00000 ffff881ff08f0000 ffff883fff358080
Feb 24 04:44:44 genetti kernel: [14121947.165913]  ffff883ff2b1c100 ffffc900840a3b90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.168542]  ffffc900840a3b98 ffffffff816db199 ffff881ff08f0000 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.168604] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.168626]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.168653]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.168682]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.168710]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.168738]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.168767]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.168823]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.168873]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.168930]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.168969]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.168997]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.169025]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.169052]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.169087]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.169113]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.169140]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.169169] zvol            D    0  1055      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.169196]  ffff881fe7b65fd8 ffff881edd570000 ffff881fe7b65b00 ffff881fffb98080
Feb 24 04:44:44 genetti kernel: [14121947.169246]  ffff881ff295b400 ffffc900840b3b90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.169296]  ffffc900840b3b98 ffffffff816db199 ffff881fe7b65b00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.169345] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.169367]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.169394]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.169423]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.169451]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.169483]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.169513]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.169569]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.169622]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.169679]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.169709]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.169737]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.169765]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.169793]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.169819]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.169845]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.169872]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.169902] zvol            D    0  1057      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.169931]  ffff881fefec5fd8 ffff883ff22d0000 ffff881fefec5b00 ffff881fff958080
Feb 24 04:44:44 genetti kernel: [14121947.169980]  ffff881ff2934100 ffffc900829bfb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.170030]  ffffc900829bfb98 ffffffff816db199 ffff881fefec5b00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.170080] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.170103]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.170130]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.170159]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.170187]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.170215]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.170244]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.170300]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.170350]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.170408]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.170438]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.170466]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.170495]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.170522]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.170548]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.170574]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.170601]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.170639] zvol            D    0  1070      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.170667]  ffff881fd5cc38d8 ffff883fa0b00000 ffff881fd5cc3400 ffff883fff3d8080
Feb 24 04:44:44 genetti kernel: [14121947.170716]  ffff883ff2b1a700 ffffc900840fbb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.170766]  ffffc900840fbb98 ffffffff816db199 ffff881fd5cc3400 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.170816] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.170844]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.170871]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.170900]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.170928]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.170956]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.170986]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171042]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171092]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171149]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171179]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171207]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.171235]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171263]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.171288]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.171315]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.171344] zvol            D    0  1072      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.171372]  ffff881fd5cc52d8 ffff881ac7b48000 ffff881fd5cc4e00 ffff881fff818080
Feb 24 04:44:44 genetti kernel: [14121947.171422]  ffffffff81c0d500 ffffc9008410bb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.171471]  ffffc9008410bb98 ffffffff816db199 ffff881fd5cc4e00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.171521] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.171543]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.171570]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.171599]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.171626]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171654]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.171684]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171739]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171788]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171846]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.171875]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171903]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.171932]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.171959]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.171985]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.172011]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.172038]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.172065] zvol            D    0  1073      2 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.172093]  ffff881fd43e52d8 ffff883f97c98000 ffff881fd43e4e00 ffff881fffa98080
Feb 24 04:44:44 genetti kernel: [14121947.172143]  ffff881ff295e800 ffffc900831ffb90 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.172192]  ffffc900831ffb98 ffffffff816db199 ffff881fd43e4e00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.172242] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.172264]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.172291]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.172320]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.172347]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.172375]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.172405]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.172460]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.172510]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.172567]  [<ffffffffa0756a07>] zvol_discard+0x227/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.172597]  [<ffffffffa045c33f>] taskq_thread+0x2bf/0x4e0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.172624]  [<ffffffff810d1310>] ? wake_up_q+0x70/0x70
Feb 24 04:44:44 genetti kernel: [14121947.172653]  [<ffffffffa045c080>] ? task_done+0xc0/0xc0 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.172680]  [<ffffffff810c7239>] kthread+0xf9/0x110
Feb 24 04:44:44 genetti kernel: [14121947.172706]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.172732]  [<ffffffff810c7140>] ? kthread_park+0x60/0x60
Feb 24 04:44:44 genetti kernel: [14121947.172759]  [<ffffffff816de052>] ret_from_fork+0x22/0x30
Feb 24 04:44:44 genetti kernel: [14121947.172787] zfs             D    0  5483      1 0x00000004
Feb 24 04:44:44 genetti kernel: [14121947.172814]  ffff883fa4136cd8 ffff883fee627c00 ffff883fa4136800 ffff883fff218080
Feb 24 04:44:44 genetti kernel: [14121947.172864]  ffff883ff2b18d00 ffffc90065773ae8 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.172914]  ffffc90065773af0 ffffffff816db199 ffff883fa4136800 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.172963] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.172985]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.173012]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.173050]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.173078]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.173106]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.173135]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.173189]  [<ffffffffa06f1484>] txg_wait_synced+0xe4/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173240]  [<ffffffffa06b1a60>] ? dsl_dataset_snapshot_check_impl+0x1c0/0x1c0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173310]  [<ffffffffa06cab29>] dsl_sync_task+0x149/0x240 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173361]  [<ffffffffa06b2570>] ? dsl_dataset_snapshot_sync_impl+0x6a0/0x6a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173442]  [<ffffffffa06b1a60>] ? dsl_dataset_snapshot_check_impl+0x1c0/0x1c0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173519]  [<ffffffffa06b2570>] ? dsl_dataset_snapshot_sync_impl+0x6a0/0x6a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173592]  [<ffffffffa06b28e4>] dsl_dataset_snapshot+0x114/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173626]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.173656]  [<ffffffff816ddb99>] ? _raw_spin_unlock+0x9/0x10
Feb 24 04:44:44 genetti kernel: [14121947.173711]  [<ffffffffa06e34c9>] ? spa_open_common+0x319/0x4a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173740]  [<ffffffff810b2c83>] ? ns_capable_common+0x63/0x80
Feb 24 04:44:44 genetti kernel: [14121947.173773]  [<ffffffffa047d9f8>] ? nvlist_lookup_common.part.71+0x78/0x80 [znvpair]
Feb 24 04:44:44 genetti kernel: [14121947.173846]  [<ffffffffa071e94a>] zfs_ioc_snapshot+0x32a/0x380 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173904]  [<ffffffffa07202e3>] zfsdev_ioctl+0x213/0x640 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.173932]  [<ffffffff811a75f2>] ? handle_mm_fault+0x342/0xfd0
Feb 24 04:44:44 genetti kernel: [14121947.173960]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.173987]  [<ffffffff81099720>] ? __do_page_fault+0x220/0x430
Feb 24 04:44:44 genetti kernel: [14121947.174014]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.174040]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.174073] zfs             D    0 19842      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.174101]  ffff881fd4002bd8 ffff883fee4e0c00 ffff881fd4002700 ffff883fff118080
Feb 24 04:44:44 genetti kernel: [14121947.174151]  ffff883ff29bb400 ffffc9005f907bb8 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.176871]  ffffc9005f907bc0 ffffffff816db199 ffff881fd4002700 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.176929] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.176951]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.176978]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.177007]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.177035]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.177064]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.177094]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.177148]  [<ffffffffa06f1484>] txg_wait_synced+0xe4/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177201]  [<ffffffffa06c41e0>] ? dsl_prop_inherit+0x50/0x50 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177254]  [<ffffffffa06cab29>] dsl_sync_task+0x149/0x240 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177306]  [<ffffffffa06c60a0>] ? dsl_props_set_sync_impl+0x140/0x140 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177373]  [<ffffffffa06c41e0>] ? dsl_prop_inherit+0x50/0x50 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177439]  [<ffffffffa06c60a0>] ? dsl_props_set_sync_impl+0x140/0x140 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177512]  [<ffffffffa06c40bb>] dsl_props_set+0x4b/0x60 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177571]  [<ffffffffa0721b27>] zfs_set_prop_nvlist+0x3b7/0x4c0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177602]  [<ffffffffa047b635>] ? nv_mem_zalloc.isra.12+0x25/0x30 [znvpair]
Feb 24 04:44:44 genetti kernel: [14121947.177675]  [<ffffffffa072358b>] zfs_ioc_set_prop+0xeb/0x120 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177732]  [<ffffffffa0720672>] zfsdev_ioctl+0x5a2/0x640 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.177760]  [<ffffffff811a75f2>] ? handle_mm_fault+0x342/0xfd0
Feb 24 04:44:44 genetti kernel: [14121947.177788]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.177815]  [<ffffffff81099720>] ? __do_page_fault+0x220/0x430
Feb 24 04:44:44 genetti kernel: [14121947.177842]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.177869]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.177900] zfs             D    0 31257  26756 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.177928]  ffff881ff0e1abd8 ffff883fecc1b400 ffff881ff0e1a700 ffff883fff0d8080
Feb 24 04:44:44 genetti kernel: [14121947.177978]  ffff883ff29bc100 ffffc90082ed7c28 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.178028]  ffffc90082ed7c30 ffffffff816db199 ffff881ff0e1a700 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.178078] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.178100]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.178127]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.178156]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.178184]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.178213]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.178242]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.178298]  [<ffffffffa06f1484>] txg_wait_synced+0xe4/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178355]  [<ffffffffa075a470>] ? dsl_destroy_snapshot_check_impl+0x130/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178426]  [<ffffffffa06cab29>] dsl_sync_task+0x149/0x240 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178483]  [<ffffffffa07594e0>] ? dsl_destroy_snapshot_sync_impl+0xba0/0xba0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178556]  [<ffffffffa075a470>] ? dsl_destroy_snapshot_check_impl+0x130/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178629]  [<ffffffffa07594e0>] ? dsl_destroy_snapshot_sync_impl+0xba0/0xba0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178701]  [<ffffffffa075a5eb>] dsl_destroy_snapshots_nvl+0x7b/0xd0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178772]  [<ffffffffa0724040>] zfs_ioc_destroy_snaps+0xa0/0xb0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178829]  [<ffffffffa07202e3>] zfsdev_ioctl+0x213/0x640 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.178857]  [<ffffffff811a75f2>] ? handle_mm_fault+0x342/0xfd0
Feb 24 04:44:44 genetti kernel: [14121947.178885]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.178912]  [<ffffffff81099720>] ? __do_page_fault+0x220/0x430
Feb 24 04:44:44 genetti kernel: [14121947.178939]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.178965]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.179065] zfs             D    0 10135      1 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.179093]  ffff883fa9db11d8 ffff883fecc1d000 ffff883fa9db0d00 ffff883fff0d8080
Feb 24 04:44:44 genetti kernel: [14121947.179143]  ffff883ff29bc100 ffffc90058373ae8 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.179204]  ffffc90058373af0 ffffffff816db199 ffff883fa9db0d00 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.179254] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.179276]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.179303]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.179332]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.179360]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.179389]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.179418]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.179473]  [<ffffffffa06f1484>] txg_wait_synced+0xe4/0x130 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179543]  [<ffffffffa06b1a60>] ? dsl_dataset_snapshot_check_impl+0x1c0/0x1c0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179614]  [<ffffffffa06cab29>] dsl_sync_task+0x149/0x240 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179666]  [<ffffffffa06b2570>] ? dsl_dataset_snapshot_sync_impl+0x6a0/0x6a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179746]  [<ffffffffa06b1a60>] ? dsl_dataset_snapshot_check_impl+0x1c0/0x1c0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179814]  [<ffffffffa06b2570>] ? dsl_dataset_snapshot_sync_impl+0x6a0/0x6a0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179882]  [<ffffffffa06b28e4>] dsl_dataset_snapshot+0x114/0x260 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.179912]  [<ffffffff811cc23a>] ? __kmalloc_node+0x1fa/0x2c0
Feb 24 04:44:44 genetti kernel: [14121947.179941]  [<ffffffffa04591ec>] ? spl_kmem_alloc_impl+0xac/0x180 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.179972]  [<ffffffffa04591ec>] ? spl_kmem_alloc_impl+0xac/0x180 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.180003]  [<ffffffffa047d9f8>] ? nvlist_lookup_common.part.71+0x78/0x80 [znvpair]
Feb 24 04:44:44 genetti kernel: [14121947.180087]  [<ffffffffa071e94a>] zfs_ioc_snapshot+0x32a/0x380 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180143]  [<ffffffffa07202e3>] zfsdev_ioctl+0x213/0x640 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180171]  [<ffffffff811a75f2>] ? handle_mm_fault+0x342/0xfd0
Feb 24 04:44:44 genetti kernel: [14121947.180199]  [<ffffffff811fbbfb>] do_vfs_ioctl+0x8b/0x5b0
Feb 24 04:44:44 genetti kernel: [14121947.180225]  [<ffffffff81099720>] ? __do_page_fault+0x220/0x430
Feb 24 04:44:44 genetti kernel: [14121947.180253]  [<ffffffff811fc194>] SyS_ioctl+0x74/0x80
Feb 24 04:44:44 genetti kernel: [14121947.180279]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
Feb 24 04:44:44 genetti kernel: [14121947.180311] rm              D    0 12095   9183 0x00000000
Feb 24 04:44:44 genetti kernel: [14121947.180339]  ffff88067b36abd8 ffff883fee540400 ffff88067b36a700 ffff883fff1d8080
Feb 24 04:44:44 genetti kernel: [14121947.180388]  ffff883ff2b18000 ffffc900870cfc18 ffffffff816d85e3 0000000000000246
Feb 24 04:44:44 genetti kernel: [14121947.180438]  ffffc900870cfc20 ffffffff816db199 ffff88067b36a700 ffff883fefe292b0
Feb 24 04:44:44 genetti kernel: [14121947.180487] Call Trace:
Feb 24 04:44:44 genetti kernel: [14121947.180510]  [<ffffffff816d85e3>] ? __schedule+0x173/0x610
Feb 24 04:44:44 genetti kernel: [14121947.180537]  [<ffffffff816db199>] ? __mutex_unlock_slowpath+0xa9/0x140
Feb 24 04:44:44 genetti kernel: [14121947.180565]  [<ffffffff816d8ab1>] schedule+0x31/0x80
Feb 24 04:44:44 genetti kernel: [14121947.180594]  [<ffffffffa04603c6>] cv_wait_common+0xe6/0x120 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.180622]  [<ffffffff810e4ff0>] ? wake_up_atomic_t+0x30/0x30
Feb 24 04:44:44 genetti kernel: [14121947.180651]  [<ffffffffa0460410>] __cv_wait+0x10/0x20 [spl]
Feb 24 04:44:44 genetti kernel: [14121947.180706]  [<ffffffffa06f1675>] txg_wait_open+0xb5/0x100 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180754]  [<ffffffffa0690d71>] dmu_free_long_range+0x231/0x410 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180811]  [<ffffffffa07170c2>] zfs_rmnode+0x262/0x320 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180866]  [<ffffffffa0737182>] ? zfs_znode_hold_exit+0xf2/0x120 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180922]  [<ffffffffa073b4d7>] zfs_zinactive+0xc7/0xe0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.180990]  [<ffffffffa073318a>] zfs_inactive+0x6a/0x1e0 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.181019]  [<ffffffff81185705>] ? truncate_pagecache+0x55/0x60
Feb 24 04:44:44 genetti kernel: [14121947.181074]  [<ffffffffa0751113>] zpl_evict_inode+0x43/0x70 [zfs]
Feb 24 04:44:44 genetti kernel: [14121947.181103]  [<ffffffff81203f90>] evict+0xc0/0x190
Feb 24 04:44:44 genetti kernel: [14121947.181128]  [<ffffffff81204401>] iput+0x191/0x200
Feb 24 04:44:44 genetti kernel: [14121947.181154]  [<ffffffff811f87c4>] do_unlinkat+0x124/0x2c0
Feb 24 04:44:44 genetti kernel: [14121947.181180]  [<ffffffff811f9366>] SyS_unlinkat+0x16/0x30
Feb 24 04:44:44 genetti kernel: [14121947.181207]  [<ffffffff816dde20>] entry_SYSCALL_64_fastpath+0x13/0x94
@fcicq
Copy link

fcicq commented Feb 27, 2018

from the trace: looks like txg is stuck (txg_wait_open & txg_wait_synced)... and blkdev_issue_discard seems to be slow.

@rugubara
Copy link
Contributor

My experience maybe completely unrelated. I experienced a very noticeable slow down of write operations to my SSD pool when I ran master versions of ZOL that don't support TRIM last summer. I can see that you use these too now.
I solved my problem by switching to the branch https://github.com/dweeezil/zfs/tree/ntrim2-next by @dweeezil and running a TRIM.
Before trimming my devs, I could observe write operations take 0.5 - 4 seconds regardless of their size.
I've never experienced io slowdowns since then.
I use very cheap SSD for my system - ata-TS256GMTS400_C056900183

@lkateley
Copy link

I have been studying this symptom for the last week or 2.. and it seems to consistent occur with several different kinds of virtualization, proxmox, xen, vmware, and virtualbox.. I also have seen that it is mostly described as being bad disks. These are high quality sas drives. Will look at trim too.

On the systems I am working on I also just discovered this.

2018-03-14 05:04:47 lp_pool 255G 42.8G 0 276 0 24.2M
2018-03-14 05:04:52 lp_pool 255G 42.8G 0 242 0 29.7M
2018-03-14 05:04:57 lp_pool 255G 42.8G 2 198 9.60K 9.74M
2018-03-14 05:05:02 lp_pool 179G 119G 458 380 56.1M 9.08M
2018-03-14 05:05:07 lp_pool 179G 119G 504 119 62.0M 11.6M
2018-03-14 05:05:12 lp_pool 179G 119G 693 89 85.4M 10.8M
2018-03-14 05:05:17 lp_pool 179G 119G 482 269 57.6M 15.1M

The txg_sync hung at 5:04:50. It is stunning that the alloc and free change radically at 05:05:02. Can this just be an accounting issue with virtualized disks?

Here is the trace
Mar 14 05:04:50 server kernel: [76223.342494] INFO: task txg_sync:3546 blocked for more than 120 seconds.
Mar 14 05:04:50 server kernel: [76223.342555] Tainted: P O 4.4.0-31-generic #50-Ubuntu
Mar 14 05:04:50 server kernel: [76223.342603] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 14 05:04:50 server kernel: [76223.342659] txg_sync D ffff880c980c7aa8 0 3546 2 0x00000000
Mar 14 05:04:50 server kernel: [76223.342667] ffff880c980c7aa8 ffff881b39111b80 ffff8810b9858dc0 ffff880c98c76e00
Mar 14 05:04:50 server kernel: [76223.342673] ffff880c980c8000 ffff8810ba516d00 7fffffffffffffff ffff8817296c70c8
Mar 14 05:04:50 server kernel: [76223.342677] 0000000000000001 ffff880c980c7ac0 ffffffff81829a25 0000000000000000
Mar 14 05:04:50 server kernel: [76223.342681] Call Trace:
Mar 14 05:04:50 server kernel: [76223.342694] [] schedule+0x35/0x80
Mar 14 05:04:50 server kernel: [76223.342699] [] schedule_timeout+0x1b5/0x270
Mar 14 05:04:50 server kernel: [76223.342706] [] ? default_wake_function+0x12/0x20
Mar 14 05:04:50 server kernel: [76223.342711] [] ? __wake_up_common+0x52/0x90
Mar 14 05:04:50 server kernel: [76223.342717] [] io_schedule_timeout+0xa4/0x110
Mar 14 05:04:50 server kernel: [76223.342736] [] cv_wait_common+0xbc/0x140 [spl]
Mar 14 05:04:50 server kernel: [76223.342741] [] ? wake_atomic_t_function+0x60/0x60
Mar 14 05:04:50 server kernel: [76223.342752] [] __cv_wait_io+0x18/0x20 [spl]
Mar 14 05:04:50 server kernel: [76223.342871] [] zio_wait+0x10e/0x1f0 [zfs]
Mar 14 05:04:50 server kernel: [76223.342926] [] dsl_pool_sync+0xb8/0x430 [zfs]
Mar 14 05:04:50 server kernel: [76223.342989] [] spa_sync+0x366/0xb30 [zfs]
Mar 14 05:04:50 server kernel: [76223.342994] [] ? default_wake_function+0x12/0x20
Mar 14 05:04:50 server kernel: [76223.343061] [] txg_sync_thread+0x3ba/0x630 [zfs]
Mar 14 05:04:50 server kernel: [76223.343126] [] ? txg_delay+0x180/0x180 [zfs]
Mar 14 05:04:50 server kernel: [76223.343135] [] thread_generic_wrapper+0x71/0x80 [spl]
Mar 14 05:04:50 server kernel: [76223.343143] [] ? __thread_exit+0x20/0x20 [spl]
Mar 14 05:04:50 server kernel: [76223.343148] [] kthread+0xd8/0xf0
Mar 14 05:04:50 server kernel: [76223.343152] [] ? kthread_create_on_node+0x1e0/0x1e0
Mar 14 05:04:50 server kernel: [76223.343157] [] ret_from_fork+0x3f/0x70
Mar 14 05:04:50 server kernel: [76223.343160] [] ? kthread_create_on_node+0x1e0/0x1e0
Mar 14 05:04:50 server kernel: [76223.343175] INFO: task rs:main Q:Reg:5817 blocked for more than 120 seconds.

@behlendorf
Copy link
Contributor

There's no quite enough information in this issue to say for certain, but #7307 which has been merged could results in this lockup. @tuomari if possible could you test the latest master which has this fix applied and see if you're still able to reproduce the issue.

@tuomari
Copy link
Author

tuomari commented Apr 9, 2018

@behlendorf I finally got the code to the server, and was not able to reproduce this problem.

@tuomari tuomari closed this as completed Apr 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants