ZVOL Panics / VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed

[mgmartin]

I've had recent,consistent zvol panics. They seem to happen regularly when I use kvm with the zvol block device. Mine is set up with no compression, kvm's Cache Mode = none, using virtio scsi. I also tried putting cache mode back to kvm's default (I think it uses writeback), but also saw the panic which make it seem o_direct is not an issue.

I started having issues with an existing zvol, so I created a new zvol and immediately see the same issue during an os installation. My previous zvol would constantly panic just running software inside the vm.

I scrubbed and ran "zdb -mc" on the pool with no issues reported.

I'm using latest git versions (as of today) of spl and zfs. I'm running with 4.9.32 kernel on Arch Linux.

My steps to re-produce (using a clean zvol):

1. zfs create -o compression=off -o checksum=off -b 128K array1/test -V 50G
2. Set up a kvm guest to use /dev/zvol/array/test as the disk.
3. Install Centos 7 from iso--the vm will eventually cause a panic on the host during the install process.

(Update Find) It appears that if I don't use -b 128K and go with the default, then there are no panics. As soon as I re-create the test zvol with the -b 128K, the panics come back. I tested on two separate pools.

I've been getting regular panics, so I've been running a debug spl/zfs. Hopefully, this ASSERT message and stacks that appear will help:

[  470.174383] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[  470.174427] PANIC at dbuf.c:1304:dbuf_unoverride()
[  470.174431] Showing stack for process 15288
[  470.174440] CPU: 1 PID: 15288 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[  470.174445] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[  470.174452]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc9002d19bc90
[  470.174472]  ffffffffa0008d82 0000000000000002 0000000000000028 ffffc9002d19bca0
[  470.174486]  ffffc9002d19bc40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[  470.174499] Call Trace:
[  470.174507]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[  470.174514]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[  470.174520]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[  470.174523]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[  470.174547]  [<ffffffffa0214d49>] ? zio_done+0x999/0x1870 [zfs]
[  470.174554]  [<ffffffffa000bc18>] ? __cv_destroy+0x68/0x240 [spl]
[  470.174568]  [<ffffffffa0215e15>] ? zio_ready+0x1f5/0x5c0 [zfs]
[  470.174582]  [<ffffffffa02108ce>] ? zio_wait+0x1ee/0x3f0 [zfs]
[  470.174597]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[  470.174611]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[  470.174624]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[  470.174638]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[  470.174651]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[  470.174657]  [<ffffffff8102464f>] ? __switch_to+0x20f/0x490
[  470.174663]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[  470.174668]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[  470.174673]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[  470.174678]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[  470.174682]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[  470.174686]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30

And here's another stack of a prior panic with some more cores:

[ 1681.856652] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.856663] PANIC at dbuf.c:1304:dbuf_unoverride()
[ 1681.856666] Showing stack for process 27037
[ 1681.856671] CPU: 5 PID: 27037 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[ 1681.856675] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[ 1681.856679]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc90023517c90
[ 1681.856694]  ffffffffa0008d82 0000000000000002 0000000000000028 ffffc90023517ca0
[ 1681.856706]  ffffc90023517c40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[ 1681.856720] Call Trace:
[ 1681.856727]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[ 1681.856737]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[ 1681.856743]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.856764] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.856765] PANIC at dbuf.c:1304:dbuf_unoverride()
[ 1681.856765] Showing stack for process 27023
[ 1681.856815]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.856819]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.856838]  [<ffffffffa0214d49>] ? zio_done+0x999/0x1870 [zfs]
[ 1681.856842]  [<ffffffffa000bc18>] ? __cv_destroy+0x68/0x240 [spl]
[ 1681.856855]  [<ffffffffa0215e15>] ? zio_ready+0x1f5/0x5c0 [zfs]
[ 1681.856868]  [<ffffffffa02108ce>] ? zio_wait+0x1ee/0x3f0 [zfs]
[ 1681.856881]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[ 1681.856893]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[ 1681.856905]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[ 1681.856918]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[ 1681.856931]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[ 1681.856938]  [<ffffffff8102464f>] ? __switch_to+0x20f/0x490
[ 1681.856943]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[ 1681.856947]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[ 1681.856951]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[ 1681.856955]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[ 1681.856958]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.856961]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.856964]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30
[ 1681.856968] CPU: 1 PID: 27023 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[ 1681.856973] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[ 1681.856978]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc900236a7c90
[ 1681.856989]  ffffffffa0008d82 ffff8804ee500200 ffffffff00000028 ffffc900236a7ca0
[ 1681.857002]  ffffc900236a7c40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[ 1681.857013] Call Trace:
[ 1681.857018]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[ 1681.857023]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[ 1681.857034]  [<ffffffffa013397c>] ? dbuf_dirty+0xcdc/0x11e0 [zfs]
[ 1681.857047]  [<ffffffffa01ac4dd>] ? txg_verify+0x1d/0xf0 [zfs]
[ 1681.857050]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.857062]  [<ffffffffa0140db0>] ? dmu_objset_userused_enabled+0x20/0x50 [zfs]
[ 1681.857075]  [<ffffffffa0144be1>] ? dmu_objset_userquota_get_ids+0x51/0x4e0 [zfs]
[ 1681.857086]  [<ffffffffa0140d4b>] ? dnode_multilist_index_func+0xbb/0x100 [zfs]
[ 1681.857089]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.857101]  [<ffffffffa01876f5>] ? multilist_sublist_lock+0x45/0xa0 [zfs]
[ 1681.857113]  [<ffffffffa0156527>] ? dnode_setdirty+0xc7/0x3c0 [zfs]
[ 1681.857126]  [<ffffffffa01877a8>] ? multilist_sublist_unlock+0x28/0x30 [zfs]
[ 1681.857136]  [<ffffffffa01332ca>] ? dbuf_dirty+0x62a/0x11e0 [zfs]
[ 1681.857144]  [<ffffffffa011b27b>] ? arc_buf_alloc_impl+0x24b/0x5b0 [zfs]
[ 1681.857153]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[ 1681.857164]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[ 1681.857174]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[ 1681.857184]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[ 1681.857195]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[ 1681.857198]  [<ffffffff8102464f>] ? __switch_to+0x20f/0x490
[ 1681.857202]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[ 1681.857205]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[ 1681.857208]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[ 1681.857212]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[ 1681.857215]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.857218]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30
[ 1681.862187] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.862201] PANIC at dbuf.c:1304:dbuf_unoverride()
[ 1681.862204] Showing stack for process 27027
[ 1681.862208] CPU: 2 PID: 27027 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[ 1681.862213] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[ 1681.862216]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc900236c7c90
[ 1681.862231]  ffffffffa0008d82 0000000000000002 0000000000000028 ffffc900236c7ca0
[ 1681.862244]  ffffc900236c7c40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[ 1681.862256] Call Trace:
[ 1681.862263]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[ 1681.862271]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[ 1681.862278]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.862282]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.862308]  [<ffffffffa0214d49>] ? zio_done+0x999/0x1870 [zfs]
[ 1681.862313]  [<ffffffffa000bc18>] ? __cv_destroy+0x68/0x240 [spl]
[ 1681.862323]  [<ffffffffa0215e15>] ? zio_ready+0x1f5/0x5c0 [zfs]
[ 1681.862334]  [<ffffffffa02108ce>] ? zio_wait+0x1ee/0x3f0 [zfs]
[ 1681.862346]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[ 1681.862360]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[ 1681.862375]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[ 1681.862392]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[ 1681.862404]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[ 1681.862407]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[ 1681.862411]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[ 1681.862413]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[ 1681.862416]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[ 1681.862418]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.862421]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30
[ 1681.863815] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.863830] PANIC at dbuf.c:1304:dbuf_unoverride()
[ 1681.863834] Showing stack for process 27015
[ 1681.863838] CPU: 7 PID: 27015 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[ 1681.863840] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[ 1681.863843]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc9002366fc90
[ 1681.863851]  ffffffffa0008d82 0000000000000002 0000000000000028 ffffc9002366fca0
[ 1681.863858]  ffffc9002366fc40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[ 1681.863865] Call Trace:
[ 1681.863870]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[ 1681.863875]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[ 1681.863878]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.863881]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.863897]  [<ffffffffa0214d49>] ? zio_done+0x999/0x1870 [zfs]
[ 1681.863901]  [<ffffffffa000bc18>] ? __cv_destroy+0x68/0x240 [spl]
[ 1681.863911]  [<ffffffffa0215e15>] ? zio_ready+0x1f5/0x5c0 [zfs]
[ 1681.863921]  [<ffffffffa02108ce>] ? zio_wait+0x1ee/0x3f0 [zfs]
[ 1681.863930]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[ 1681.863939]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[ 1681.863954]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[ 1681.863970]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[ 1681.863984]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[ 1681.863989]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[ 1681.863993]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[ 1681.863996]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[ 1681.863999]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[ 1681.864002]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.864004]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.864007]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30
[ 1681.865049] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.865058] PANIC at dbuf.c:1304:dbuf_unoverride()
[ 1681.865059] Showing stack for process 27035
[ 1681.865062] CPU: 2 PID: 27035 Comm: zvol Tainted: P           O    4.9.32-mgm #4
[ 1681.865064] Hardware name: System manufacturer System Product Name/P8Z77-V DELUXE, BIOS 2104 08/13/2013
[ 1681.865066]  0000000000000000 ffffffff81293564 ffffffffa02ae5b0 ffffc90023707c90
[ 1681.865074]  ffffffffa0008d82 ffff88053c64e200 ffffffff00000028 ffffc90023707ca0
[ 1681.865082]  ffffc90023707c40 6428594649524556 6c642e74643e2d72 7265766f5f72642e
[ 1681.865090] Call Trace:
[ 1681.865094]  [<ffffffff81293564>] ? dump_stack+0x6f/0x8b
[ 1681.865098]  [<ffffffffa0008d82>] ? spl_panic+0xc2/0xf0 [spl]
[ 1681.865110]  [<ffffffffa013397c>] ? dbuf_dirty+0xcdc/0x11e0 [zfs]
[ 1681.865123]  [<ffffffffa01ac4dd>] ? txg_verify+0x1d/0xf0 [zfs]
[ 1681.865126]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.865137]  [<ffffffffa0140db0>] ? dmu_objset_userused_enabled+0x20/0x50 [zfs]
[ 1681.865148]  [<ffffffffa0144be1>] ? dmu_objset_userquota_get_ids+0x51/0x4e0 [zfs]
[ 1681.865158]  [<ffffffffa0140d4b>] ? dnode_multilist_index_func+0xbb/0x100 [zfs]
[ 1681.865161]  [<ffffffff8158019c>] ? mutex_lock+0x1c/0x40
[ 1681.865173]  [<ffffffffa01876f5>] ? multilist_sublist_lock+0x45/0xa0 [zfs]
[ 1681.865184]  [<ffffffffa0156527>] ? dnode_setdirty+0xc7/0x3c0 [zfs]
[ 1681.865199]  [<ffffffffa01877a8>] ? multilist_sublist_unlock+0x28/0x30 [zfs]
[ 1681.865214]  [<ffffffffa01332ca>] ? dbuf_dirty+0x62a/0x11e0 [zfs]
[ 1681.865229]  [<ffffffffa011b27b>] ? arc_buf_alloc_impl+0x24b/0x5b0 [zfs]
[ 1681.865243]  [<ffffffffa012a38f>] ? dbuf_unoverride+0x15f/0x190 [zfs]
[ 1681.865253]  [<ffffffffa012a5a3>] ? dbuf_redirty+0x43/0xd0 [zfs]
[ 1681.865262]  [<ffffffffa0133f1e>] ? dmu_buf_will_dirty+0x9e/0x1c0 [zfs]
[ 1681.865272]  [<ffffffffa013eeb7>] ? dmu_write_uio_dnode+0xa7/0x1c0 [zfs]
[ 1681.865283]  [<ffffffffa0222391>] ? zvol_write+0x191/0x5d0 [zfs]
[ 1681.865287]  [<ffffffffa0006258>] ? taskq_thread+0x298/0x590 [spl]
[ 1681.865290]  [<ffffffff81092f50>] ? wake_up_q+0x80/0x80
[ 1681.865293]  [<ffffffffa0005fc0>] ? taskq_thread_should_stop+0x90/0x90 [spl]
[ 1681.865297]  [<ffffffff8108a801>] ? kthread+0xe1/0x100
[ 1681.865300]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.865303]  [<ffffffff8108a720>] ? kthread_park+0x60/0x60
[ 1681.865305]  [<ffffffff81582995>] ? ret_from_fork+0x25/0x30
[ 1681.868647] VERIFY(dr->dt.dl.dr_override_state != DR_IN_DMU_SYNC) failed
[ 1681.868672] PANIC at dbuf.c:1304:dbuf_unoverride()

[mgmartin]

I've been unable to re-produce the issue now for a few days. There were reports of a library causing kvm issues with a recent Arch update, and I had other seg-faults with the kvm process. I've switched to running a different distro now. If anything new appears, I'll open a new issue.