SPLError: 10949:0:(dbuf.c:2095:dbuf_add_ref()) SPL PANIC

[massmio]

Hello,
every time I try to copy some files to the partition /dev/trunk/home1 mounted at /home/zfs_home I hit the following bug:

SPLError: 10949:0:(dbuf.c:2095:dbuf_add_ref()) SPL PANIC
SPL: Showing stack for process 10949
Pid: 10949, comm: zvol/1 Tainted: P            2.6.35-zen2 #1
Call Trace:
 [] ? spl_debug_bug+0x7e/0xd0 [spl]
 [] ? dmu_buf_add_ref+0x51/0x60 [zfs]
 [] ? dbuf_dirty+0x249/0x740 [zfs]
 [] ? dmu_buf_will_fill+0x45/0xe0 [zfs]
 [] ? dmu_write_req+0x27d/0x2b0 [zfs]
 [] ? zvol_write+0xd4/0x570 [zfs]
 [] ? mempool_free+0x7f/0x90
 [] ? mempool_free+0x7f/0x90
 [] ? __wake_up+0x3e/0x60
 [] ? taskq_thread+0x17b/0x300 [spl]
 [] ? default_wake_function+0x0/0x10
 [] ? taskq_thread+0x0/0x300 [spl]
 [] ? kthread+0x74/0x80
 [] ? kthread+0x0/0x80
 [] ? kernel_thread_helper+0x6/0x10
SPL: Dumping log to /tmp/spl-log.1284982595.10949

System: vmware, gentoo, zen-kernel-2.6.35-zen2, 32bit, booting with vmalloc=512M

$ zpool status
pool: trunk
state: ONLINE
scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    trunk       ONLINE       0     0     0
      -0        ONLINE       0     0     0
        sdc1    ONLINE       0     0     0
        sdd1    ONLINE       0     0     0
        sde1    ONLINE       0     0     0
        sdf1    ONLINE       0     0     0
    spares
      sdg1      AVAIL

errors: No known data errors

$ zfs list
trunk 1.03G 1.89G 28.4K /trunk
trunk/home 1.03G 2.88G 38.2M -

Kind regards,
massmio

[behlendorf]

Thank for the bug report Massmio, it looks like we must me missing a dbuf reference in dbuf_add_ref() and we're tripping the VERIFY(). We'll take a look.

[peglarr]

Hi Brian, I saw the same bug in a Fedora 14, 2.6.35.10-74.fc14.i686, 32 bit, booting with vmalloc=512M as well. 0.5.2 stack. Just so you know...cheers, Rob Peglar

[behlendorf]

This is getting pretty stale and has not been observed in a long time so I'm closing it. Plus it's really a zfs bug so it should be tracker there, we can open a new one if this if hit again.

[wiloyee]

I have the same issue too.

SPL: The /etc/hostid file is not found.
SPL: Loaded module v0.6.0, using hostid 0x030a0cfa
ZFS: Loaded module v0.6.0, ZFS pool version 28, ZFS filesystem version 5
VERIFY(refcount_add(&db->db_holds, tag) > 1) failed
SPLError: 6185:0:(dbuf.c:2107:dbuf_add_ref()) SPL PANIC
SPL: Showing stack for process 6185
Pid: 6185, comm: mc Tainted: P 2.6.38-gentoo-r6 #12
Call Trace:
[] ? spl_debug_dumpstack+0x25/0x40 [spl]
[] ? spl_debug_bug+0x80/0x110 [spl]
[] ? dmu_buf_add_ref+0x4e/0x3e0 [zfs]
[] ? dbuf_dirty+0x26f/0x7b0 [zfs]
[] ? dmu_buf_will_dirty+0x45/0x90 [zfs]
[] ? dmu_buf_hold_array_by_bonus+0x126/0x200 [zfs]
[] ? dmu_write_uio_dbuf+0x50/0x70 [zfs]
[] ? zfs_write+0xf8f/0x1070 [zfs]
[] ? arc_buf_remove_ref+0xee/0x140 [zfs]
[] ? zpl_write_common+0x50/0x12c0 [zfs]
[] ? security_file_permission+0x1f/0xa0
[] ? zpl_write_common+0xbf/0x12c0 [zfs]
[] ? vfs_write+0x9a/0x140
[] ? zpl_write_common+0x60/0x12c0 [zfs]
[] ? sys_write+0x3d/0x70
[] ? sysenter_do_call+0x12/0x26
SPL: Dumping log to /tmp/spl-log.1316860779.6185
INFO: task txg_quiesce:6167 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
txg_quiesce D d0fe5f00 0 6167 2 0x00000000
d0fe5f10 00000046 00000002 d0fe5f00 00000000 c03a2d80 d0fe5e88 c01268fa
00000000 cefd56a0 c04f3f40 c04f3f40 cefd56a0 c04f3f40 cefd590c 8a30948c
000093ab c04f3f40 cefd5908 c04f3f40 8a30789e 000093ab c04f3f40 cefd56a0
Call Trace:
[] ? check_preempt_curr+0x6a/0x80
[] ? __wake_up+0x40/0x50
[] ? prepare_to_wait_exclusive+0x43/0x70
[] __cv_timedwait+0x7a/0xd0 [spl]
[] ? autoremove_wake_function+0x0/0x40
[] __cv_wait+0xd/0x10 [spl]
[] txg_sync_start+0x711/0x7e0 [zfs]
[] ? txg_sync_start+0x580/0x7e0 [zfs]
[] __thread_create+0x179/0x190 [spl]
[] ? __thread_create+0x120/0x190 [spl]
[] kthread+0x74/0x80
[] ? kthread+0x0/0x80

SPL creates a log file.

I don't know how to upload a file. should i add a hexdump ? ... the file is very short.

[behlendorf]

The log won't contain anything useful in this case. Do you happen to have a test case which caused this?

[Pinkbyte]

I have same issue.

Arch: x86_64
OS: Gentoo Linux
SPL: 0.6.0_rc6 with compatibility patches for 3.1 and 3.2(from git tree)
ZFS: 0.6.0_rc6 with compatibility patches for 3.1 and 3.2(from git tree)
Kernel: 3.2.1

Testcase: one pool, one empty volume(test, for example), compression and deduplication are enabled. Doing 'emerge --sync'(write many little files on volume through rsync) hangs with such message. I will try to do tests without compression/deduplication and with latest snapshot...

Update: i am sorry, test machine was x86, and, as i now, spl/zfs are not supported on x86 arch..

[ryao]

@Pinkbyte I suggest using the 9999 ebuilds from the portage tree. The 0.6.0_rc6 release has known issues.