Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

ZFS causing segmentation faults and crashing the system. #49

Closed
uclageek opened this Issue Sep 13, 2011 · 7 comments

Comments

Projects
None yet
3 participants

I have been try to figure out what is going on with my system I have had ZFS running for a while with no issue but for some reason it will let me read ~700MB and then throws a segmentation fault and crashes the system.

This has been happening for about a week. I rebooted it today and now ZFS will simply not load.

Not sure how much of these call traces are useful but here is the one which does not allow ZFS to load at all. If you need the others just let me know and I will post them.

I'm using Ubuntu 11.04 Server. 8GB of RAM and AMD Phenom 8450e

Sep 13 16:29:40 Soundwave kernel: [ 64.961048] zfs: freeing free segment (offset=9622259040256 size=2048)
Sep 13 16:29:40 Soundwave kernel: [ 64.961058] SPLError: 2990:0:(spl-err.c:48:vpanic()) SPL PANIC
Sep 13 16:29:40 Soundwave kernel: [ 64.961060] SPL: Showing stack for process 2990
Sep 13 16:29:40 Soundwave kernel: [ 64.961064] Pid: 2990, comm: z_wr_iss/2 Tainted: P 2.6.38-11-server #48-Ubuntu
Sep 13 16:29:40 Soundwave kernel: [ 64.961067] Call Trace:
Sep 13 16:29:40 Soundwave kernel: [ 64.961084] [] ? spl_debug_dumpstack+0x27/0x40 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961091] [] ? spl_debug_bug+0x81/0xe0 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961100] [] ? vpanic+0x65/0x80 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961107] [] ? populate_rootfs_wait+0x158/0x820
Sep 13 16:29:40 Soundwave kernel: [ 64.961111] [] ? __kmalloc+0xff/0x140
Sep 13 16:29:40 Soundwave kernel: [ 64.961118] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961122] [] ? default_spin_lock_flags+0x9/0x10
Sep 13 16:29:40 Soundwave kernel: [ 64.961128] [] ? vcmn_err+0x72/0x80 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961133] [] ? alloc_pages_current+0xa5/0x110
Sep 13 16:29:40 Soundwave kernel: [ 64.961136] [] ? new_slab+0x1c3/0x290
Sep 13 16:29:40 Soundwave kernel: [ 64.961139] [] ? __slab_alloc+0x1b2/0x390
Sep 13 16:29:40 Soundwave kernel: [ 64.961145] [] ? kmem_alloc_debug+0xeb/0x130 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961198] [] ? zfs_panic_recover+0x52/0x60 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961224] [] ? space_map_remove+0x20c/0x340 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961244] [] ? dmu_read+0x134/0x180 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961271] [] ? space_map_load+0x187/0x320 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961295] [] ? metaslab_activate+0xdb/0x160 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961321] [] ? metaslab_alloc+0x546/0x930 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961348] [] ? zio_dva_allocate+0x96/0x370 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961372] [] ? zio_push_transform+0x51/0xb0 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961398] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961421] [] ? zio_execute+0xb0/0x150 [zfs]
Sep 13 16:29:40 Soundwave kernel: [ 64.961428] [] ? taskq_thread+0x1c0/0x390 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961432] [] ? default_wake_function+0x0/0x20
Sep 13 16:29:40 Soundwave kernel: [ 64.961438] [] ? taskq_thread+0x0/0x390 [spl]
Sep 13 16:29:40 Soundwave kernel: [ 64.961443] [] ? kthread+0x96/0xa0
Sep 13 16:29:40 Soundwave kernel: [ 64.961447] [] ? kernel_thread_helper+0x4/0x10
Sep 13 16:29:40 Soundwave kernel: [ 64.961450] [] ? kthread+0x0/0xa0
Sep 13 16:29:40 Soundwave kernel: [ 64.961452] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:29:40 Soundwave kernel: [ 64.961507] SPL: Dumping log to /tmp/spl-log.1315956580.2990
Sep 13 16:29:50 Soundwave x-session-manager[2691]: WARNING: Could not launch application 'gnome-user-share.desktop': Unable to start application: Failed to execute child process "/usr/lib/gnome-user-share/gnome-user-share" (No such file or directory)
Sep 13 16:32:36 Soundwave kernel: [ 241.140095] INFO: task mount.zfs:2794 blocked for more than 120 seconds.
Sep 13 16:32:36 Soundwave kernel: [ 241.140102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:32:36 Soundwave kernel: [ 241.140108] mount.zfs D 0000000000000000 0 2794 2792 0x00000000
Sep 13 16:32:36 Soundwave kernel: [ 241.140118] ffff88020f843b48 0000000000000086 ffff88020f843fd8 ffff88020f842000
Sep 13 16:32:36 Soundwave kernel: [ 241.140126] 0000000000013d00 ffff8802096d03b8 ffff88020f843fd8 0000000000013d00
Sep 13 16:32:36 Soundwave kernel: [ 241.140133] ffffffff81a0b020 ffff8802096d0000 000000020f843b58 ffff8801f526fa80
Sep 13 16:32:36 Soundwave kernel: [ 241.140139] Call Trace:
Sep 13 16:32:36 Soundwave kernel: [ 241.140168] [] cv_wait_common+0x77/0xd0 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.140179] [] ? autoremove_wake_function+0x0/0x40
Sep 13 16:32:36 Soundwave kernel: [ 241.140190] [] ? __mutex_unlock_slowpath+0x4c/0x60
Sep 13 16:32:36 Soundwave kernel: [ 241.140205] [] __cv_wait+0x13/0x20 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.140280] [] txg_wait_synced+0x7b/0xa0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140332] [] spa_load+0x10ba/0x1400 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140395] [] ? txg_list_create+0x2f/0x60 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140446] [] spa_load+0x69b/0x1400 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140501] [] spa_load_best+0x4e/0x200 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140557] [] spa_open_common+0x14f/0x350 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140608] [] spa_open+0x13/0x20 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140667] [] pool_status_check+0x40/0xa0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140716] [] zfsdev_ioctl+0x155/0x1b0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140727] [] do_vfs_ioctl+0x8f/0x320
Sep 13 16:32:36 Soundwave kernel: [ 241.140733] [] sys_ioctl+0x91/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.140740] [] system_call_fastpath+0x16/0x1b
Sep 13 16:32:36 Soundwave kernel: [ 241.140748] INFO: task z_wr_iss/0:2988 blocked for more than 120 seconds.
Sep 13 16:32:36 Soundwave kernel: [ 241.140752] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:32:36 Soundwave kernel: [ 241.140756] z_wr_iss/0 D 0000000000000000 0 2988 2 0x00000000
Sep 13 16:32:36 Soundwave kernel: [ 241.140763] ffff8801e4d37b00 0000000000000046 ffff8801e4d37fd8 ffff8801e4d36000
Sep 13 16:32:36 Soundwave kernel: [ 241.140771] 0000000000013d00 ffff8801eb975f38 ffff8801e4d37fd8 0000000000013d00
Sep 13 16:32:36 Soundwave kernel: [ 241.140779] ffff8801eba9c4a0 ffff8801eb975b80 0000020003c54c00 ffff8801e454c000
Sep 13 16:32:36 Soundwave kernel: [ 241.140786] Call Trace:
Sep 13 16:32:36 Soundwave kernel: [ 241.140792] [] __mutex_lock_slowpath+0xf7/0x180
Sep 13 16:32:36 Soundwave kernel: [ 241.140798] [] mutex_lock+0x23/0x50
Sep 13 16:32:36 Soundwave kernel: [ 241.140850] [] metaslab_alloc+0x4d9/0x930 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140902] [] zio_dva_allocate+0x96/0x370 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.140952] [] ? zio_push_transform+0x51/0xb0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141003] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141052] [] zio_ready+0x332/0x3e0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141111] [] zio_execute+0xb0/0x150 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141122] [] ? default_spin_lock_flags+0x9/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141136] [] taskq_thread+0x1c0/0x390 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141143] [] ? default_wake_function+0x0/0x20
Sep 13 16:32:36 Soundwave kernel: [ 241.141156] [] ? taskq_thread+0x0/0x390 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141165] [] kthread+0x96/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.141171] [] kernel_thread_helper+0x4/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141177] [] ? kthread+0x0/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.141183] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141188] INFO: task z_wr_iss/2:2990 blocked for more than 120 seconds.
Sep 13 16:32:36 Soundwave kernel: [ 241.141191] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:32:36 Soundwave kernel: [ 241.141195] z_wr_iss/2 D 0000000000000002 0 2990 2 0x00000000
Sep 13 16:32:36 Soundwave kernel: [ 241.141202] ffff8801e4d3b790 0000000000000046 ffff8801e4d3bfd8 ffff8801e4d3a000
Sep 13 16:32:36 Soundwave kernel: [ 241.141208] 0000000000013d00 ffff88020b02b178 ffff8801e4d3bfd8 0000000000013d00
Sep 13 16:32:36 Soundwave kernel: [ 241.141218] ffff88020fd2db80 ffff88020b02adc0 ffff8801e4d3b758 0000000000000000
Sep 13 16:32:36 Soundwave kernel: [ 241.141229] Call Trace:
Sep 13 16:32:36 Soundwave kernel: [ 241.141240] [] spl_debug_bug+0xbd/0xe0 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141262] [] vpanic+0x65/0x80 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141271] [] ? populate_rootfs_wait+0x158/0x820
Sep 13 16:32:36 Soundwave kernel: [ 241.141281] [] ? __kmalloc+0xff/0x140
Sep 13 16:32:36 Soundwave kernel: [ 241.141296] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141304] [] ? default_spin_lock_flags+0x9/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141319] [] vcmn_err+0x72/0x80 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141329] [] ? alloc_pages_current+0xa5/0x110
Sep 13 16:32:36 Soundwave kernel: [ 241.141335] [] ? new_slab+0x1c3/0x290
Sep 13 16:32:36 Soundwave kernel: [ 241.141340] [] ? __slab_alloc+0x1b2/0x390
Sep 13 16:32:36 Soundwave kernel: [ 241.141352] [] ? kmem_alloc_debug+0xeb/0x130 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141407] [] zfs_panic_recover+0x52/0x60 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141461] [] space_map_remove+0x20c/0x340 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141504] [] ? dmu_read+0x134/0x180 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141563] [] space_map_load+0x187/0x320 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141612] [] metaslab_activate+0xdb/0x160 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141661] [] metaslab_alloc+0x546/0x930 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141711] [] zio_dva_allocate+0x96/0x370 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141764] [] ? zio_push_transform+0x51/0xb0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141814] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141864] [] zio_execute+0xb0/0x150 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.141880] [] taskq_thread+0x1c0/0x390 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141886] [] ? default_wake_function+0x0/0x20
Sep 13 16:32:36 Soundwave kernel: [ 241.141902] [] ? taskq_thread+0x0/0x390 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.141911] [] kthread+0x96/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.141919] [] kernel_thread_helper+0x4/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141925] [] ? kthread+0x0/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.141932] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.141952] INFO: task txg_sync:3152 blocked for more than 120 seconds.
Sep 13 16:32:36 Soundwave kernel: [ 241.141956] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:32:36 Soundwave kernel: [ 241.141959] txg_sync D 0000000000000001 0 3152 2 0x00000000
Sep 13 16:32:36 Soundwave kernel: [ 241.141969] ffff8801d4a97bc0 0000000000000046 ffff8801d4a97fd8 ffff8801d4a96000
Sep 13 16:32:36 Soundwave kernel: [ 241.141975] 0000000000013d00 ffff8801e4f7b178 ffff8801d4a97fd8 0000000000013d00
Sep 13 16:32:36 Soundwave kernel: [ 241.141982] ffff88020ba75b80 ffff8801e4f7adc0 ffff8801f5c52200 ffffc90018198ea8
Sep 13 16:32:36 Soundwave kernel: [ 241.141988] Call Trace:
Sep 13 16:32:36 Soundwave kernel: [ 241.142003] [] cv_wait_common+0x77/0xd0 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.142012] [] ? autoremove_wake_function+0x0/0x40
Sep 13 16:32:36 Soundwave kernel: [ 241.142026] [] __cv_wait+0x13/0x20 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.142073] [] zio_wait+0xf3/0x1b0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.142121] [] dsl_pool_sync+0x2d3/0x4a0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.142172] [] spa_sync+0x3ab/0x9b0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.142181] [] ? autoremove_wake_function+0x16/0x40
Sep 13 16:32:36 Soundwave kernel: [ 241.142190] [] ? __wake_up+0x53/0x70
Sep 13 16:32:36 Soundwave kernel: [ 241.142241] [] txg_sync_thread+0x241/0x3c0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.142293] [] ? txg_sync_thread+0x0/0x3c0 [zfs]
Sep 13 16:32:36 Soundwave kernel: [ 241.142313] [] thread_generic_wrapper+0x78/0x90 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.142328] [] ? thread_generic_wrapper+0x0/0x90 [spl]
Sep 13 16:32:36 Soundwave kernel: [ 241.142336] [] kthread+0x96/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.142342] [] kernel_thread_helper+0x4/0x10
Sep 13 16:32:36 Soundwave kernel: [ 241.142348] [] ? kthread+0x0/0xa0
Sep 13 16:32:36 Soundwave kernel: [ 241.142353] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:32:59 Soundwave kernel: [ 264.600836] show_signal_msg: 21 callbacks suppressed
Sep 13 16:32:59 Soundwave kernel: [ 264.600842] apt-get[3612]: segfault at 7fad1665b000 ip 00007fad153e0cfb sp 00007fffc58b4eb0 error 4 in libapt-pkg.so.4.10.1[7fad15392000+10b000]
Sep 13 16:34:36 Soundwave kernel: [ 361.140084] INFO: task mount.zfs:2794 blocked for more than 120 seconds.
Sep 13 16:34:36 Soundwave kernel: [ 361.140091] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:34:36 Soundwave kernel: [ 361.140097] mount.zfs D 0000000000000000 0 2794 2792 0x00000000
Sep 13 16:34:36 Soundwave kernel: [ 361.140106] ffff88020f843b48 0000000000000086 ffff88020f843fd8 ffff88020f842000
Sep 13 16:34:36 Soundwave kernel: [ 361.140114] 0000000000013d00 ffff8802096d03b8 ffff88020f843fd8 0000000000013d00
Sep 13 16:34:36 Soundwave kernel: [ 361.140122] ffffffff81a0b020 ffff8802096d0000 000000020f843b58 ffff8801f526fa80
Sep 13 16:34:36 Soundwave kernel: [ 361.140128] Call Trace:
Sep 13 16:34:36 Soundwave kernel: [ 361.140159] [] cv_wait_common+0x77/0xd0 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.140169] [] ? autoremove_wake_function+0x0/0x40
Sep 13 16:34:36 Soundwave kernel: [ 361.140180] [] ? __mutex_unlock_slowpath+0x4c/0x60
Sep 13 16:34:36 Soundwave kernel: [ 361.140195] [] __cv_wait+0x13/0x20 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.140275] [] txg_wait_synced+0x7b/0xa0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140328] [] spa_load+0x10ba/0x1400 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140390] [] ? txg_list_create+0x2f/0x60 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140441] [] spa_load+0x69b/0x1400 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140497] [] spa_load_best+0x4e/0x200 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140555] [] spa_open_common+0x14f/0x350 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140605] [] spa_open+0x13/0x20 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140665] [] pool_status_check+0x40/0xa0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140718] [] zfsdev_ioctl+0x155/0x1b0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140730] [] do_vfs_ioctl+0x8f/0x320
Sep 13 16:34:36 Soundwave kernel: [ 361.140737] [] sys_ioctl+0x91/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.140743] [] system_call_fastpath+0x16/0x1b
Sep 13 16:34:36 Soundwave kernel: [ 361.140751] INFO: task z_wr_iss/0:2988 blocked for more than 120 seconds.
Sep 13 16:34:36 Soundwave kernel: [ 361.140755] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:34:36 Soundwave kernel: [ 361.140759] z_wr_iss/0 D 0000000000000000 0 2988 2 0x00000000
Sep 13 16:34:36 Soundwave kernel: [ 361.140770] ffff8801e4d37b00 0000000000000046 ffff8801e4d37fd8 ffff8801e4d36000
Sep 13 16:34:36 Soundwave kernel: [ 361.140777] 0000000000013d00 ffff8801eb975f38 ffff8801e4d37fd8 0000000000013d00
Sep 13 16:34:36 Soundwave kernel: [ 361.140784] ffff8801eba9c4a0 ffff8801eb975b80 0000020003c54c00 ffff8801e454c000
Sep 13 16:34:36 Soundwave kernel: [ 361.140790] Call Trace:
Sep 13 16:34:36 Soundwave kernel: [ 361.140796] [] __mutex_lock_slowpath+0xf7/0x180
Sep 13 16:34:36 Soundwave kernel: [ 361.140802] [] mutex_lock+0x23/0x50
Sep 13 16:34:36 Soundwave kernel: [ 361.140855] [] metaslab_alloc+0x4d9/0x930 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140910] [] zio_dva_allocate+0x96/0x370 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.140960] [] ? zio_push_transform+0x51/0xb0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141014] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141063] [] zio_ready+0x332/0x3e0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141121] [] zio_execute+0xb0/0x150 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141132] [] ? default_spin_lock_flags+0x9/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141148] [] taskq_thread+0x1c0/0x390 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141154] [] ? default_wake_function+0x0/0x20
Sep 13 16:34:36 Soundwave kernel: [ 361.141167] [] ? taskq_thread+0x0/0x390 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141176] [] kthread+0x96/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.141182] [] kernel_thread_helper+0x4/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141188] [] ? kthread+0x0/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.141194] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141199] INFO: task z_wr_iss/2:2990 blocked for more than 120 seconds.
Sep 13 16:34:36 Soundwave kernel: [ 361.141202] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:34:36 Soundwave kernel: [ 361.141206] z_wr_iss/2 D 0000000000000002 0 2990 2 0x00000000
Sep 13 16:34:36 Soundwave kernel: [ 361.141216] ffff8801e4d3b790 0000000000000046 ffff8801e4d3bfd8 ffff8801e4d3a000
Sep 13 16:34:36 Soundwave kernel: [ 361.141222] 0000000000013d00 ffff88020b02b178 ffff8801e4d3bfd8 0000000000013d00
Sep 13 16:34:36 Soundwave kernel: [ 361.141228] ffff88020fd2db80 ffff88020b02adc0 ffff8801e4d3b758 0000000000000000
Sep 13 16:34:36 Soundwave kernel: [ 361.141235] Call Trace:
Sep 13 16:34:36 Soundwave kernel: [ 361.141248] [] spl_debug_bug+0xbd/0xe0 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141268] [] vpanic+0x65/0x80 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141278] [] ? populate_rootfs_wait+0x158/0x820
Sep 13 16:34:36 Soundwave kernel: [ 361.141284] [] ? __kmalloc+0xff/0x140
Sep 13 16:34:36 Soundwave kernel: [ 361.141299] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141304] [] ? default_spin_lock_flags+0x9/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141321] [] vcmn_err+0x72/0x80 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141332] [] ? alloc_pages_current+0xa5/0x110
Sep 13 16:34:36 Soundwave kernel: [ 361.141338] [] ? new_slab+0x1c3/0x290
Sep 13 16:34:36 Soundwave kernel: [ 361.141344] [] ? __slab_alloc+0x1b2/0x390
Sep 13 16:34:36 Soundwave kernel: [ 361.141356] [] ? kmem_alloc_debug+0xeb/0x130 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141410] [] zfs_panic_recover+0x52/0x60 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141462] [] space_map_remove+0x20c/0x340 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141505] [] ? dmu_read+0x134/0x180 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141561] [] space_map_load+0x187/0x320 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141611] [] metaslab_activate+0xdb/0x160 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141660] [] metaslab_alloc+0x546/0x930 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141707] [] zio_dva_allocate+0x96/0x370 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141757] [] ? zio_push_transform+0x51/0xb0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141810] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141858] [] zio_execute+0xb0/0x150 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.141874] [] taskq_thread+0x1c0/0x390 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141879] [] ? default_wake_function+0x0/0x20
Sep 13 16:34:36 Soundwave kernel: [ 361.141892] [] ? taskq_thread+0x0/0x390 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.141901] [] kthread+0x96/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.141910] [] kernel_thread_helper+0x4/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141915] [] ? kthread+0x0/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.141921] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.141944] INFO: task txg_sync:3152 blocked for more than 120 seconds.
Sep 13 16:34:36 Soundwave kernel: [ 361.141947] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:34:36 Soundwave kernel: [ 361.141951] txg_sync D 0000000000000001 0 3152 2 0x00000000
Sep 13 16:34:36 Soundwave kernel: [ 361.141961] ffff8801d4a97bc0 0000000000000046 ffff8801d4a97fd8 ffff8801d4a96000
Sep 13 16:34:36 Soundwave kernel: [ 361.141967] 0000000000013d00 ffff8801e4f7b178 ffff8801d4a97fd8 0000000000013d00
Sep 13 16:34:36 Soundwave kernel: [ 361.141973] ffff88020ba75b80 ffff8801e4f7adc0 ffff8801f5c52200 ffffc90018198ea8
Sep 13 16:34:36 Soundwave kernel: [ 361.141979] Call Trace:
Sep 13 16:34:36 Soundwave kernel: [ 361.141993] [] cv_wait_common+0x77/0xd0 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.142002] [] ? autoremove_wake_function+0x0/0x40
Sep 13 16:34:36 Soundwave kernel: [ 361.142016] [] __cv_wait+0x13/0x20 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.142065] [] zio_wait+0xf3/0x1b0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142117] [] dsl_pool_sync+0x2d3/0x4a0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142168] [] spa_sync+0x3ab/0x9b0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142178] [] ? autoremove_wake_function+0x16/0x40
Sep 13 16:34:36 Soundwave kernel: [ 361.142186] [] ? __wake_up+0x53/0x70
Sep 13 16:34:36 Soundwave kernel: [ 361.142236] [] txg_sync_thread+0x241/0x3c0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142288] [] ? txg_sync_thread+0x0/0x3c0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142302] [] thread_generic_wrapper+0x78/0x90 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.142314] [] ? thread_generic_wrapper+0x0/0x90 [spl]
Sep 13 16:34:36 Soundwave kernel: [ 361.142320] [] kthread+0x96/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.142328] [] kernel_thread_helper+0x4/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.142334] [] ? kthread+0x0/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.142341] [] ? kernel_thread_helper+0x0/0x10
Sep 13 16:34:36 Soundwave kernel: [ 361.142349] INFO: task zpool:3489 blocked for more than 120 seconds.
Sep 13 16:34:36 Soundwave kernel: [ 361.142352] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:34:36 Soundwave kernel: [ 361.142356] zpool D 0000000000000001 0 3489 3432 0x00000000
Sep 13 16:34:36 Soundwave kernel: [ 361.142366] ffff88020f87fda8 0000000000000086 ffff88020f87ffd8 ffff88020f87e000
Sep 13 16:34:36 Soundwave kernel: [ 361.142372] 0000000000013d00 ffff8801eb9883b8 ffff88020f87ffd8 0000000000013d00
Sep 13 16:34:36 Soundwave kernel: [ 361.142381] ffff88020fd28000 ffff8801eb988000 ffff88020f87fdb8 ffffffffa08255c0
Sep 13 16:34:36 Soundwave kernel: [ 361.142387] Call Trace:
Sep 13 16:34:36 Soundwave kernel: [ 361.142393] [] __mutex_lock_slowpath+0xf7/0x180
Sep 13 16:34:36 Soundwave kernel: [ 361.142401] [] mutex_lock+0x23/0x50
Sep 13 16:34:36 Soundwave kernel: [ 361.142452] [] spa_all_configs+0x52/0x120 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142504] [] zfs_ioc_pool_configs+0x2e/0x60 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142556] [] zfsdev_ioctl+0xf6/0x1b0 [zfs]
Sep 13 16:34:36 Soundwave kernel: [ 361.142566] [] do_vfs_ioctl+0x8f/0x320
Sep 13 16:34:36 Soundwave kernel: [ 361.142572] [] sys_ioctl+0x91/0xa0
Sep 13 16:34:36 Soundwave kernel: [ 361.142579] [] system_call_fastpath+0x16/0x1b
Sep 13 16:35:01 Soundwave CRON[4447]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 13 16:36:36 Soundwave kernel: [ 481.140121] INFO: task mount.zfs:2794 blocked for more than 120 seconds.
Sep 13 16:36:36 Soundwave kernel: [ 481.140127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 13 16:36:36 Soundwave kernel: [ 481.140133] mount.zfs D 0000000000000000 0 2794 2792 0x00000000
Sep 13 16:36:36 Soundwave kernel: [ 481.140143] ffff88020f843b48 0000000000000086 ffff88020f843fd8 ffff88020f842000
Sep 13 16:36:36 Soundwave kernel: [ 481.140150] 0000000000013d00 ffff8802096d03b8 ffff88020f843fd8 0000000000013d00
Sep 13 16:36:36 Soundwave kernel: [ 481.140160] ffffffff81a0b020 ffff8802096d0000 000000020f843b58 ffff8801f526fa80
Sep 13 16:36:36 Soundwave kernel: [ 481.140167] Call Trace:
Sep 13 16:36:36 Soundwave kernel: [ 481.140198] [] cv_wait_common+0x77/0xd0 [spl]
Sep 13 16:36:36 Soundwave kernel: [ 481.140208] [] ? autoremove_wake_function+0x0/0x40
Sep 13 16:36:36 Soundwave kernel: [ 481.140219] [] ? __mutex_unlock_slowpath+0x4c/0x60
Sep 13 16:36:36 Soundwave kernel: [ 481.140234] [] __cv_wait+0x13/0x20 [spl]
Sep 13 16:36:36 Soundwave kernel: [ 481.140308] [] txg_wait_synced+0x7b/0xa0 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140365] [] spa_load+0x10ba/0x1400 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140429] [] ? txg_list_create+0x2f/0x60 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140479] [] spa_load+0x69b/0x1400 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140534] [] spa_load_best+0x4e/0x200 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140593] [] spa_open_common+0x14f/0x350 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140646] [] spa_open+0x13/0x20 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140705] [] pool_status_check+0x40/0xa0 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140756] [] zfsdev_ioctl+0x155/0x1b0 [zfs]
Sep 13 16:36:36 Soundwave kernel: [ 481.140765] [] do_vfs_ioctl+0x8f/0x320
Sep 13 16:36:36 Soundwave kernel: [ 481.140771] [] sys_ioctl+0x91/0xa0
Sep 13 16:36:36 Soundwave kernel: [ 481.140778] [] system_call_fastpath+0x16/0x1b

Owner

behlendorf commented Sep 14, 2011

It looks like you've found an upstream zfs bug regarding corrupt space maps. You can try the following command which will check the exported pool and skip over the space maps. After it completes, which may be several hours, try importing your pool and then scrubbing it double check everything.

$ sudo zdb -e -bcsvL

The original thread suggesting this fix can be found here:

http://sigtar.com/2009/10/19/opensolaris-zfs-recovery-after-kernel-panic/

Hi Brian,

I tried that command and I got a segmentation fault.

Here is the error.

[ 224.492904] SPLError: 3188:0:(spl-err.c:48:vpanic()) SPL PANIC
[ 224.494172] SPL: Showing stack for process 3188
[ 224.494180] Pid: 3188, comm: z_wr_iss/1 Tainted: P 2.6.38-11-server #48-Ubuntu
[ 224.494184] Call Trace:
[ 224.494208] [] ? spl_debug_dumpstack+0x27/0x40 [spl]
[ 224.494225] [] ? spl_debug_bug+0x81/0xe0 [spl]
[ 224.494242] [] ? vpanic+0x65/0x80 [spl]
[ 224.494252] [] ? populate_rootfs_wait+0x158/0x820
[ 224.494259] [] ? __kmalloc+0xff/0x140
[ 224.494275] [] ? kmem_alloc_debug+0xbb/0x130 [spl]
[ 224.494282] [] ? default_spin_lock_flags+0x9/0x10
[ 224.494295] [] ? vcmn_err+0x72/0x80 [spl]
[ 224.494306] [] ? alloc_pages_current+0xa5/0x110
[ 224.494312] [] ? new_slab+0x1c3/0x290
[ 224.494318] [] ? __slab_alloc+0x1b2/0x390
[ 224.494329] [] ? kmem_alloc_debug+0xeb/0x130 [spl]
[ 224.494407] [] ? zfs_panic_recover+0x52/0x60 [zfs]
[ 224.494461] [] ? space_map_remove+0x20c/0x340 [zfs]
[ 224.494507] [] ? dmu_read+0x134/0x180 [zfs]
[ 224.494565] [] ? space_map_load+0x187/0x320 [zfs]
[ 224.494612] [] ? metaslab_activate+0xdb/0x160 [zfs]
[ 224.494667] [] ? metaslab_alloc+0x546/0x930 [zfs]
[ 224.494720] [] ? zio_dva_allocate+0x96/0x370 [zfs]
[ 224.494772] [] ? zio_push_transform+0x51/0xb0 [zfs]
[ 224.494826] [] ? zio_checksum_compute+0xd1/0x160 [zfs]
[ 224.494876] [] ? zio_execute+0xb0/0x150 [zfs]
[ 224.494890] [] ? taskq_thread+0x1c0/0x390 [spl]
[ 224.494898] [] ? default_wake_function+0x0/0x20
[ 224.494911] [] ? taskq_thread+0x0/0x390 [spl]
[ 224.494921] [] ? kthread+0x96/0xa0
[ 224.494927] [] ? kernel_thread_helper+0x4/0x10
[ 224.494933] [] ? kthread+0x0/0xa0
[ 224.494940] [] ? kernel_thread_helper+0x0/0x10
[ 224.495022] SPL: Dumping log to /tmp/spl-log.1316220806.3188

Just ran it again and got segmentation fault again but this time it didn't crash the system.

Here's the error

[47056.794786] zdb[3217]: segfault at 68 ip 00007fb7b33d3044 sp 00007fb7b437e740 error 4 in libzpool.so.1.0.0[7fb7b33b4000+b2000]

Owner

behlendorf commented Sep 26, 2011

The crash appears to be caused due to not having 'zfs_recovery' set. Unfortunately, this isn't quite implemented in the Linux port, but it is there under Solaris. The suggested commands should work there until we get this addressed. Alternately, you could try just commenting out the offending assertion for now which is basically what zfs recovery would do.

I tried importing it on Solaris Express 11 but it says the vdev has corrupt data and is unavailable. Also using zdb the command says "file/directory already exists". I checked the labels on the vdev and it looks like the 1st 2 are corrupt and I'm not really sure how to fix this.


LABEL 0

failed to unpack label 0

LABEL 1

failed to unpack label 1

LABEL 2

version: 28
name: 'Stuff'
state: 0
txg: 1159976
pool_guid: 17463683870246760487
hostname: 'Soundwave.local'
top_guid: 17898975408034390453
guid: 17898975408034390453
vdev_children: 4
vdev_tree:
    type: 'disk'
    id: 2
    guid: 17898975408034390453
    path: '/dev/sdj1'
    whole_disk: 0
    metaslab_array: 177
    metaslab_shift: 23
    ashift: 9
    asize: 500102070272
    is_log: 0
    DTL: 715
    create_txg: 823634

LABEL 3

version: 28
name: 'Stuff'
state: 0
txg: 1159976
pool_guid: 17463683870246760487
hostname: 'Soundwave.local'
top_guid: 17898975408034390453
guid: 17898975408034390453
vdev_children: 4
vdev_tree:
    type: 'disk'
    id: 2
    guid: 17898975408034390453
    path: '/dev/sdj1'
    whole_disk: 0
    metaslab_array: 177
    metaslab_shift: 23
    ashift: 9
    asize: 500102070272
    is_log: 0
    DTL: 715
    create_txg: 823634

I use latest ZFS on Gentoo with 3.0.4 kernel. Run "zdb -e -bcsvL data" , after 12 hours a have this message:

Sep 29 09:53:55 fs01 kernel: [68512.957207] zdb invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Sep 29 09:53:55 fs01 kernel: [68512.957214] Pid: 2013, comm: zdb Tainted: P 3.0.4 #10
Sep 29 09:53:55 fs01 kernel: [68512.957221] Call Trace:
Sep 29 09:53:55 fs01 kernel: [68512.957250] [] dump_header.clone.8+0x73/0x190
Sep 29 09:53:55 fs01 kernel: [68512.957266] [] ? ___ratelimit+0x93/0x120
Sep 29 09:53:55 fs01 kernel: [68512.957270] [] oom_kill_process.clone.11+0x7b/0x160
Sep 29 09:53:55 fs01 kernel: [68512.957273] [] ? select_bad_process.clone.12+0x90/0x130
Sep 29 09:53:55 fs01 kernel: [68512.957276] [] out_of_memory+0x115/0x240
Sep 29 09:53:55 fs01 kernel: [68512.957292] [] ? _raw_spin_unlock+0x9/0x10
Sep 29 09:53:55 fs01 kernel: [68512.957296] [] __alloc_pages_nodemask+0x75d/0x770
Sep 29 09:53:55 fs01 kernel: [68512.957305] [] do_anonymous_page.clone.58+0x105/0x270
Sep 29 09:53:55 fs01 kernel: [68512.957313] [] handle_pte_fault+0x1a1/0x1c0
Sep 29 09:53:55 fs01 kernel: [68512.957316] [] handle_mm_fault+0x159/0x2c0
Sep 29 09:53:55 fs01 kernel: [68512.957320] [] __get_user_pages+0x13c/0x590
Sep 29 09:53:55 fs01 kernel: [68512.957323] [] get_user_pages+0x4d/0x50
Sep 29 09:53:55 fs01 kernel: [68512.957337] [] get_user_pages_fast+0x13f/0x190
Sep 29 09:53:55 fs01 kernel: [68512.957346] [] dio_refill_pages+0x3d/0x130
Sep 29 09:53:55 fs01 kernel: [68512.957349] [] dio_get_page+0x3d/0x60
Sep 29 09:53:55 fs01 kernel: [68512.957352] [] do_direct_IO+0x6a/0x3d0
Sep 29 09:53:55 fs01 kernel: [68512.957355] [] direct_io_worker+0x1e5/0x470
Sep 29 09:53:55 fs01 kernel: [68512.957358] [] __blockdev_direct_IO+0x249/0x2a0
Sep 29 09:53:55 fs01 kernel: [68512.957361] [] ? blkdev_get_block+0x70/0x70
Sep 29 09:53:55 fs01 kernel: [68512.957365] [] ? check_preempt_wakeup+0x125/0x1d0
Sep 29 09:53:55 fs01 kernel: [68512.957368] [] blkdev_direct_IO+0x52/0x60
Sep 29 09:53:55 fs01 kernel: [68512.957370] [] ? blkdev_get_block+0x70/0x70
Sep 29 09:53:55 fs01 kernel: [68512.957374] [] generic_file_aio_read+0x27c/0x290
Sep 29 09:53:55 fs01 kernel: [68512.957377] [] ? ttwu_do_wakeup+0x1c/0xa0
Sep 29 09:53:55 fs01 kernel: [68512.957382] [] do_sync_read+0xd2/0x110
Sep 29 09:53:55 fs01 kernel: [68512.957387] [] ? finish_task_switch+0x5c/0xe0
Sep 29 09:53:55 fs01 kernel: [68512.957412] [] ? security_file_permission+0x94/0xb0
Sep 29 09:53:55 fs01 kernel: [68512.957415] [] vfs_read+0xc3/0x180
Sep 29 09:53:55 fs01 kernel: [68512.957418] [] sys_pread64+0xa2/0xb0
Sep 29 09:53:55 fs01 kernel: [68512.957422] [] system_call_fastpath+0x16/0x1b
Sep 29 09:53:55 fs01 kernel: [68512.957424] Mem-Info:
Sep 29 09:53:55 fs01 kernel: [68512.957429] DMA per-cpu:
Sep 29 09:53:55 fs01 kernel: [68512.957431] CPU 0: hi: 0, btch: 1 usd: 0
Sep 29 09:53:55 fs01 kernel: [68512.957433] CPU 1: hi: 0, btch: 1 usd: 0
Sep 29 09:53:55 fs01 kernel: [68512.957435] DMA32 per-cpu:
Sep 29 09:53:55 fs01 kernel: [68512.957437] CPU 0: hi: 186, btch: 31 usd: 34
Sep 29 09:53:55 fs01 kernel: [68512.957439] CPU 1: hi: 186, btch: 31 usd: 58
Sep 29 09:53:55 fs01 kernel: [68512.957440] Normal per-cpu:
Sep 29 09:53:55 fs01 kernel: [68512.957442] CPU 0: hi: 186, btch: 31 usd: 60
Sep 29 09:53:55 fs01 kernel: [68512.957444] CPU 1: hi: 186, btch: 31 usd: 109
Sep 29 09:53:55 fs01 kernel: [68512.957448] active_anon:997460 inactive_anon:233311 isolated_anon:0
Sep 29 09:53:55 fs01 kernel: [68512.957449] active_file:113 inactive_file:77 isolated_file:0
Sep 29 09:53:55 fs01 kernel: [68512.957450] unevictable:771 dirty:0 writeback:29 unstable:0
Sep 29 09:53:55 fs01 kernel: [68512.957451] free:21923 slab_reclaimable:632 slab_unreclaimable:2050
Sep 29 09:53:55 fs01 kernel: [68512.957452] mapped:619 shmem:6 pagetables:2998 bounce:0
Sep 29 09:53:55 fs01 kernel: [68512.957459] DMA free:15924kB min:204kB low:252kB high:304kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15700kB mlocked:0kB dirty:0k
B writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB
bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Sep 29 09:53:55 fs01 kernel: [68512.957464] lowmem_reserve[]: 0 3896 5032 5032
Sep 29 09:53:55 fs01 kernel: [68512.957474] DMA32 free:56648kB min:52164kB low:65204kB high:78244kB active_anon:3250080kB ina
ctive_anon:652972kB active_file:380kB inactive_file:124kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:39896
64kB mlocked:0kB dirty:0kB writeback:8kB mapped:420kB shmem:0kB slab_reclaimable:44kB slab_unreclaimable:248kB kernel_stack:1
6kB pagetables:5612kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:17836 all_unreclaimable? yes
Sep 29 09:53:55 fs01 kernel: [68512.957479] lowmem_reserve[]: 0 0 1136 1136
Sep 29 09:53:55 fs01 kernel: [68512.957486] Normal free:15120kB min:15212kB low:19012kB high:22816kB active_anon:739760kB ina
ctive_anon:280272kB active_file:72kB inactive_file:184kB unevictable:3084kB isolated(anon):0kB isolated(file):0kB present:116
3520kB mlocked:3084kB dirty:0kB writeback:108kB mapped:2056kB shmem:24kB slab_reclaimable:2484kB slab_unreclaimable:7952kB ke
rnel_stack:2256kB pagetables:6380kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:396 all_unreclaimable? yes
Sep 29 09:53:55 fs01 kernel: [68512.957492] lowmem_reserve[]: 0 0 0 0
Sep 29 09:53:55 fs01 kernel: [68512.957497] DMA: 1_4kB 0_8kB 1_16kB 1_32kB 2_64kB 1_128kB 1_256kB 0_512kB 1_1024kB 1_2048kB 3
_4096kB = 15924kB
Sep 29 09:53:55 fs01 kernel: [68512.957506] DMA32: 4_4kB 1_8kB 5_16kB 275_32kB 328_64kB 37_128kB 12_256kB 9_512kB 4_1024kB 1*
2048kB 2_4096kB = 56648kB
Sep 29 09:53:55 fs01 kernel: [68512.957517] Normal: 214_4kB 171_8kB 122_16kB 72_32kB 33_64kB 13_128kB 11_256kB 4_512kB 0_1024
kB 0_2048kB 0_4096kB = 15120kB
Sep 29 09:53:55 fs01 kernel: [68512.957526] 15899 total pagecache pages
Sep 29 09:53:55 fs01 kernel: [68512.957528] 15226 pages in swap cache
Sep 29 09:53:55 fs01 kernel: [68512.957530] Swap cache stats: add 1776896, delete 1761670, find 698966/852152
Sep 29 09:53:55 fs01 kernel: [68512.957532] Free swap = 0kB
Sep 29 09:53:55 fs01 kernel: [68512.957533] Total swap = 506040kB
Sep 29 09:53:55 fs01 kernel: [68512.981100] 1310704 pages RAM
Sep 29 09:53:55 fs01 kernel: [68512.981103] 42148 pages reserved
Sep 29 09:53:55 fs01 kernel: [68512.981104] 4663 pages shared
Sep 29 09:53:55 fs01 kernel: [68512.981105] 1244589 pages non-shared
Sep 29 09:53:55 fs01 kernel: [68512.981107] [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Sep 29 09:53:55 fs01 kernel: [68512.981115] [ 700] 0 700 3134 93 0 -17 -1000 udevd
Sep 29 09:53:55 fs01 kernel: [68512.981122] [ 1739] 0 1739 3166 61 1 -17 -1000 udevd
Sep 29 09:53:55 fs01 kernel: [68512.981125] [ 1740] 0 1740 3166 61 0 -17 -1000 udevd
Sep 29 09:53:55 fs01 kernel: [68512.981129] [ 1757] 0 1757 1031 110 1 0 0 iscsid
Sep 29 09:53:55 fs01 kernel: [68512.981132] [ 1758] 0 1758 3324 772 0 -17 -1000 iscsid
Sep 29 09:53:55 fs01 kernel: [68512.981136] [ 1795] 0 1795 5573 36 1 0 0 syslog-ng
Sep 29 09:53:55 fs01 kernel: [68512.981139] [ 1796] 0 1796 13979 197 1 0 0 syslog-ng
Sep 29 09:53:55 fs01 kernel: [68512.981142] [ 1813] 0 1813 7869 102 0 -17 -1000 sshd
Sep 29 09:53:55 fs01 kernel: [68512.981145] [ 1838] 0 1838 4132 160 1 0 0 cron
Sep 29 09:53:55 fs01 kernel: [68512.981148] [ 1851] 0 1851 14540 197 1 0 0 login
Sep 29 09:53:55 fs01 kernel: [68512.981151] [ 1852] 0 1852 2561 166 0 0 0 agetty
Sep 29 09:53:55 fs01 kernel: [68512.981155] [ 1853] 0 1853 2561 166 1 0 0 agetty
Sep 29 09:53:55 fs01 kernel: [68512.981158] [ 1854] 0 1854 2561 166 0 0 0 agetty
Sep 29 09:53:55 fs01 kernel: [68512.981161] [ 1855] 0 1855 2561 166 0 0 0 agetty
Sep 29 09:53:55 fs01 kernel: [68512.981164] [ 1856] 0 1856 2561 166 1 0 0 agetty
Sep 29 09:53:55 fs01 kernel: [68512.981167] [ 1857] 0 1857 17351 217 0 0 0 sshd
Sep 29 09:53:55 fs01 kernel: [68512.981170] [ 1861] 0 1861 4448 181 1 0 0 bash
Sep 29 09:53:55 fs01 kernel: [68512.981173] [ 2006] 0 2006 1374306 1215592 0 0 0 zdb
Sep 29 09:53:55 fs01 kernel: [68512.981177] [ 2252] 0 2252 4448 181 1 0 0 bash
Sep 29 09:53:55 fs01 kernel: [68512.981180] [ 2257] 0 2257 4783 269 0 0 0 top
Sep 29 09:53:55 fs01 kernel: [68512.981183] [ 2258] 0 2258 17351 291 0 0 0 sshd
Sep 29 09:53:55 fs01 kernel: [68512.981186] [ 2261] 0 2261 4480 183 1 0 0 bash
Sep 29 09:53:55 fs01 kernel: [68512.981189] [ 4781] 0 4781 4785 270 0 0 0 top
Sep 29 09:53:55 fs01 kernel: [68512.981193] Out of memory: Kill process 2006 (zdb) score 932 or sacrifice child
Sep 29 09:53:55 fs01 kernel: [68512.981219] Killed process 2006 (zdb) total-vm:5497224kB, anon-rss:4861536kB, file-rss:832kB
Sep 29 09:54:05 fs01 -- MARK --

Hi I realize this is late but classes started so this had to take a back seat. I tried running the commands on Solaris but since the zpool is for some reason marked as a root pool solaris can not run zdb on it.

Also I'm not sure what you mean by

"Alternately, you could try just commenting out the offending assertion for now which is basically what zfs recovery would do"

How do I do this?

@behlendorf behlendorf closed this Jun 14, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment