BUG: soft lockup - CPU stuck for 67s #81
Comments
|
There's nothing really wrong here, the warnings are just advisory that we're having a hard time mapping virtual memory. Are you perhaps running this on a 32-bit system? Long term this can only be addressed by moving zfs away from its dependence on virtual memory. |
|
Responding to the 1st message, this is on: which has 64G RAM, 8 cores and is not touching swap when this is occurring. While these warnings seem to have no ill effects, we've seen similar ones that cause each core to lock up in turn before the system itself crashes. |
|
I see your using the 2.6.37 kernel, you might try a 2.6.39 or newer kernel. There was a significant performance regression for vmalloc introduced in the 2.6.28 kernel which wasn't fixed until 2.6.39. Since currently zfs is still a heavy consumer of vmalloc and this is where your tripped soft lockup it may help. See the following upstream kernel commit. |
|
Thanks Brian, alternative choices for OS would be opensuse 12.1 (Kernel 3.1) or CentOS 6.x. (Kernel 2.6.32). I believe the former has not been tested in your buildbot yet and for the latter I'm not sure if the regression has been backported yet. Jeff, may be we can figure that out. dipe On Jan 26, 2012, at 9:57 AM, Brian Behlendorfreply@reply.github.com wrote:
|
|
OpenSuse 12.1 should work although there may be an outstanding issue building packages still. CentOS 6.x is my target platform so I know it works there, and has the needed fix. |
|
Thanks so much for your assistance on this matter. We'll see if either there's a backport of the vmalloc fix(es) to opensuse 11.4 or move to an enterprise Linux. |
|
I would call this closed for now |
ghost
closed this as 
This happens when I copy large files via nfs to this system. (Kernel 2.6.37, zfs 0.6.0-rc6). Copy happens with ca 60-70 MB/s
Anyone knows what could be the issue
on console:
Message from syslogd@orca1 at Jan 24 18:22:51 ...
kernel:[1066884.745025] Code: d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 <39> c2 74 07 f3 90 0f b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28
syslog:
an 24 17:04:01 orca1 smartd[3718]: Device: /dev/sdc [SAT], Read Summary SMART Error Log failed
Jan 24 17:04:02 orca1 smartd[3718]: Device: /dev/sde [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Jan 24 17:04:12 orca1 smartd[3718]: Device: /dev/sdl [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 187 to 193
Jan 24 17:10:01 orca1 /USR/SBIN/CRON[31147]: (root) CMD (/usr/local/sbin/syslog_check.sh >>/var/tmp/syslog_check 2>&1)
Jan 24 17:18:54 orca1 kernel: [1062972.284011] BUG: soft lockup - CPU#2 stuck for 67s! [z_wr_iss/2:5852]
Jan 24 17:18:54 orca1 kernel: [1062972.284015] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.284067] CPU 2
Jan 24 17:18:54 orca1 kernel: [1062972.284068] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.284095]
Jan 24 17:18:54 orca1 kernel: [1062972.284098] Pid: 5852, comm: z_wr_iss/2 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062972.284102] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062972.284111] RSP: 0018:ffff880f917e9968 EFLAGS: 00000283
Jan 24 17:18:54 orca1 kernel: [1062972.284112] RAX: 0000000000003a62 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062972.284114] RDX: 0000000000003a5d RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062972.284115] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062972.284117] R10: ffffc90350c1e020 R11: 0000000000000023 R12: 0000000200000041
Jan 24 17:18:54 orca1 kernel: [1062972.284119] R13: ffff880ffffece00 R14: 00000000000612d0 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062972.284120] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284122] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062972.284124] CR2: 00007ff925765000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062972.284126] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062972.284129] Process z_wr_iss/2 (pid: 5852, threadinfo ffff880f917e8000, task ffff880f91bea140)
Jan 24 17:18:54 orca1 kernel: [1062972.284131] Stack:
Jan 24 17:18:54 orca1 kernel: [1062972.284140] ffffffff81122ba4 ffff880f91bea140 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062972.284144] ffffc90000005000 ffffc90000004fff ffff880273a462a0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062972.284147] ffff880760b748c0 0000000000000000 ffff880760b748c0 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062972.284150] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.284165] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.284170] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.284173] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.284187] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284204] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284263] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284390] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284508] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284595] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284714] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284799] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.284902] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284912] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.284917] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.284920] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062972.284945] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.284947] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.284951] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.284954] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.284959] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284969] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.284993] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285108] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285225] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285311] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285428] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285511] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.285611] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.285619] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.285622] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.652010] BUG: soft lockup - CPU#6 stuck for 67s! [z_wr_iss/6:5856]
Jan 24 17:18:54 orca1 kernel: [1062972.652014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.652066] CPU 6
Jan 24 17:18:54 orca1 kernel: [1062972.652067] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062972.652094]
Jan 24 17:18:54 orca1 kernel: [1062972.652096] Pid: 5856, comm: z_wr_iss/6 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062972.652100] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062972.652109] RSP: 0018:ffff880f913b98f8 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1062972.652110] RAX: 0000000000003a66 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062972.652112] RDX: 0000000000003a64 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062972.652114] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062972.652115] R10: ffffc90350c36020 R11: 0000000000000031 R12: 0000000200000000
Jan 24 17:18:54 orca1 kernel: [1062972.652117] R13: ffff880ffffece00 R14: 0000000000000002 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062972.652118] FS: 0000000000000000(0000) GS:ffff8800cfd80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062972.652122] CR2: 00007ff925766000 CR3: 00000007662f8000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062972.652124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062972.652127] Process z_wr_iss/6 (pid: 5856, threadinfo ffff880f913b8000, task ffff880f815ee340)
Jan 24 17:18:54 orca1 kernel: [1062972.652129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062972.652137] ffffffff81122ba4 ffff880ffffece68 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062972.652141] ffffc90000005000 ffffc90000004fff ffff8802e2e1a6c0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062972.652144] ffff880274018940 0000000000000000 ffff880274018940 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062972.652147] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.652162] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.652167] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.652170] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.652184] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.652200] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.652257] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652382] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652501] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652587] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652705] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652789] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.652908] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653011] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653021] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.653027] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062972.653029] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062972.653054] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062972.653057] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062972.653059] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062972.653062] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062972.653068] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653077] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653101] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653216] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653332] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653417] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653534] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653617] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653734] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062972.653834] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062972.653842] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062972.653844] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.100012] BUG: soft lockup - CPU#0 stuck for 67s! [z_wr_iss/0:5850]
Jan 24 17:18:54 orca1 kernel: [1062984.100016] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.100066] CPU 0
Jan 24 17:18:54 orca1 kernel: [1062984.100066] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.100094]
Jan 24 17:18:54 orca1 kernel: [1062984.100096] Pid: 5850, comm: z_wr_iss/0 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.100100] RIP: 0010:[] [] _raw_spin_lock+0x18/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.100109] RSP: 0018:ffff880fb4d978f8 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1062984.100110] RAX: 0000000000003b44 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.100112] RDX: 0000000000003b42 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.100113] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.100115] R10: ffffc90355184020 R11: 0000000000000061 R12: 000000027daba0c0
Jan 24 17:18:54 orca1 kernel: [1062984.100117] R13: ffff880ffffece00 R14: ffffe8ffffffffff R15: ffffffffb34f0f58
Jan 24 17:18:54 orca1 kernel: [1062984.100118] FS: 0000000000000000(0000) GS:ffff8800cfc00000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.100122] CR2: 00007ff92576a000 CR3: 0000000001a03000 CR4: 00000000000006f0
Jan 24 17:18:54 orca1 kernel: [1062984.100123] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.100127] Process z_wr_iss/0 (pid: 5850, threadinfo ffff880fb4d96000, task ffff880f7daba0c0)
Jan 24 17:18:54 orca1 kernel: [1062984.100129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.100137] ffffffff81122ba4 0000000000000246 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.100141] ffffc90000005000 ffffc90000004fff ffff880b257dd620 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.100143] ffff8805f1e92b40 0000000000000000 ffff8805f1e92b40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.100146] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.100160] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.100165] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.100169] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.100182] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.100198] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.100253] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100377] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100495] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100581] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100700] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100783] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.100903] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101006] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101016] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.101021] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.101024] Code: 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c 89 64 24
Jan 24 17:18:54 orca1 kernel: [1062984.101048] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.101051] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.101054] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.101057] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.101063] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101072] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101096] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101210] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101327] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101412] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101529] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101612] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101729] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.101829] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.101836] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.101839] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.192012] BUG: soft lockup - CPU#1 stuck for 67s! [z_wr_iss/1:5851]
Jan 24 17:18:54 orca1 kernel: [1062984.192016] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.192067] CPU 1
Jan 24 17:18:54 orca1 kernel: [1062984.192068] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.192095]
Jan 24 17:18:54 orca1 kernel: [1062984.192098] Pid: 5851, comm: z_wr_iss/1 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.192101] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.192110] RSP: 0018:ffff880fb33ab8f8 EFLAGS: 00000297
Jan 24 17:18:54 orca1 kernel: [1062984.192112] RAX: 0000000000003b49 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.192113] RDX: 0000000000003b43 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.192115] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.192116] R10: ffff880fff805160 R11: ffff880a1301d030 R12: 0000000200000010
Jan 24 17:18:54 orca1 kernel: [1062984.192118] R13: ffff880ffffece00 R14: 000000000000001f R15: ffffffff810fa855
Jan 24 17:18:54 orca1 kernel: [1062984.192120] FS: 0000000000000000(0000) GS:ffff8800cfc40000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.192122] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.192123] CR2: 00007ffdb4824000 CR3: 000000046a01a000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.192125] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.192127] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.192129] Process z_wr_iss/1 (pid: 5851, threadinfo ffff880fb33aa000, task ffff880f815ec100)
Jan 24 17:18:54 orca1 kernel: [1062984.192130] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.192139] ffffffff81122ba4 ffff880ffffece68 ffffffff81138231 000000000000003c
Jan 24 17:18:54 orca1 kernel: [1062984.192142] ffffc90000005000 ffffc90000004fff 0000000000000000 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.192145] ffff880b0a201e40 0000000000000000 ffff880b0a201e40 0000000000000001
Jan 24 17:18:54 orca1 kernel: [1062984.192148] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.192163] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.192168] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.192172] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.192185] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.192201] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.192257] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192383] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192501] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192587] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192706] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192790] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.192909] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193011] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193022] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.193027] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.193030] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.193054] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.193056] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.193059] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.193062] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.193068] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193077] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193101] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193215] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193332] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193417] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193534] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193617] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193734] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.193834] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.193841] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.193844] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.376010] BUG: soft lockup - CPU#3 stuck for 67s! [z_wr_iss/3:5853]
Jan 24 17:18:54 orca1 kernel: [1062984.376012] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.376051] CPU 3
Jan 24 17:18:54 orca1 kernel: [1062984.376052] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.376079]
Jan 24 17:18:54 orca1 kernel: [1062984.376081] Pid: 5853, comm: z_wr_iss/3 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.376084] RIP: 0010:[] [] rb_next+0x27/0x50
Jan 24 17:18:54 orca1 kernel: [1062984.376092] RSP: 0018:ffff880fa7dcf9d8 EFLAGS: 00000286
Jan 24 17:18:54 orca1 kernel: [1062984.376093] RAX: ffff880c5739a5d8 RBX: ffffffffffffffff RCX: ffff8807842ce558
Jan 24 17:18:54 orca1 kernel: [1062984.376095] RDX: ffff8809b1968e58 RSI: 0000000000000000 RDI: ffff88015516a658
Jan 24 17:18:54 orca1 kernel: [1062984.376096] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.376098] R10: ffffc9035502e020 R11: 000000000000002c R12: ffff880f8f7f8180
Jan 24 17:18:54 orca1 kernel: [1062984.376099] R13: ffffc90355196000 R14: ffffe8ffffffffff R15: ffff8802e75c3958
Jan 24 17:18:54 orca1 kernel: [1062984.376101] FS: 0000000000000000(0000) GS:ffff8800cfcc0000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.376103] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.376104] CR2: 00007f6295d7b000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.376106] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.376108] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.376110] Process z_wr_iss/3 (pid: 5853, threadinfo ffff880fa7dce000, task ffff880f8f7f8180)
Jan 24 17:18:54 orca1 kernel: [1062984.376111] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.376117] ffffffff81122c88 ffff880fa7dcfbd0 0000000000000001 0000000081a3ae64
Jan 24 17:18:54 orca1 kernel: [1062984.376121] ffffc90000009000 ffffc90000008fff ffffc90000000000 0000000000009000
Jan 24 17:18:54 orca1 kernel: [1062984.376124] ffff8802e75c3d40 0000000000000000 ffff8802e75c3d40 0000000000000212
Jan 24 17:18:54 orca1 kernel: [1062984.376126] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.376137] [] alloc_vmap_area+0x1b8/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.376141] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.376145] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.376155] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376167] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376204] [] vdev_raidz_io_start+0x2ad/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376293] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376411] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376494] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376614] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376715] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376724] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.376728] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.376730] Code: f3 c3 f3 c3 48 8b 17 31 c0 48 89 d1 48 83 e1 fc 48 39 cf 74 34 48 8b 47 08 48 85 c0 75 09 eb 1a 0f 1f 40 00 48 89 d0 48 8b 50 10 <48> 85 d2 75 f4 f3 c3 66 90 48 8b 10 48 89 c7 48 89 d0 48 83 e0
Jan 24 17:18:54 orca1 kernel: [1062984.376754] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.376756] [] alloc_vmap_area+0x1b8/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.376759] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.376762] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.376768] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376777] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.376802] [] vdev_raidz_io_start+0x2ad/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.376886] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377002] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377085] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377202] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.377302] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.377309] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.377312] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.468011] BUG: soft lockup - CPU#4 stuck for 67s! [z_wr_iss/4:5854]
Jan 24 17:18:54 orca1 kernel: [1062984.468015] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.468064] CPU 4
Jan 24 17:18:54 orca1 kernel: [1062984.468065] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.468091]
Jan 24 17:18:54 orca1 kernel: [1062984.468093] Pid: 5854, comm: z_wr_iss/4 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.468097] RIP: 0010:[] [] _raw_spin_lock+0x18/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.468106] RSP: 0018:ffff880f947c9968 EFLAGS: 00000297
Jan 24 17:18:54 orca1 kernel: [1062984.468108] RAX: 0000000000003b4d RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.468109] RDX: 0000000000003b49 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.468111] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.468112] R10: ffffc903551ba020 R11: 000000000000007b R12: ffff880ffffece00
Jan 24 17:18:54 orca1 kernel: [1062984.468114] R13: ffffffff810fb3e6 R14: 0000000000001270 R15: ffff880ffffedc00
Jan 24 17:18:54 orca1 kernel: [1062984.468116] FS: 0000000000000000(0000) GS:ffff8800cfd00000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468117] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.468119] CR2: 00007ff925767000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.468121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.468124] Process z_wr_iss/4 (pid: 5854, threadinfo ffff880f947c8000, task ffff880f903c61c0)
Jan 24 17:18:54 orca1 kernel: [1062984.468126] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.468134] ffffffff81122ba4 ffff880f903c61c0 ffffffff551bdfff 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468138] ffffc90000005000 ffffc90000004fff ffff880ffffece00 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.468141] ffff880a54931bc0 0000000000000000 ffff880a54931bc0 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.468144] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.468158] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.468163] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.468166] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.468180] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468196] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468249] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468372] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468490] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468576] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468694] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468778] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.468880] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468890] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.468895] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.468897] Code: 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 0f b7 17 f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c 89 64 24
Jan 24 17:18:54 orca1 kernel: [1062984.468922] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.468924] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.468927] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.468930] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.468936] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468944] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.468968] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469083] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469199] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469284] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469400] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469483] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.469582] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.469590] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.469592] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.560010] BUG: soft lockup - CPU#5 stuck for 67s! [z_wr_iss/5:5855]
Jan 24 17:18:54 orca1 kernel: [1062984.560013] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.560066] CPU 5
Jan 24 17:18:54 orca1 kernel: [1062984.560066] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.560093]
Jan 24 17:18:54 orca1 kernel: [1062984.560096] Pid: 5855, comm: z_wr_iss/5 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.560100] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.560109] RSP: 0018:ffff880ffefb78f8 EFLAGS: 00000287
Jan 24 17:18:54 orca1 kernel: [1062984.560110] RAX: 0000000000003b50 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.560112] RDX: 0000000000003b4a RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.560113] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.560115] R10: ffffc903551d0020 R11: 000000000000002a R12: 0000000200000246
Jan 24 17:18:54 orca1 kernel: [1062984.560117] R13: ffff880ffffece00 R14: ffffffff8110db74 R15: fffffffffffece00
Jan 24 17:18:54 orca1 kernel: [1062984.560118] FS: 0000000000000000(0000) GS:ffff8800cfd40000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560120] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.560122] CR2: 00007ff925760000 CR3: 0000000f8c5cc000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.560124] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560125] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.560127] Process z_wr_iss/5 (pid: 5855, threadinfo ffff880ffefb6000, task ffff880ffadf2600)
Jan 24 17:18:54 orca1 kernel: [1062984.560129] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.560140] ffffffff81122ba4 ffff880ffadf2600 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.560144] ffffc90000005000 ffffc90000004fff ffff88047c197b80 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.560147] ffff88083b6832c0 0000000000000000 ffff88083b6832c0 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.560150] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.560165] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.560170] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.560173] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.560187] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.560203] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.560259] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560384] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560502] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560589] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560707] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560791] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.560911] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561013] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561024] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.561029] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.561032] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.561057] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.561059] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.561062] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.561065] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.561071] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561080] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561104] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561219] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561335] [] vdev_raidz_io_start+0x692/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561420] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561537] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561620] [] zio_ready+0x311/0x3c0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561738] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.561838] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.561845] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.561848] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.744011] BUG: soft lockup - CPU#7 stuck for 67s! [z_wr_iss/7:5857]
Jan 24 17:18:54 orca1 kernel: [1062984.744014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.744062] CPU 7
Jan 24 17:18:54 orca1 kernel: [1062984.744063] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1062984.744090]
Jan 24 17:18:54 orca1 kernel: [1062984.744092] Pid: 5857, comm: z_wr_iss/7 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1062984.744096] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1062984.744104] RSP: 0018:ffff880f901b9968 EFLAGS: 00000283
Jan 24 17:18:54 orca1 kernel: [1062984.744105] RAX: 0000000000003b53 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1062984.744107] RDX: 0000000000003b4e RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1062984.744109] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1062984.744110] R10: ffffc903551e2020 R11: 000000000000006a R12: 0000000200000041
Jan 24 17:18:54 orca1 kernel: [1062984.744112] R13: ffff880ffffece00 R14: 00000000000612d0 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1062984.744114] FS: 0000000000000000(0000) GS:ffff8800cfdc0000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744115] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1062984.744117] CR2: 00007ff925768000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1062984.744119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1062984.744122] Process z_wr_iss/7 (pid: 5857, threadinfo ffff880f901b8000, task ffff880f89bb2200)
Jan 24 17:18:54 orca1 kernel: [1062984.744123] Stack:
Jan 24 17:18:54 orca1 kernel: [1062984.744131] ffffffff81122ba4 ffff880f89bb2200 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1062984.744135] ffffc90000005000 ffffc90000004fff ffff8806a6e7e900 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1062984.744138] ffff880ac9df4e40 0000000000000000 ffff880ac9df4e40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1062984.744141] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.744154] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.744160] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.744163] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.744175] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744191] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744242] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744362] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744480] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744566] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744684] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744768] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.744870] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744879] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.744884] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1062984.744887] Code: 74 07 f3 90 0f b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 17 eb f5 c3 90 90 90 90 90 48 83 ec 28 48 89 6c 24 08 4c
Jan 24 17:18:54 orca1 kernel: [1062984.744911] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1062984.744914] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1062984.744917] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1062984.744919] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1062984.744925] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744934] [] spl_kmem_cache_alloc+0x335/0x600 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.744958] [] zio_create+0x6d/0x460 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745072] [] zio_vdev_child_io+0xb5/0xe0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745188] [] vdev_raidz_io_start+0x5ef/0x6d0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745273] [] zio_nowait+0x9e/0x160 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745389] [] vdev_mirror_io_start+0x177/0x3b0 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745472] [] zio_execute+0x96/0x130 [zfs]
Jan 24 17:18:54 orca1 kernel: [1062984.745571] [] taskq_thread+0x1d6/0x3a0 [spl]
Jan 24 17:18:54 orca1 kernel: [1062984.745579] [] kthread+0x96/0xa0
Jan 24 17:18:54 orca1 kernel: [1062984.745582] [] kernel_thread_helper+0x4/0x10
Jan 24 17:18:54 orca1 kernel: [1063056.284010] BUG: soft lockup - CPU#2 stuck for 67s! [z_wr_iss/2:5852]
Jan 24 17:18:54 orca1 kernel: [1063056.284014] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1063056.284063] CPU 2
Jan 24 17:18:54 orca1 kernel: [1063056.284064] Modules linked in: nfsd exportfs autofs4 snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore snd_page_alloc edd rpcsec_gss_krb5 nfs lockd fscache nfs_acl auth_rpcgss sunrpc microcode mperf dm_mod zfs(P) zcommon(P) znvpair(P) zavl(P) zunicode(P) spl zlib_deflate radeon ttm iTCO_wdt drm_kms_helper iTCO_vendor_support i5000_edac edac_core sr_mod i5k_amb sg drm e1000e cdrom ppdev ib_mthca ib_mad ioatdma parport_pc ib_core floppy parport i2c_i801 i2c_algo_bit pcspkr dca container button shpchp pci_hotplug ghes hed ext4 jbd2 crc16 uhci_hcd ehci_hcd usbcore sata_mv sata_sil24 ata_generic fan processor thermal thermal_sys [last unloaded: preloadtrace]
Jan 24 17:18:54 orca1 kernel: [1063056.284091]
Jan 24 17:18:54 orca1 kernel: [1063056.284093] Pid: 5852, comm: z_wr_iss/2 Tainted: P 2.6.37.6-0.9-default #1 Supermicro X7DB8/X7DB8
Jan 24 17:18:54 orca1 kernel: [1063056.284097] RIP: 0010:[] [] _raw_spin_lock+0x15/0x20
Jan 24 17:18:54 orca1 kernel: [1063056.284105] RSP: 0018:ffff880f917e9968 EFLAGS: 00000293
Jan 24 17:18:54 orca1 kernel: [1063056.284107] RAX: 00000000000040a2 RBX: ffffffffffffffff RCX: ffffe8ffffffffff
Jan 24 17:18:54 orca1 kernel: [1063056.284109] RDX: 00000000000040a0 RSI: 0000000000000000 RDI: ffffffff81d45580
Jan 24 17:18:54 orca1 kernel: [1063056.284110] RBP: ffffffff810037ee R08: 00000000ffffffff R09: 0000000000001270
Jan 24 17:18:54 orca1 kernel: [1063056.284112] R10: ffffc90374dd0020 R11: 000000000000006f R12: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284113] R13: ffff880ffffece00 R14: 0000000000000002 R15: ffffffff00000000
Jan 24 17:18:54 orca1 kernel: [1063056.284115] FS: 0000000000000000(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284117] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 24 17:18:54 orca1 kernel: [1063056.284119] CR2: 00007ff925765000 CR3: 0000000001a03000 CR4: 00000000000006e0
Jan 24 17:18:54 orca1 kernel: [1063056.284120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284122] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 24 17:18:54 orca1 kernel: [1063056.284124] Process z_wr_iss/2 (pid: 5852, threadinfo ffff880f917e8000, task ffff880f91bea140)
Jan 24 17:18:54 orca1 kernel: [1063056.284125] Stack:
Jan 24 17:18:54 orca1 kernel: [1063056.284133] ffffffff81122ba4 0000000000001272 8000000000000163 0000000088f45068
Jan 24 17:18:54 orca1 kernel: [1063056.284137] ffffc90000005000 ffffc90000004fff ffff880675f8fbc0 0000000000005000
Jan 24 17:18:54 orca1 kernel: [1063056.284140] ffff880ab32d6a40 0000000000000000 ffff880ab32d6a40 0000008000000000
Jan 24 17:18:54 orca1 kernel: [1063056.284143] Call Trace:
Jan 24 17:18:54 orca1 kernel: [1063056.284158] [] alloc_vmap_area+0xd4/0x370
Jan 24 17:18:54 orca1 kernel: [1063056.284162] [] __get_vm_area_node+0xe8/0x220
Jan 24 17:18:54 orca1 kernel: [1063056.284166] [] __vmalloc_node+0x89/0xc0
Jan 24 17:18:54 orca1 kernel: [1063056.284179] [] kv_alloc+0x5e/0x90 [spl]
Jan 24 17:18:54 orca1 kernel: [1063056.284195] [
The text was updated successfully, but these errors were encountered: