Kernel panics when i start incremental recv.

[morphinz]

System information

Type	Version/Name
Distribution Name	ARCH Linux
Distribution Version	SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018
Linux Kernel	Server A=4.13.11-1-ARCH & B=4.14.13-1-ARCH
Architecture	x86_64
ZFS Version	Server A=0.7.3-1 & B=0.7.5-1
SPL Version	Server A=0.7.3-1 & B=0.7.5-1

Describe the problem you're observing

When i start zfs send -recv with resume token kernel panics!..
I have 2 pool on 2 server and running replication on them via WAN. 
On "A" pool to "B" pool i have 10++ dataset replication but only 1 of them has this problem.
When i start send-recv on this dataset with resume token to B, "B" node goes panic everytime.
Other datasets are just fine. I don't see any log when i start them.

I tried with older kernel and newer kernel or "pti=off" nothing changed.
And i have this problem only on 1 dataset. For this reason I think the problem is related to zfs.

Include any warning/errors/backtraces from the system logs

When i start zfs send with resume token kernel gives this log and panics:

Jan 29 18:33:58 FKM1 kernel: general protection fault: 0000 [#1] PREEMPT SMP PTI
Jan 29 18:33:58 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:33:58 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:33:58 FKM1 kernel: CPU: 22 PID: 24026 Comm: receive_writer Tainted: P           O    4.14.13-1-ARCH #1
Jan 29 18:33:58 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:33:58 FKM1 kernel: task: ffff955bf5615880 task.stack: ffffbacef1e98000
Jan 29 18:33:58 FKM1 kernel: RIP: 0010:__kmalloc_node+0x16d/0x2d0
Jan 29 18:33:58 FKM1 kernel: RSP: 0018:ffffbacef1e9bce0 EFLAGS: 00010206
Jan 29 18:33:58 FKM1 kernel: RAX: 0000000000000000 RBX: 79df0a76b6f263ff RCX: 000000038f33f116
Jan 29 18:33:58 FKM1 kernel: RDX: 000000038f33f096 RSI: 000000038f33f096 RDI: 0000000000022d60
Jan 29 18:33:58 FKM1 kernel: RBP: ffff955be69dfe00 R08: ffff951e7f407000 R09: ffff955be69dfe00
Jan 29 18:33:58 FKM1 kernel: R10: 86209f2d506f9dff R11: ffffffffc0653097 R12: 000000000140c200
Jan 29 18:33:58 FKM1 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff951e7f407000
Jan 29 18:33:58 FKM1 kernel: FS:  0000000000000000(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:33:58 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:33:58 FKM1 kernel: CR2: 00007f8a6009f000 CR3: 0000001c4500a001 CR4: 00000000003606e0
Jan 29 18:33:58 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:33:58 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:33:58 FKM1 kernel: Call Trace:
Jan 29 18:33:58 FKM1 kernel:  ? spl_kmem_alloc+0xd0/0x180 [spl]
Jan 29 18:33:58 FKM1 kernel:  spl_kmem_zalloc+0xc7/0x180 [spl]
Jan 29 18:33:58 FKM1 kernel:  dbuf_dirty+0x200/0x820 [zfs]
Jan 29 18:33:58 FKM1 kernel:  receive_object+0x25f/0x470 [zfs]
Jan 29 18:33:58 FKM1 kernel:  ? dmu_free_long_range+0x351/0x440 [zfs]
Jan 29 18:33:58 FKM1 kernel:  receive_writer_thread+0x574/0x9d0 [zfs]
Jan 29 18:33:58 FKM1 kernel:  ? _raw_spin_unlock_irqrestore+0x26/0x30
Jan 29 18:33:58 FKM1 kernel:  ? set_user_nice.part.70+0x140/0x220
Jan 29 18:33:58 FKM1 kernel:  ? receive_free.isra.13+0xb0/0xb0 [zfs]
Jan 29 18:33:58 FKM1 kernel:  ? __thread_exit+0x20/0x20 [spl]
Jan 29 18:33:58 FKM1 kernel:  thread_generic_wrapper+0x6d/0x80 [spl]
Jan 29 18:33:58 FKM1 kernel:  kthread+0x118/0x130
Jan 29 18:33:58 FKM1 kernel:  ? kthread_create_on_node+0x70/0x70
Jan 29 18:33:58 FKM1 kernel:  ret_from_fork+0x1f/0x30
Jan 29 18:33:58 FKM1 kernel: Code: 8b 11 4d 33 97 40 01 00 00 4c 89 cb 4c 31 d3 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 f0 fe ff ff 4d 39 d1 74 14 49 63 47 20 48 01 c3 <48> 33 1b 49 33 9f 40 01 00 00 0f 18 0b 41 f7 c4 00 80 00 00 0f
Jan 29 18:33:58 FKM1 kernel: RIP: __kmalloc_node+0x16d/0x2d0 RSP: ffffbacef1e9bce0
Jan 29 18:33:58 FKM1 kernel: ---[ end trace fec613348e6273b5 ]---
Jan 29 18:34:01 FKM1 kernel: general protection fault: 0000 [#2] PREEMPT SMP PTI
Jan 29 18:34:01 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:01 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:01 FKM1 kernel: CPU: 22 PID: 15180 Comm: java Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:01 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:01 FKM1 kernel: task: ffff951e781949c0 task.stack: ffffbace4e5e8000
Jan 29 18:34:01 FKM1 kernel: RIP: 0010:kmem_cache_alloc_node+0xb4/0x1c0
Jan 29 18:34:01 FKM1 kernel: RSP: 0018:ffffbace4e5ebb98 EFLAGS: 00010206
Jan 29 18:34:01 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 79df0a76b6f263ff RCX: 000000038f33f196
Jan 29 18:34:01 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 0000000000022d60
Jan 29 18:34:01 FKM1 kernel: RBP: 00000000014000c0 R08: ffffffff8b52d7b7 R09: ffff951c0bbbd800
Jan 29 18:34:01 FKM1 kernel: R10: ffff951e786a0000 R11: 0000000000000000 R12: 00000000ffffffff
Jan 29 18:34:01 FKM1 kernel: R13: 79df0a76b6f263ff R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:01 FKM1 kernel: FS:  00007f8cc443a700(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:01 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:01 FKM1 kernel: CR2: 00007f8a6009f000 CR3: 0000003f0fe34006 CR4: 00000000003606e0
Jan 29 18:34:01 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:01 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:01 FKM1 kernel: Call Trace:
Jan 29 18:34:01 FKM1 kernel:  __alloc_skb+0x57/0x1e0
Jan 29 18:34:01 FKM1 kernel:  ? tcp_current_mss+0x51/0xa0
Jan 29 18:34:01 FKM1 kernel:  sk_stream_alloc_skb+0x4d/0x1e0
Jan 29 18:34:01 FKM1 kernel:  tcp_sendmsg_locked+0x3c4/0xe50
Jan 29 18:34:01 FKM1 kernel:  ? update_curr+0x154/0x1c0
Jan 29 18:34:01 FKM1 kernel:  ? update_curr+0x70/0x1c0
Jan 29 18:34:01 FKM1 kernel:  tcp_sendmsg+0x27/0x40
Jan 29 18:34:01 FKM1 kernel:  sock_write_iter+0x9d/0x110
Jan 29 18:34:01 FKM1 kernel:  do_iter_readv_writev+0x11a/0x170
Jan 29 18:34:01 FKM1 kernel:  do_iter_write+0x80/0x190
Jan 29 18:34:01 FKM1 kernel:  vfs_writev+0x98/0x110
Jan 29 18:34:01 FKM1 kernel:  ? ep_poll+0xa7/0x3a0
Jan 29 18:34:01 FKM1 kernel:  ? do_writev+0x5e/0xf0
Jan 29 18:34:01 FKM1 kernel:  do_writev+0x5e/0xf0
Jan 29 18:34:01 FKM1 kernel:  ? SyS_read+0x76/0xc0
Jan 29 18:34:01 FKM1 kernel:  entry_SYSCALL_64_fastpath+0x1a/0x7d
Jan 29 18:34:01 FKM1 kernel: RIP: 0033:0x7f8d4547fe54
Jan 29 18:34:01 FKM1 kernel: RSP: 002b:00007f8cc4437990 EFLAGS: 00000293
Jan 29 18:34:01 FKM1 kernel: Code: 74 56 48 85 c0 74 51 49 63 56 1c 31 f6 4c 89 ef e8 e2 1f 46 00 eb 41 49 63 5e 20 49 8b 3e 48 8d 8a 80 00 00 00 4c 89 e8 4c 01 eb <48> 33 1b 49 33 9e 40 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 0f
Jan 29 18:34:01 FKM1 kernel: RIP: kmem_cache_alloc_node+0xb4/0x1c0 RSP: ffffbace4e5ebb98
Jan 29 18:34:01 FKM1 kernel: ---[ end trace fec613348e6273b6 ]---
Jan 29 18:34:02 FKM1 kernel: general protection fault: 0000 [#3] PREEMPT SMP PTI
Jan 29 18:34:02 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:02 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:02 FKM1 kernel: CPU: 22 PID: 11687 Comm: java Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:02 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:02 FKM1 kernel: task: ffff955e7561e740 task.stack: ffffbacec6640000
Jan 29 18:34:02 FKM1 kernel: RIP: 0010:kmem_cache_alloc_node+0xb4/0x1c0
Jan 29 18:34:02 FKM1 kernel: RSP: 0018:ffffbacec6643c70 EFLAGS: 00010206
Jan 29 18:34:02 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 79df0a76b6f263ff RCX: 000000038f33f196
Jan 29 18:34:02 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 0000000000022d60
Jan 29 18:34:02 FKM1 kernel: RBP: 00000000014000c0 R08: ffffffff8b52d7b7 R09: ffff951ba6ed2800
Jan 29 18:34:02 FKM1 kernel: R10: 0000000000014fc2 R11: 0000000000000000 R12: 00000000ffffffff
Jan 29 18:34:02 FKM1 kernel: R13: 79df0a76b6f263ff R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:02 FKM1 kernel: FS:  00007f8c75f3b700(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:02 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:02 FKM1 kernel: CR2: 00007f8a6009f000 CR3: 0000003f0fe34006 CR4: 00000000003606e0
Jan 29 18:34:02 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:02 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:02 FKM1 kernel: Call Trace:
Jan 29 18:34:02 FKM1 kernel:  __alloc_skb+0x57/0x1e0
Jan 29 18:34:02 FKM1 kernel:  ? tcp_current_mss+0x51/0xa0
Jan 29 18:34:02 FKM1 kernel:  sk_stream_alloc_skb+0x4d/0x1e0
Jan 29 18:34:02 FKM1 kernel:  tcp_sendmsg_locked+0x3c4/0xe50
Jan 29 18:34:02 FKM1 kernel:  tcp_sendmsg+0x27/0x40
Jan 29 18:34:02 FKM1 kernel:  sock_write_iter+0x9d/0x110
Jan 29 18:34:02 FKM1 kernel:  __vfs_write+0xf9/0x170
Jan 29 18:34:02 FKM1 kernel:  vfs_write+0xad/0x1a0
Jan 29 18:34:02 FKM1 kernel:  SyS_write+0x52/0xc0
Jan 29 18:34:02 FKM1 kernel:  entry_SYSCALL_64_fastpath+0x1a/0x7d
Jan 29 18:34:02 FKM1 kernel: RIP: 0033:0x7f8d45b748cb
Jan 29 18:34:02 FKM1 kernel: RSP: 002b:00007f8c75f38ba0 EFLAGS: 00000293
Jan 29 18:34:02 FKM1 kernel: Code: 74 56 48 85 c0 74 51 49 63 56 1c 31 f6 4c 89 ef e8 e2 1f 46 00 eb 41 49 63 5e 20 49 8b 3e 48 8d 8a 80 00 00 00 4c 89 e8 4c 01 eb <48> 33 1b 49 33 9e 40 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 0f
Jan 29 18:34:02 FKM1 kernel: RIP: kmem_cache_alloc_node+0xb4/0x1c0 RSP: ffffbacec6643c70
Jan 29 18:34:02 FKM1 kernel: ---[ end trace fec613348e6273b7 ]---
Jan 29 18:34:08 FKM1 kernel: general protection fault: 0000 [#4] PREEMPT SMP PTI
Jan 29 18:34:08 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:08 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:08 FKM1 kernel: CPU: 22 PID: 14476 Comm: java Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:08 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:08 FKM1 kernel: task: ffff951dac773b00 task.stack: ffffbaced36e8000
Jan 29 18:34:08 FKM1 kernel: RIP: 0010:kmem_cache_alloc_node+0xb4/0x1c0
Jan 29 18:34:08 FKM1 kernel: RSP: 0018:ffffbaced36ebb98 EFLAGS: 00010206
Jan 29 18:34:08 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 79df0a76b6f263ff RCX: 000000038f33f196
Jan 29 18:34:08 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 0000000000022d60
Jan 29 18:34:08 FKM1 kernel: RBP: 00000000014000c0 R08: ffffffff8b52d7b7 R09: ffff951c0ff6ac00
Jan 29 18:34:08 FKM1 kernel: R10: ffff951e786a0000 R11: 0000000000000000 R12: 00000000ffffffff
Jan 29 18:34:08 FKM1 kernel: R13: 79df0a76b6f263ff R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:08 FKM1 kernel: FS:  00007f8d03d51700(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:08 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:08 FKM1 kernel: CR2: 00007f8c43e9c4d7 CR3: 0000003f0fe34002 CR4: 00000000003606e0
Jan 29 18:34:08 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:08 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:08 FKM1 kernel: Call Trace:
Jan 29 18:34:08 FKM1 kernel:  __alloc_skb+0x57/0x1e0
Jan 29 18:34:08 FKM1 kernel:  ? tcp_current_mss+0x51/0xa0
Jan 29 18:34:08 FKM1 kernel:  sk_stream_alloc_skb+0x4d/0x1e0
Jan 29 18:34:08 FKM1 kernel:  tcp_sendmsg_locked+0x3c4/0xe50
Jan 29 18:34:08 FKM1 kernel:  ? update_load_avg+0x41d/0x590
Jan 29 18:34:08 FKM1 kernel:  tcp_sendmsg+0x27/0x40
Jan 29 18:34:08 FKM1 kernel:  sock_write_iter+0x9d/0x110
Jan 29 18:34:08 FKM1 kernel:  do_iter_readv_writev+0x11a/0x170
Jan 29 18:34:08 FKM1 kernel:  do_iter_write+0x80/0x190
Jan 29 18:34:08 FKM1 kernel:  vfs_writev+0x98/0x110
Jan 29 18:34:08 FKM1 kernel:  ? ep_poll+0xa7/0x3a0
Jan 29 18:34:08 FKM1 kernel:  ? do_writev+0x5e/0xf0
Jan 29 18:34:08 FKM1 kernel:  do_writev+0x5e/0xf0
Jan 29 18:34:08 FKM1 kernel:  ? SyS_read+0x76/0xc0
Jan 29 18:34:08 FKM1 kernel:  entry_SYSCALL_64_fastpath+0x1a/0x7d
Jan 29 18:34:08 FKM1 kernel: RIP: 0033:0x7f8d4547fe54
Jan 29 18:34:08 FKM1 kernel: RSP: 002b:00007f8d03d4ea10 EFLAGS: 00000293
Jan 29 18:34:08 FKM1 kernel: Code: 74 56 48 85 c0 74 51 49 63 56 1c 31 f6 4c 89 ef e8 e2 1f 46 00 eb 41 49 63 5e 20 49 8b 3e 48 8d 8a 80 00 00 00 4c 89 e8 4c 01 eb <48> 33 1b 49 33 9e 40 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 0f
Jan 29 18:34:08 FKM1 kernel: RIP: kmem_cache_alloc_node+0xb4/0x1c0 RSP: ffffbaced36ebb98
Jan 29 18:34:08 FKM1 kernel: ---[ end trace fec613348e6273b8 ]---
Jan 29 18:34:11 FKM1 kernel: general protection fault: 0000 [#5] PREEMPT SMP PTI
Jan 29 18:34:11 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:11 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:11 FKM1 kernel: CPU: 22 PID: 12274 Comm: java Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:11 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:11 FKM1 kernel: task: ffff955e3beeac40 task.stack: ffffbaced9d24000
Jan 29 18:34:11 FKM1 kernel: RIP: 0010:kmem_cache_alloc_node+0xb4/0x1c0
Jan 29 18:34:11 FKM1 kernel: RSP: 0018:ffffbaced9d27b98 EFLAGS: 00010206
Jan 29 18:34:11 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 79df0a76b6f263ff RCX: 000000038f33f196
Jan 29 18:34:11 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 0000000000022d60
Jan 29 18:34:11 FKM1 kernel: RBP: 00000000014000c0 R08: ffffffff8b52d7b7 R09: ffff951a18112500
Jan 29 18:34:11 FKM1 kernel: R10: ffff951e786a0000 R11: 0000000000000000 R12: 00000000ffffffff
Jan 29 18:34:11 FKM1 kernel: R13: 79df0a76b6f263ff R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:11 FKM1 kernel: FS:  00007f8d05a30700(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:11 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:11 FKM1 kernel: CR2: 00007ff730a90020 CR3: 0000003f0fe34002 CR4: 00000000003606e0
Jan 29 18:34:11 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:11 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:11 FKM1 kernel: Call Trace:
Jan 29 18:34:11 FKM1 kernel:  __alloc_skb+0x57/0x1e0
Jan 29 18:34:11 FKM1 kernel:  ? tcp_current_mss+0x51/0xa0
Jan 29 18:34:11 FKM1 kernel:  sk_stream_alloc_skb+0x4d/0x1e0
Jan 29 18:34:11 FKM1 kernel:  tcp_sendmsg_locked+0x3c4/0xe50
Jan 29 18:34:11 FKM1 kernel:  tcp_sendmsg+0x27/0x40
Jan 29 18:34:11 FKM1 kernel:  sock_write_iter+0x9d/0x110
Jan 29 18:34:11 FKM1 kernel:  do_iter_readv_writev+0x11a/0x170
Jan 29 18:34:11 FKM1 kernel:  do_iter_write+0x80/0x190
Jan 29 18:34:11 FKM1 kernel:  vfs_writev+0x98/0x110
Jan 29 18:34:11 FKM1 kernel:  ? ep_poll+0xa7/0x3a0
Jan 29 18:34:11 FKM1 kernel:  ? do_writev+0x5e/0xf0
Jan 29 18:34:11 FKM1 kernel:  do_writev+0x5e/0xf0
Jan 29 18:34:11 FKM1 kernel:  ? SyS_read+0x76/0xc0
Jan 29 18:34:11 FKM1 kernel:  entry_SYSCALL_64_fastpath+0x1a/0x7d
Jan 29 18:34:11 FKM1 kernel: RIP: 0033:0x7f8d4547fe54
Jan 29 18:34:11 FKM1 kernel: RSP: 002b:00007f8d05a2dc10 EFLAGS: 00000293
Jan 29 18:34:11 FKM1 kernel: Code: 74 56 48 85 c0 74 51 49 63 56 1c 31 f6 4c 89 ef e8 e2 1f 46 00 eb 41 49 63 5e 20 49 8b 3e 48 8d 8a 80 00 00 00 4c 89 e8 4c 01 eb <48> 33 1b 49 33 9e 40 01 00 00 65 48 0f c7 0f 0f 94 c0 84 c0 0f
Jan 29 18:34:11 FKM1 kernel: RIP: kmem_cache_alloc_node+0xb4/0x1c0 RSP: ffffbaced9d27b98
Jan 29 18:34:11 FKM1 kernel: ---[ end trace fec613348e6273b9 ]---
Jan 29 18:34:13 FKM1 kernel: general protection fault: 0000 [#6] PREEMPT SMP PTI
Jan 29 18:34:13 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:13 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:13 FKM1 kernel: CPU: 22 PID: 24692 Comm: IPaddr2 Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:13 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:13 FKM1 kernel: task: ffff951c68588ec0 task.stack: ffffbacef772c000
Jan 29 18:34:13 FKM1 kernel: RIP: 0010:__kmalloc+0x95/0x220
Jan 29 18:34:13 FKM1 kernel: RSP: 0018:ffffbacef772fd38 EFLAGS: 00010206
Jan 29 18:34:13 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 00000000000001f8 RCX: 000000038f33f196
Jan 29 18:34:13 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 79df0a76b6f263ff
Jan 29 18:34:13 FKM1 kernel: RBP: 79df0a76b6f263ff R08: 0000000000022d60 R09: ffffffff8b2944e9
Jan 29 18:34:13 FKM1 kernel: R10: 8080808080808080 R11: ffff951b41322f00 R12: 00000000014000c0
Jan 29 18:34:13 FKM1 kernel: R13: 00000000000001f8 R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:13 FKM1 kernel: FS:  00007f0b38498040(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:13 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:13 FKM1 kernel: CR2: 00000000006d6fa4 CR3: 0000003ad2a6c005 CR4: 00000000003606e0
Jan 29 18:34:13 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:13 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:13 FKM1 kernel: Call Trace:
Jan 29 18:34:13 FKM1 kernel:  load_elf_phdrs+0x59/0xb0
Jan 29 18:34:13 FKM1 kernel:  load_elf_binary+0x109/0x15c0
Jan 29 18:34:13 FKM1 kernel:  ? get_user_pages_remote+0x137/0x1f0
Jan 29 18:34:13 FKM1 kernel:  ? __check_object_size+0xaf/0x1b0
Jan 29 18:34:13 FKM1 kernel:  ? _copy_from_user+0x37/0x60
Jan 29 18:34:13 FKM1 kernel:  ? copy_strings.isra.25+0x2a7/0x450
Jan 29 18:34:13 FKM1 kernel:  search_binary_handler+0x76/0x230
Jan 29 18:34:13 FKM1 kernel:  do_execveat_common.isra.36+0x6a5/0x830
Jan 29 18:34:13 FKM1 kernel:  SyS_execve+0x2c/0x40
Jan 29 18:34:13 FKM1 kernel:  do_syscall_64+0x55/0x110
Jan 29 18:34:13 FKM1 kernel:  entry_SYSCALL64_slow_path+0x25/0x25
Jan 29 18:34:13 FKM1 kernel: RIP: 0033:0x7f0b37b45f17
Jan 29 18:34:13 FKM1 kernel: RSP: 002b:00007ffd2d12be48 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
Jan 29 18:34:13 FKM1 kernel: RAX: ffffffffffffffda RBX: 0000000002688ab0 RCX: 00007f0b37b45f17
Jan 29 18:34:13 FKM1 kernel: RDX: 00000000026809e0 RSI: 0000000002688cc0 RDI: 0000000002688ab0
Jan 29 18:34:13 FKM1 kernel: RBP: 0000000000000000 R08: 00007ffd2d12be40 R09: 0000000000000000
Jan 29 18:34:13 FKM1 kernel: R10: 00000000000005b8 R11: 0000000000000206 R12: 0000000002688ab0
Jan 29 18:34:13 FKM1 kernel: R13: 0000000002688cc0 R14: 00000000026809e0 R15: 0000000000000040
Jan 29 18:34:13 FKM1 kernel: Code: 48 83 78 10 00 48 8b 28 0f 84 30 01 00 00 48 85 ed 0f 84 27 01 00 00 49 63 7e 20 4d 8b 06 48 8d 8a 80 00 00 00 48 89 e8 48 01 ef <4c> 8b 17 4d 33 96 40 01 00 00 48 89 fb 4c 31 d3 65 49 0f c7 08
Jan 29 18:34:13 FKM1 kernel: RIP: __kmalloc+0x95/0x220 RSP: ffffbacef772fd38
Jan 29 18:34:13 FKM1 kernel: ---[ end trace fec613348e6273ba ]---
Jan 29 18:34:13 FKM1 kernel: BUG: Bad rss-counter state mm:ffff951e764f2640 idx:1 val:1
Jan 29 18:34:31 FKM1 kernel: general protection fault: 0000 [#7] PREEMPT SMP PTI
Jan 29 18:34:31 FKM1 kernel: Modules linked in: bridge stp llc dm_service_time scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr ipmi_ssif ttm input_leds drm_kms_helper led_class drm evdev joydev 8021q agpgart mousedev mrp syscopyarea sysfillrect mac_hid ixgbe igb sysimgblt i2c_i801 fb_sys_fops mei_me mdio ioatdma mei ptp i2c_algo_bit pps_core lpc_ich dca ipmi_si shpchp tpm_tis ipmi_devintf tpm_tis_core tpm ipmi_msghandler wmi acpi_power_meter acpi_pad button bonding nfsd sch_fq_codel auth_rpcgss oid_registry nfs_acl lockd grace sunrpc ip_tables x_tables zfs(PO) zunicode(PO)
Jan 29 18:34:31 FKM1 kernel:  zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ses enclosure sd_mod hid_generic usbhid hid ahci libahci libata mpt3sas raid_class scsi_transport_sas xhci_pci dm_multipath dm_mod scsi_mod xfs libcrc32c crc32c_generic crc32c_intel xhci_hcd usbcore usb_common
Jan 29 18:34:31 FKM1 kernel: CPU: 22 PID: 25260 Comm: IPaddr2 Tainted: P      D    O    4.14.13-1-ARCH #1
Jan 29 18:34:31 FKM1 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Jan 29 18:34:31 FKM1 kernel: task: ffff955e5de52c40 task.stack: ffffbacefca14000
Jan 29 18:34:31 FKM1 kernel: RIP: 0010:__kmalloc+0x95/0x220
Jan 29 18:34:31 FKM1 kernel: RSP: 0018:ffffbacefca17d38 EFLAGS: 00010206
Jan 29 18:34:31 FKM1 kernel: RAX: 79df0a76b6f263ff RBX: 00000000000001f8 RCX: 000000038f33f196
Jan 29 18:34:31 FKM1 kernel: RDX: 000000038f33f116 RSI: 000000038f33f116 RDI: 79df0a76b6f263ff
Jan 29 18:34:31 FKM1 kernel: RBP: 79df0a76b6f263ff R08: 0000000000022d60 R09: ffffffff8b2944e9
Jan 29 18:34:31 FKM1 kernel: R10: 8080808080808080 R11: ffff951b41322a00 R12: 00000000014000c0
Jan 29 18:34:31 FKM1 kernel: R13: 00000000000001f8 R14: ffff951e7f407000 R15: ffff951e7f407000
Jan 29 18:34:31 FKM1 kernel: FS:  00007f4428029040(0000) GS:ffff951e7fd80000(0000) knlGS:0000000000000000
Jan 29 18:34:31 FKM1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 18:34:31 FKM1 kernel: CR2: 00007f44279c5650 CR3: 0000003a847e2006 CR4: 00000000003606e0
Jan 29 18:34:31 FKM1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 18:34:31 FKM1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 18:34:31 FKM1 kernel: Call Trace:
Jan 29 18:34:31 FKM1 kernel:  load_elf_phdrs+0x59/0xb0
Jan 29 18:34:31 FKM1 kernel:  load_elf_binary+0x109/0x15c0
Jan 29 18:34:31 FKM1 kernel:  ? get_user_pages_remote+0x137/0x1f0
Jan 29 18:34:31 FKM1 kernel:  ? __check_object_size+0xaf/0x1b0
Jan 29 18:34:31 FKM1 kernel:  ? _copy_from_user+0x37/0x60
Jan 29 18:34:31 FKM1 kernel:  ? copy_strings.isra.25+0x2a7/0x450
Jan 29 18:34:31 FKM1 kernel:  search_binary_handler+0x76/0x230
Jan 29 18:34:31 FKM1 kernel:  do_execveat_common.isra.36+0x6a5/0x830
Jan 29 18:34:31 FKM1 kernel:  SyS_execve+0x2c/0x40
Jan 29 18:34:31 FKM1 kernel:  do_syscall_64+0x55/0x110
Jan 29 18:34:31 FKM1 kernel:  entry_SYSCALL64_slow_path+0x25/0x25
Jan 29 18:34:31 FKM1 kernel: RIP: 0033:0x7f44276d6f17
Jan 29 18:34:31 FKM1 kernel: RSP: 002b:00007ffcbd1db9c8 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
Jan 29 18:34:31 FKM1 kernel: RAX: ffffffffffffffda RBX: 000000000162ea40 RCX: 00007f44276d6f17
Jan 29 18:34:31 FKM1 kernel: RDX: 00000000016289e0 RSI: 000000000162e6f0 RDI: 000000000162ea40
Jan 29 18:34:31 FKM1 kernel: RBP: 0000000000000000 R08: 00007f44279c6720 R09: 00007f4428029040
Jan 29 18:34:31 FKM1 kernel: R10: 00000000000005b8 R11: 0000000000000202 R12: 000000000162ea40
Jan 29 18:34:31 FKM1 kernel: R13: 000000000162e6f0 R14: 00000000016289e0 R15: 0000000000000040
Jan 29 18:34:31 FKM1 kernel: Code: 48 83 78 10 00 48 8b 28 0f 84 30 01 00 00 48 85 ed 0f 84 27 01 00 00 49 63 7e 20 4d 8b 06 48 8d 8a 80 00 00 00 48 89 e8 48 01 ef <4c> 8b 17 4d 33 96 40 01 00 00 48 89 fb 4c 31 d3 65 49 0f c7 08
Jan 29 18:34:31 FKM1 kernel: RIP: __kmalloc+0x95/0x220 RSP: ffffbacefca17d38
Jan 29 18:34:31 FKM1 kernel: ---[ end trace fec613348e6273bb ]---
Write failed: Broken pipe

NAME                  PROPERTY              VALUE                                                                                                                                                                                                                                                                                                                                                  SOURCE
poolB/dataset  type                  filesystem                                  -
poolB/dataset  creation              Wed Jan 17 12:49 2018                       -
poolB/dataset  used                  27.2T										-
poolB/dataset  available             43.8T										-
poolB/dataset  referenced            27.2T										-
poolB/dataset  compressratio         1.06x										-
poolB/dataset  mounted               yes											-
poolB/dataset  quota                 71T											received
poolB/dataset  reservation           none										received
poolB/dataset  recordsize            128K										received
poolB/dataset  mountpoint            /apool/test									default
poolB/dataset  sharenfs              off											default
poolB/dataset  checksum              on											default
poolB/dataset  compression           off											local
poolB/dataset  atime                 off											received
poolB/dataset  devices               on											default
poolB/dataset  exec                  on											default
poolB/dataset  setuid                on											default
poolB/dataset  readonly              off											received
poolB/dataset  zoned                 off											default
poolB/dataset  snapdir               hidden										default
poolB/dataset  aclinherit            passthrough									received
poolB/dataset  createtxg             1208910										-
poolB/dataset  canmount              on											default
poolB/dataset  xattr                 sa											received
poolB/dataset  copies                1											default
poolB/dataset  version               5											-
poolB/dataset  utf8only              off											-
poolB/dataset  normalization         none										-
poolB/dataset  casesensitivity       mixed										-
poolB/dataset  vscan                 off											default
poolB/dataset  nbmand                off											default
poolB/dataset  sharesmb              off											default
poolB/dataset  refquota              none										default
poolB/dataset  refreservation        none										default
poolB/dataset  guid                  4761591032891098445							-
poolB/dataset  primarycache          none										local
poolB/dataset  secondarycache        all											received
poolB/dataset  usedbysnapshots       0B											-
poolB/dataset  usedbydataset         27.2T										-
poolB/dataset  usedbychildren        0B											-
poolB/dataset  usedbyrefreservation  0B											-
poolB/dataset  logbias               latency										received
poolB/dataset  dedup                 off											received
poolB/dataset  mlslabel              none										default
poolB/dataset  sync                  standard									default
poolB/dataset  dnodesize             auto										received
poolB/dataset  refcompressratio      1.06x										-
poolB/dataset  written               0											-
poolB/dataset  logicalused           28.4T										-
poolB/dataset  logicalreferenced     28.4T										-
poolB/dataset  volmode               default										default
poolB/dataset  filesystem_limit      none										default
poolB/dataset  snapshot_limit        none										default
poolB/dataset  filesystem_count      none										default
poolB/dataset  snapshot_count        none										default
poolB/dataset  snapdev               hidden										default
poolB/dataset  acltype               posixacl									received
poolB/dataset  context               none										default
poolB/dataset  fscontext             none										default
poolB/dataset  defcontext            none										default
poolB/dataset  rootcontext           none										default
poolB/dataset  relatime              off											received
poolB/dataset  redundant_metadata    all											default
poolB/dataset  overlay               off											default
poolB/dataset  receive_resume_token  1-1291bde31b-118-789c636064000310a501c49c50360710a715e5e7a69766a63040c11d65b9f2a5f65add0a40363b92bafca4acd4e412081f0430e4d3d28a534b18e00024cf86249f5459925acc802a8facbf241fe28a8b2aeb12d9e73c3ee38024cf0996cf4bcc4d656048ce292ec8cfcfd14fcc4b4cc9cf29cdcc4d4c4f2d76303435343336b1b034333437b10c720df0019bcfcd80f077727e6e41516a71717e36c20d00f495297d  -

[beren12]

Did you say on IRC you were using encryption on this pool?

[morphinz]

@beren12 No. I don't use encryption on both system.

[morphinz]

Hello again.
I upgraded A server with "0.7.6-1" and B server 0.7.6-1 & after that I try zfs-git Nothing changed.

I try to send with resume token and B server got panic again with same log..
After that I rollback the destination and I try to send an incremental with new snapshot on source.
B server got panic again with same log.

After that I run a scrub on destination:

pool: FKM1
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 0 days 19:01:45 with 0 errors on Mon Feb 26 14:22:26 2018

(The future are encryption which I don't use, nothing important)

As you can see the pool have 0 issue.
Now I'm gonna run a scrub on source but this will take 2 day atleast so I need to wait Friday.
After scrub I will try again but if there is any way to get more log about the problem can you tell me please?

See you soon dear friends.

[tcaputi]

@kpande

fwiw I'm hitting a similar problem on master with encrypted datasets but it deadlocks so hard the audio playing (when it happens) skips forever and I do not get any stack traces.

Usually crashing like that is caused when the code hits a call to panic(). If you are using a GUI or ssh you can probably get a stack trace from the raw tty. Let me know if you can get a stack trace from this.

@morphinz and @kpande
This looks like either a double free or a buffer overrun, which are usually pretty hard to debug without being able to reproduce the problem. Fortunately, the spl has 2 build options (--enable-debug-kmem-tracking and --enable-debug-kmem, which can be provided to ./configure) which can help debug double frees. If one occurs, it will ASSERT right at the problem and provide useful info for debugging. Could you guys give that option a try (with the latest code from master if possible) and see what happens? If it is a buffer overrun then I will need to see if I can reproduce it myself.

[tcaputi]

@kpande. With the configure options there is a good chance it might not hard panic like that and you won't need the cable. You can also get to a raw terminal (not blocked by X) with Ctrl + alt + f5 (or any other f-key).

[tcaputi]

@kpande Was this comment meant to be made here? If so, that was not the kind of double free I was talking about. That one is an on-disk spacemap double free, while this one might be a memory double free.

[morphinz]

Sorry for the late answer.

I ran Scrub on both pool "repaired 0B - with 0 errors"
After Scrub I try to send an incremental "I don't use resume token" and B server got panic again...
Nothing changed!

After that I tried few weird things;
Information:
1- The dataset named "xXx" 27T.
2- xXx has 1 snapshot on source and destination. This was the first Send-Revc snaphot.
3- If I take a new snapshot and try to send Incremental to B server, B server panics.

On A server "which is the source" I decide to try a FULL send-recv "In same pool".
zfs send -Rc aserver\xXx@FirstSnapshot | zfs recv aserver\xXx-new
Everything was Fine.
After that I tried to send Incremental, "aserver\xXx@SecondSnapshot | zfs recv aserver\xXx-new" A server got panic exactly like B server.
I tried to send Full "-Rc aserver\xXx@SecondSnapshot | zfs recv aserver\xXx-BRANDNEW" nothing changed, A server got panic when FirstSnapshot over and Second begin.

I'm confused. What is going on???

zfs rollback aserver\xXx-new@FirstSnapshot
dd if=/dev/zero of=/xXx-new/test count=100 bs=1G
zfs send -cI aserver\xXx-new@NewSnapshot | ssh Bserver zfs recv aserver\xXx

Gues what? Everything was JUST FINE... I didnt get panic.
This reminds me a simple question: What the heck is that?

IF I clone first snapshot and write some file on it I was able to RESUME.
BUT when I try to take a snapshot on active dataset I cant resume it.. How is that possible?
The change was only 970G on first snapshot to Second..

@kpande Yes I don't use encryption. I was trying zfs-git and the feature was encryption.

@tcaputi I have bigger dataset than the "xXx" but I only have the problem on 2 dataset. 1 of them 27t and the other was 80T.
But like I said I have different datasets too like "40T, 60T, 80T" and they don't have the problem.

[tcaputi]

@morphinz I'm a little confused at the moment. Does this issue only happen when you use a resume token? In your comment from yesterday you don't mention the resume token in your commands. When the kernel panics is the stack trace the same as the one you posted, or are you unable to get a stack trace at all?

@kpande any updates here?

[morphinz]

@tcaputi When I open the issue I was thinking this is all about with resume token but now %100 I'm sure its not. Also I don't use resume token anymore. I have the issue with or without resume token.

Also today I create a bookmark from that snapshot and I delete the snapshot because I was thinking the snapshot will cause the problem and deleting will be solve my problem but it did not.. I got same result and same stack trace.

As you can see at below my source is "Apool/xXx" and destination "Apool/xXx-testing" in same pool.
PS: "I changed pool and dataset name for easy understanding."

NAME                                              AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  USERREFS  CREATION
Apool/xXx                                        42.8T  28.2T     47.1M   28.2T          0B         0B         -  Fri Nov  3  0:48 2017
Apool/xXx#1516182586920REPL-bookmark               -      -         -       -             -          -         -  Wed Jan 17 12:49 2018
Apool/xXx@1520955317597REPL                        -  47.1M         -       -             -          -         1  Tue Mar 13 18:35 2018

NAME                                            AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  USERREFS  CREATION
Apool/xXx-testing                    43.8T  27.2T        0B   27.2T             0B         0B         -  Thu Mar  8 21:33 2018
Apool/xXx-testing@1516182586920REPL      -     0B         -       -              -          -         0  Wed Jan 17 12:49 2018

zfs send -i Apool/xXx#1516182586920REPL-bookmark Apool/xXx@NewlyCreatedSNAP| zfs receive -sv Apool/xXx

And I got panic.
If I got any free time I will compile zfs for debuging the issue and I will share my result.

[morphinz]

@tcaputi sorry for the outstanding delay. I was busy with other stuff. Also I thought I found a workaround for this problem and that bought me some time. However my workaround failed and this is a top priority for me again :(

Since it's been a while I've setup a fresh new platform in order to make the problem perfectly reproduciable. I have tried a couple of distributions, kernel and zfs versions. Finally I've compiled the latest spl bits with DEBUG parameters (--enable-debug-kmem-tracking and --enable-debug-kmem).

First let me share my various distro and version results.

Arch Linux

• Kernel: 4.14.29
• Zfs: 0.7.7
• Command: zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol
• Output: receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130
• Crash log:

Mar 24 21:09:20 FKM2 zed[19821]: eid=12 class=history_event pool_guid=0x2F6EE146278F0CC1
Mar 24 21:09:22 FKM2 kernel: general protection fault: 0000 [#1] SMP NOPTI
Mar 24 21:09:22 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi ttm drm_kms_helper ipmi_ssif drm 8021q evdev mrp joydev agpgart syscopyarea mousedev input_leds sysfillrect igb led_class mei_me sysimgblt mac_hid fb_sys_fops i2c_i801 mei lpc_ich i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter button acpi_pad bonding sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zavl(PO)
Mar 24 21:09:22 FKM2 kernel:  icp(PO) ses enclosure zcommon(PO) znvpair(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:09:22 FKM2 kernel: CPU: 7 PID: 19823 Comm: receive_writer Tainted: P           O    4.14.29-1-lts #1
Mar 24 21:09:22 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:09:22 FKM2 kernel: task: ffff9b1cb7256580 task.stack: ffffb2fa6a9ec000
Mar 24 21:09:22 FKM2 kernel: RIP: 0010:__kmalloc_node+0x15e/0x2a0
Mar 24 21:09:22 FKM2 kernel: RSP: 0018:ffffb2fa6a9efce0 EFLAGS: 00010282
Mar 24 21:09:22 FKM2 kernel: RAX: 0000000000000000 RBX: 92558eb84083930e RCX: 0000000000604cd4
Mar 24 21:09:22 FKM2 kernel: RDX: 0000000000604cd3 RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:09:22 FKM2 kernel: RBP: ffff9adc88b9dc00 R08: ffff9adc88b9dc00 R09: ffff9adcbf407000
Mar 24 21:09:22 FKM2 kernel: R10: 6daa1464c83a4f0e R11: ffffffffc02b4097 R12: 000000000140c200
Mar 24 21:09:22 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff9adcbf407000
Mar 24 21:09:22 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff9adcbfbc0000(0000) knlGS:0000000000000000
Mar 24 21:09:22 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:09:22 FKM2 kernel: CR2: 00007fd448a31eb0 CR3: 00000014a700a002 CR4: 00000000003606e0
Mar 24 21:09:22 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:09:22 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:09:22 FKM2 kernel: Call Trace:
Mar 24 21:09:22 FKM2 kernel:  ? spl_kmem_alloc+0xd0/0x180 [spl]
Mar 24 21:09:22 FKM2 kernel:  spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:09:22 FKM2 kernel:  dbuf_dirty+0x1ff/0x830 [zfs]
Mar 24 21:09:22 FKM2 kernel:  receive_object+0x248/0x460 [zfs]
Mar 24 21:09:22 FKM2 kernel:  ? dmu_free_long_range+0x35b/0x440 [zfs]
Mar 24 21:09:22 FKM2 kernel:  receive_writer_thread+0x580/0x9c0 [zfs]
Mar 24 21:09:22 FKM2 kernel:  ? set_curr_task_fair+0x29/0x60
Mar 24 21:09:22 FKM2 kernel:  ? set_user_nice.part.71+0x14e/0x240
Mar 24 21:09:22 FKM2 kernel:  ? receive_free.isra.13+0xb0/0xb0 [zfs]
Mar 24 21:09:22 FKM2 kernel:  ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:09:22 FKM2 kernel:  thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:09:22 FKM2 kernel:  kthread+0x11a/0x130
Mar 24 21:09:22 FKM2 kernel:  ? kthread_create_on_node+0x70/0x70
Mar 24 21:09:22 FKM2 kernel:  ret_from_fork+0x35/0x40
Mar 24 21:09:22 FKM2 kernel: Code: 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 ff fe ff ff 4d 39 d0 74 14 49 63 47 20 48 01 c3 <48> 33 1b 49 33 9f 40 01 00 00 0f 18 0b 41 f7 c4 00 80 00 00 0f
Mar 24 21:09:22 FKM2 kernel: RIP: __kmalloc_node+0x15e/0x2a0 RSP: ffffb2fa6a9efce0
Mar 24 21:09:22 FKM2 kernel: ---[ end trace 5ee89471431f04fb ]---
Mar 24 21:09:27 FKM2 kernel: general protection fault: 0000 [#2] SMP NOPTI
Mar 24 21:09:27 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi ttm drm_kms_helper ipmi_ssif drm 8021q evdev mrp joydev agpgart syscopyarea mousedev input_leds sysfillrect igb led_class mei_me sysimgblt mac_hid fb_sys_fops i2c_i801 mei lpc_ich i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter button acpi_pad bonding sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zavl(PO)
Mar 24 21:09:27 FKM2 kernel:  icp(PO) ses enclosure zcommon(PO) znvpair(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:09:27 FKM2 kernel: CPU: 7 PID: 3504 Comm: txg_sync Tainted: P      D    O    4.14.29-1-lts #1
Mar 24 21:09:27 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:09:27 FKM2 kernel: task: ffff9adcadb52b80 task.stack: ffffb2fa21520000
Mar 24 21:09:27 FKM2 kernel: RIP: 0010:__kmalloc_node+0x132/0x2a0
Mar 24 21:09:27 FKM2 kernel: RSP: 0018:ffffb2fa21523bf8 EFLAGS: 00010282
Mar 24 21:09:27 FKM2 kernel: RAX: 92558eb84083930e RBX: 000000000140c200 RCX: 0000000000604cd5
Mar 24 21:09:27 FKM2 kernel: RDX: 0000000000604cd4 RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:09:27 FKM2 kernel: RBP: 92558eb84083930e R08: 92558eb84083930e R09: ffff9adcbf407000
Mar 24 21:09:27 FKM2 kernel: R10: 0000000000001800 R11: ffffffffc02b4097 R12: 000000000140c200
Mar 24 21:09:27 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff9adcbf407000
Mar 24 21:09:27 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff9adcbfbc0000(0000) knlGS:0000000000000000
Mar 24 21:09:27 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:09:27 FKM2 kernel: CR2: 00007fd448a31eb0 CR3: 00000014a700a002 CR4: 00000000003606e0
Mar 24 21:09:27 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:09:27 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:09:27 FKM2 kernel: Call Trace:
Mar 24 21:09:27 FKM2 kernel:  ? update_load_avg+0x41d/0x580
Mar 24 21:09:27 FKM2 kernel:  spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:09:27 FKM2 kernel:  dbuf_dirty+0x1ff/0x830 [zfs]
Mar 24 21:09:27 FKM2 kernel:  dsl_dataset_sync+0x21/0x220 [zfs]
Mar 24 21:09:27 FKM2 kernel:  dsl_pool_sync+0xa1/0x420 [zfs]
Mar 24 21:09:27 FKM2 kernel:  spa_sync+0x424/0xcf0 [zfs]
Mar 24 21:09:27 FKM2 kernel:  txg_sync_thread+0x2cb/0x490 [zfs]
Mar 24 21:09:27 FKM2 kernel:  ? txg_delay+0x1b0/0x1b0 [zfs]
Mar 24 21:09:27 FKM2 kernel:  ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:09:27 FKM2 kernel:  thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:09:27 FKM2 kernel:  kthread+0x11a/0x130
Mar 24 21:09:27 FKM2 kernel:  ? kthread_create_on_node+0x70/0x70
Mar 24 21:09:27 FKM2 kernel:  ret_from_fork+0x35/0x40
Mar 24 21:09:27 FKM2 kernel: Code: 00 49 89 c7 4c 8b 5c 24 08 4c 8b 0c 24 4d 85 ff 0f 85 34 ff ff ff 31 ed eb 5c 4d 63 47 20 49 8b 3f 48 8d 4a 01 48 89 e8 49 01 e8 <4d> 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f
Mar 24 21:09:27 FKM2 kernel: RIP: __kmalloc_node+0x132/0x2a0 RSP: ffffb2fa21523bf8
Mar 24 21:09:27 FKM2 kernel: ---[ end trace 5ee89471431f04fc ]---

Arch Linux

• Kernel: 4.14.29
• Zfs: Git-r3383.387b6856d
• Command: zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol
• Output: receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130 **Segmentation fault**
• Crash log:

Mar 24 21:40:56 FKM2 zed[29809]: eid=12 class=history_event pool_guid=0x2F6EE146278F0CC1
Mar 24 21:40:57 FKM2 kernel: general protection fault: 0000 [#1] SMP NOPTI
Mar 24 21:40:57 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi 8021q mrp ipmi_ssif ttm drm_kms_helper drm evdev joydev input_leds agpgart syscopyarea mousedev mei_me igb sysfillrect led_class sysimgblt fb_sys_fops mac_hid mei lpc_ich i2c_i801 i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi bonding acpi_power_meter button acpi_pad sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zlua(PO)
Mar 24 21:40:57 FKM2 kernel:  ses enclosure zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:40:57 FKM2 kernel: CPU: 21 PID: 29811 Comm: receive_writer Tainted: P           O    4.14.29-1-lts #1
Mar 24 21:40:57 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:40:57 FKM2 kernel: task: ffff8b727dfd0e80 task.stack: ffffa9603e510000
Mar 24 21:40:57 FKM2 kernel: RIP: 0010:__kmalloc_node+0x15e/0x2a0
Mar 24 21:40:57 FKM2 kernel: RSP: 0018:ffffa9603e513c78 EFLAGS: 00010202
Mar 24 21:40:57 FKM2 kernel: RAX: 0000000000000000 RBX: 402d0c9ea1f45245 RCX: 00000000000038fd
Mar 24 21:40:57 FKM2 kernel: RDX: 00000000000038fc RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:40:57 FKM2 kernel: RBP: ffff8b7278567e00 R08: ffff8b7278567e00 R09: ffff8b72ff407000
Mar 24 21:40:57 FKM2 kernel: R10: bfd287ecd9a22c45 R11: ffffffffc03d8097 R12: 000000000140c200
Mar 24 21:40:57 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff8b72ff407000
Mar 24 21:40:57 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff8b72ffd40000(0000) knlGS:0000000000000000
Mar 24 21:40:57 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:40:57 FKM2 kernel: CR2: 00007f2c12bbe020 CR3: 0000005eac00a005 CR4: 00000000003606e0
Mar 24 21:40:57 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:40:57 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:40:57 FKM2 kernel: Call Trace:
Mar 24 21:40:57 FKM2 kernel:  ? spl_kmem_alloc+0xd0/0x180 [spl]
Mar 24 21:40:57 FKM2 kernel:  spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:40:57 FKM2 kernel:  dbuf_dirty+0x1ff/0x840 [zfs]
Mar 24 21:40:57 FKM2 kernel:  receive_object+0x576/0x920 [zfs]
Mar 24 21:40:57 FKM2 kernel:  ? dnode_rele_and_unlock+0x4e/0x80 [zfs]
Mar 24 21:40:57 FKM2 kernel:  receive_writer_thread+0x610/0xb60 [zfs]
Mar 24 21:40:57 FKM2 kernel:  ? set_curr_task_fair+0x29/0x60
Mar 24 21:40:57 FKM2 kernel:  ? receive_freeobjects.isra.14+0x110/0x110 [zfs]
Mar 24 21:40:57 FKM2 kernel:  ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:40:57 FKM2 kernel:  ? thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:40:57 FKM2 kernel:  ? receive_freeobjects.isra.14+0x110/0x110 [zfs]
Mar 24 21:40:57 FKM2 kernel:  thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:40:57 FKM2 kernel:  kthread+0x11a/0x130
Mar 24 21:40:57 FKM2 kernel:  ? kthread_create_on_node+0x70/0x70
Mar 24 21:40:57 FKM2 kernel:  ret_from_fork+0x35/0x40
Mar 24 21:40:57 FKM2 kernel: Code: 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 ff fe ff ff 4d 39 d0 74 14 49 63 47 20 48 01 c3 <48> 33 1b 49 33 9f 40 01 00 00 0f 18 0b 41 f7 c4 00 80 00 00 0f
Mar 24 21:40:57 FKM2 kernel: RIP: __kmalloc_node+0x15e/0x2a0 RSP: ffffa9603e513c78
Mar 24 21:40:57 FKM2 kernel: ---[ end trace a3b314cdabd6aff2 ]---
Mar 24 21:41:01 FKM2 kernel: general protection fault: 0000 [#2] SMP NOPTI
Mar 24 21:41:01 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi 8021q mrp ipmi_ssif ttm drm_kms_helper drm evdev joydev input_leds agpgart syscopyarea mousedev mei_me igb sysfillrect led_class sysimgblt fb_sys_fops mac_hid mei lpc_ich i2c_i801 i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi bonding acpi_power_meter button acpi_pad sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zlua(PO)
Mar 24 21:41:01 FKM2 kernel:  ses enclosure zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:41:01 FKM2 kernel: CPU: 21 PID: 29643 Comm: zfs Tainted: P      D    O    4.14.29-1-lts #1
Mar 24 21:41:01 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:41:01 FKM2 kernel: task: ffff8b72d7c2d700 task.stack: ffffa960311a0000
Mar 24 21:41:01 FKM2 kernel: RIP: 0010:__kmalloc_node+0x132/0x2a0
Mar 24 21:41:01 FKM2 kernel: RSP: 0018:ffffa960311a3140 EFLAGS: 00010202
Mar 24 21:41:01 FKM2 kernel: RAX: 402d0c9ea1f45245 RBX: ffff8bb2ec444b60 RCX: 00000000000038fe
Mar 24 21:41:01 FKM2 kernel: RDX: 00000000000038fd RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:41:01 FKM2 kernel: RBP: 402d0c9ea1f45245 R08: 402d0c9ea1f45245 R09: ffff8b72ff407000
Mar 24 21:41:01 FKM2 kernel: R10: ffffa960311a3188 R11: ffffffff8b3011ab R12: 0000000001011200
Mar 24 21:41:01 FKM2 kernel: R13: 00000000000001d0 R14: 00000000ffffffff R15: ffff8b72ff407000
Mar 24 21:41:01 FKM2 kernel: FS:  00007f89dd8dc7c0(0000) GS:ffff8b72ffd40000(0000) knlGS:0000000000000000
Mar 24 21:41:01 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:41:01 FKM2 kernel: CR2: 00007f2c12bbe020 CR3: 0000007f3fe56004 CR4: 00000000003606e0
Mar 24 21:41:01 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:41:01 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:41:01 FKM2 kernel: Call Trace:
Mar 24 21:41:01 FKM2 kernel:  ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel:  ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel:  alloc_request_size+0x2b/0x70
Mar 24 21:41:01 FKM2 kernel:  mempool_alloc+0x65/0x190
Mar 24 21:41:01 FKM2 kernel:  ? dm_any_congested+0x60/0x60 [dm_mod]
Mar 24 21:41:01 FKM2 kernel:  get_request+0x3b6/0x770
Mar 24 21:41:01 FKM2 kernel:  ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel:  blk_queue_bio+0x114/0x400
Mar 24 21:41:01 FKM2 kernel:  generic_make_request+0x123/0x300
Mar 24 21:41:01 FKM2 kernel:  ? submit_bio+0x6c/0x140
Mar 24 21:41:01 FKM2 kernel:  ? bio_alloc_bioset+0xc7/0x1e0
Mar 24 21:41:01 FKM2 kernel:  submit_bio+0x6c/0x140
Mar 24 21:41:01 FKM2 kernel:  vdev_disk_io_start+0x4f5/0x7e0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel:  vdev_raidz_io_start+0x145/0x2c0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? vdev_config_sync+0x190/0x190 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel:  vdev_mirror_io_start+0x92/0x180 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? spa_config_enter+0xb2/0x100 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zio_vdev_io_start+0x17c/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? tsd_get_by_thread+0x2a/0x40 [spl]
Mar 24 21:41:01 FKM2 kernel:  ? taskq_member+0x14/0x20 [spl]
Mar 24 21:41:01 FKM2 kernel:  zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel:  arc_read+0x8a2/0xf20 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? dbuf_rele_and_unlock+0x510/0x510 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? zio_null+0x2b/0x30 [zfs]
Mar 24 21:41:01 FKM2 kernel:  dbuf_read+0x26b/0xb10 [zfs]
Mar 24 21:41:01 FKM2 kernel:  dnode_hold_impl+0x140/0xc40 [zfs]
Mar 24 21:41:01 FKM2 kernel:  dmu_object_info+0x2d/0x80 [zfs]
Mar 24 21:41:01 FKM2 kernel:  dmu_recv_stream+0x8da/0xfb0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? nvlist_free+0x7b/0xb0 [znvpair]
Mar 24 21:41:01 FKM2 kernel:  zfs_ioc_recv_impl+0x576/0x1010 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? nvs_native_nvp_op+0x1d0/0x1d0 [znvpair]
Mar 24 21:41:01 FKM2 kernel:  zfs_ioc_recv+0x1d3/0x300 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? spa_name_compare+0xa/0x20 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? avl_find+0x58/0x90 [zavl]
Mar 24 21:41:01 FKM2 kernel:  ? spa_lookup+0xad/0xd0 [zfs]
Mar 24 21:41:01 FKM2 kernel:  ? cap_inode_getsecurity+0x230/0x230
Mar 24 21:41:01 FKM2 kernel:  ? security_capable+0x47/0x60
Mar 24 21:41:01 FKM2 kernel:  ? cap_inode_getsecurity+0x230/0x230
Mar 24 21:41:01 FKM2 kernel:  ? security_capable+0x47/0x60
Mar 24 21:41:01 FKM2 kernel:  ? __kmalloc+0x17e/0x200
Mar 24 21:41:01 FKM2 kernel:  ? priv_policy.isra.3.part.4+0xd/0x20 [zfs]
Mar 24 21:41:01 FKM2 kernel:  zfsdev_ioctl+0x1e1/0x650 [zfs]
Mar 24 21:41:01 FKM2 kernel:  do_vfs_ioctl+0xa4/0x630
Mar 24 21:41:01 FKM2 kernel:  ? handle_mm_fault+0xaa/0x1e0
Mar 24 21:41:01 FKM2 kernel:  SyS_ioctl+0x74/0x80
Mar 24 21:41:01 FKM2 kernel:  do_syscall_64+0x67/0x120
Mar 24 21:41:01 FKM2 kernel:  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Mar 24 21:41:01 FKM2 kernel: RIP: 0033:0x7f89dc960d87
Mar 24 21:41:01 FKM2 kernel: RSP: 002b:00007ffcbdc751e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Mar 24 21:41:01 FKM2 kernel: RAX: ffffffffffffffda RBX: 00005606678d02d0 RCX: 00007f89dc960d87
Mar 24 21:41:01 FKM2 kernel: RDX: 00007ffcbdc75390 RSI: 0000000000005a1b RDI: 0000000000000005
Mar 24 21:41:01 FKM2 kernel: RBP: 00007ffcbdc75390 R08: 0000000000000000 R09: 00007f89dcc25b20
Mar 24 21:41:01 FKM2 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffcbdc83c88
Mar 24 21:41:01 FKM2 kernel: R13: 00007ffcbdc7f790 R14: 00007ffcbdc7d090 R15: 0000000000000000
Mar 24 21:41:01 FKM2 kernel: Code: 00 49 89 c7 4c 8b 5c 24 08 4c 8b 0c 24 4d 85 ff 0f 85 34 ff ff ff 31 ed eb 5c 4d 63 47 20 49 8b 3f 48 8d 4a 01 48 89 e8 49 01 e8 <4d> 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f
Mar 24 21:41:01 FKM2 kernel: RIP: __kmalloc_node+0x132/0x2a0 RSP: ffffa960311a3140
Mar 24 21:41:01 FKM2 kernel: ---[ end trace a3b314cdabd6aff3 ]---

Centos 7

• Kernel: 3.10.0-693.21.1.el7.x86_64
• Zfs: 0.7.7
• Command:zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol
• Output: receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130
• Crash log: Unfournately there is no crash log but there is a console screen capture.

Centos 7

• Kernel: 4.16.0-1.el7.elrepo.x86_64
• Zfs: 0.7.7
• Command:zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol
• Output: receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130
• Crash log:

Nis 04 13:43:47 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 04 13:43:47 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) fuse btrfs xor zstd_decompress zstd_compress xxhash raid6_pq ufs hfsplus hfs vfat msdos fat ext4 mbcache jbd2 dm_service_time ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack bonding ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
Nis 04 13:43:47 FKM2 kernel:  pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds ses enclosure sg i2c_i801 lpc_ich mei_me mei mfd_core ioatdma wmi shpchp ipmi_si ipmi_devintf acpi_power_meter ipmi_msghandler acpi_pad dm_multipath ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mpt3sas mdio ptp raid_class i2c_algo_bit pps_core dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 04 13:43:48 FKM2 kernel: CPU: 4 PID: 10367 Comm: receive_writer Tainted: P           O     4.16.0-1.el7.elrepo.x86_64 #1
Nis 04 13:43:48 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 04 13:43:48 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 04 13:43:48 FKM2 kernel: RSP: 0018:ffffc9000e0b7c30 EFLAGS: 00010286
Nis 04 13:43:48 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 0000000000014e19
Nis 04 13:43:48 FKM2 kernel: RDX: 0000000000014e18 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 04 13:43:48 FKM2 kernel: RBP: ffffc9000e0b7c30 R08: ffff883f7fb25f60 R09: ffff883f6e2d0200
Nis 04 13:43:48 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa07ff1bb R12: 000000000140c200
Nis 04 13:43:48 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 04 13:43:48 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff883f7fb00000(0000) knlGS:0000000000000000
Nis 04 13:43:48 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 04 13:43:48 FKM2 kernel: CR2: 00000000025eaf8c CR3: 000000000220a002 CR4: 00000000003606e0
Nis 04 13:43:48 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nis 04 13:43:48 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nis 04 13:43:48 FKM2 kernel: Call Trace:
Nis 04 13:43:48 FKM2 kernel:  __kmalloc_node+0x1d0/0x280
Nis 04 13:43:48 FKM2 kernel:  spl_kmem_zalloc+0xbb/0x160 [spl]
Nis 04 13:43:48 FKM2 kernel:  dbuf_dirty+0x1ea/0x860 [zfs]
Nis 04 13:43:48 FKM2 kernel:  ? dnode_rele_and_unlock+0x53/0x90 [zfs]
Nis 04 13:43:48 FKM2 kernel:  dmu_buf_will_dirty+0x11c/0x130 [zfs]
Nis 04 13:43:48 FKM2 kernel:  receive_object+0x379/0x430 [zfs]
Nis 04 13:43:48 FKM2 kernel:  receive_writer_thread+0x289/0xa30 [zfs]
Nis 04 13:43:48 FKM2 kernel:  ? set_next_entity+0x93/0xe0
Nis 04 13:43:48 FKM2 kernel:  ? do_dump+0x980/0x980 [zfs]
Nis 04 13:43:48 FKM2 kernel:  ? spl_kmem_free+0x2a/0x40 [spl]
Nis 04 13:43:48 FKM2 kernel:  ? do_dump+0x980/0x980 [zfs]
Nis 04 13:43:48 FKM2 kernel:  thread_generic_wrapper+0x74/0x90 [spl]
Nis 04 13:43:48 FKM2 kernel:  kthread+0x105/0x140
Nis 04 13:43:48 FKM2 kernel:  ? __thread_exit+0x20/0x20 [spl]
Nis 04 13:43:48 FKM2 kernel:  ? kthread_bind+0x20/0x20
Nis 04 13:43:48 FKM2 kernel:  ret_from_fork+0x35/0x40
Nis 04 13:43:48 FKM2 kernel: Code: 31 d2 e8 a3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 48
Nis 04 13:43:48 FKM2 kernel: RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: ffffc9000e0b7c30
Nis 04 13:43:48 FKM2 kernel: ---[ end trace d9616e2c3c7ce6f2 ]---

And finally here is the results of the latest bits of spl/zfs with SPL DEBUG flags.

Centos 7 - SPL DEBUG

• Kernel: 4.16.0-1.el7.elrepo.x86_64
• Zfs: v0.7.0-405_g533ea04 (latest bits as of Apr 5)
• Command: zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol
• Output: receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130
• Crash log:

Nis 06 02:29:34 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 02:29:34 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) ip6t_rpfilter bonding ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack dm_service_time ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 02:29:34 FKM2 kernel:  mei_me lpc_ich i2c_i801 mei ses mfd_core enclosure sg ioatdma ipmi_si ipmi_devintf wmi acpi_power_meter ipmi_msghandler shpchp acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe igb libata mdio mpt3sas ptp raid_class pps_core i2c_algo_bit dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 02:29:34 FKM2 kernel: CPU: 11 PID: 24929 Comm: receive_writer Tainted: P           O     4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 02:29:34 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 02:29:34 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 02:29:34 FKM2 kernel: RSP: 0018:ffffc9000f1b7be8 EFLAGS: 00010286
Nis 06 02:29:34 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 0000000000006ec7
Nis 06 02:29:34 FKM2 kernel: RDX: 0000000000006ec6 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 02:29:34 FKM2 kernel: RBP: ffffc9000f1b7be8 R08: ffff887f7f2e5f60 R09: ffff887eaf5c0400
Nis 06 02:29:34 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa068570f R12: 000000000140c200
Nis 06 02:29:34 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 06 02:29:34 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff887f7f2c0000(0000) knlGS:0000000000000000
Nis 06 02:29:34 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 06 02:29:34 FKM2 kernel: CR2: 00007f3520024048 CR3: 000000000220a004 CR4: 00000000003606e0
Nis 06 02:29:34 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nis 06 02:29:34 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nis 06 02:29:34 FKM2 kernel: Call Trace:
Nis 06 02:29:34 FKM2 kernel:  __kmalloc_node+0x1d0/0x280
Nis 06 02:29:34 FKM2 kernel:  spl_kmem_zalloc+0x22f/0x320 [spl]
Nis 06 02:29:34 FKM2 kernel:  dbuf_dirty+0x1ea/0x860 [zfs]
Nis 06 02:29:34 FKM2 kernel:  ? dnode_rele_and_unlock+0x53/0x90 [zfs]
Nis 06 02:29:34 FKM2 kernel:  dmu_buf_will_dirty_impl+0x11d/0x130 [zfs]
Nis 06 02:29:34 FKM2 kernel:  dmu_buf_will_dirty+0x16/0x20 [zfs]
Nis 06 02:29:34 FKM2 kernel:  receive_object+0x4f9/0x900 [zfs]
Nis 06 02:29:34 FKM2 kernel:  ? kfree+0x125/0x170
Nis 06 02:29:34 FKM2 kernel:  ? _cond_resched+0x19/0x30
Nis 06 02:29:34 FKM2 kernel:  ? mutex_lock+0x12/0x40
Nis 06 02:29:34 FKM2 kernel:  receive_writer_thread+0x40d/0xb60 [zfs]
Nis 06 02:29:34 FKM2 kernel:  ? set_next_entity+0x93/0xe0
Nis 06 02:29:34 FKM2 kernel:  ? do_dump+0xd70/0xd70 [zfs]
Nis 06 02:29:34 FKM2 kernel:  thread_generic_wrapper+0x74/0x90 [spl]
Nis 06 02:29:34 FKM2 kernel:  kthread+0x105/0x140
Nis 06 02:29:34 FKM2 kernel:  ? __thread_exit+0x20/0x20 [spl]
Nis 06 02:29:34 FKM2 kernel:  ? kthread_bind+0x20/0x20
Nis 06 02:29:34 FKM2 kernel:  ret_from_fork+0x35/0x40
Nis 06 02:29:34 FKM2 kernel: Code: 31 d2 e8 a3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 48
Nis 06 02:29:34 FKM2 kernel: RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: ffffc9000f1b7be8
Nis 06 02:29:34 FKM2 kernel: ---[ end trace f99762de42dd24be ]---

As supplementary information here is my pool: (encryption & project_quota features are not enabled)

zpool status
pool: fkmmedium
     id: 3417916858648431809
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
    some features will not be available without an explicit 'zpool upgrade'.
    config:

    NAME                    STATE     READ WRITE CKSUM
    fkmmedium               ONLINE       0     0     0
      raidz2-0              ONLINE       0     0     0
        35000c500844bbd4b1  ONLINE       0     0     0
        35000c50095f3fa87   ONLINE       0     0     0
        35000c500844b8017   ONLINE       0     0     0
        35000c5009604e493   ONLINE       0     0     0
        35000c500844b9e1f1  ONLINE       0     0     0
        35000c50095f9e53f1  ONLINE       0     0     0
        35000c50095ff4abf1  ONLINE       0     0     0
      raidz2-1              ONLINE       0     0     0
        35000c50095bf136b1  ONLINE       0     0     0
        35000c5009604c2ab1  ONLINE       0     0     0
        35000c500844bd82f1  ONLINE       0     0     0
        35000c50095f852b7   ONLINE       0     0     0
        35000c50095f4e74f1  ONLINE       0     0     0
        35000c5009604c7af1  ONLINE       0     0     0
        35000c50095fe3b53   ONLINE       0     0     0
      raidz2-2              ONLINE       0     0     0
        35000c50095f38377   ONLINE       0     0     0
        35000c50096071d7f1  ONLINE       0     0     0
        35000c500960f1e23   ONLINE       0     0     0
        35000c500844c14bf1  ONLINE       0     0     0
        35000c5009609d11f1  ONLINE       0     0     0
        35000c500960c0f4b1  ONLINE       0     0     0
        35000c500844bae13   ONLINE       0     0     0
      raidz2-3              ONLINE       0     0     0
        35000c500960f3507   ONLINE       0     0     0
        35000c50095f5e8c7   ONLINE       0     0     0
        35000c50095f60dd3   ONLINE       0     0     0
        35000c500844a7353   ONLINE       0     0     0
        35000c50095f7deb3   ONLINE       0     0     0
        35000c500960c67e7   ONLINE       0     0     0
        35000c500960c4a3f1  ONLINE       0     0     0
      raidz2-4              ONLINE       0     0     0
        35000c5009603592b1  ONLINE       0     0     0
        35000c500844ad437   ONLINE       0     0     0
        35000c50095f33f07   ONLINE       0     0     0
        35000c50096034fcb1  ONLINE       0     0     0
        35000c500960925ff1  ONLINE       0     0     0
        35000c500844be673   ONLINE       0     0     0
        35000c50096125517   ONLINE       0     0     0
      raidz2-5              ONLINE       0     0     0
        35000c50095f92327   ONLINE       0     0     0
        35000c500844bca53   ONLINE       0     0     0
        35000c50095f4b277   ONLINE       0     0     0
        35000c500844bd9d3   ONLINE       0     0     0
        35000c50095f7df27   ONLINE       0     0     0
        35000c500844bd4bb1  ONLINE       0     0     0
        35000c5009604ee5f1  ONLINE       0     0     0
      raidz2-6              ONLINE       0     0     0
        35000c50095ddc787   ONLINE       0     0     0
        35000c50095ff45df1  ONLINE       0     0     0
        35000c50096071ebb1  ONLINE       0     0     0
        35000c5009609dc8b1  ONLINE       0     0     0
        35000c50095f3e27b1  ONLINE       0     0     0
        35000c500844bc42f1  ONLINE       0     0     0
        35000c50095fa4327   ONLINE       0     0     0
      raidz2-7              ONLINE       0     0     0
        35000c500960eb903   ONLINE       0     0     0
        35000c500844b114b1  ONLINE       0     0     0
        35000c50095e30d63   ONLINE       0     0     0
        35000c500960e81e3   ONLINE       0     0     0
        35000c500844ba6ff1  ONLINE       0     0     0
        35000c500844ad27f1  ONLINE       0     0     0
        35000c50095f5afef1  ONLINE       0     0     0
      raidz2-8              ONLINE       0     0     0
        35000c50095f9b4f7   ONLINE       0     0     0
        35000c500844b127f1  ONLINE       0     0     0
        35000c50095f7d71f1  ONLINE       0     0     0
        35000c50095e27c17   ONLINE       0     0     0
        35000c500844bdc1f1  ONLINE       0     0     0
        35000c500844b5fab1  ONLINE       0     0     0
        35000c50095e1a507   ONLINE       0     0     0
      raidz2-9              ONLINE       0     0     0
        35000c500844bef67   ONLINE       0     0     0
        35000c50095ede39f1  ONLINE       0     0     0
        35000c50095f843af1  ONLINE       0     0     0
        35000c500844bc45f1  ONLINE       0     0     0
        35000c50095f4b45b1  ONLINE       0     0     0
        35000c500844b81b7   ONLINE       0     0     0
        35000c500844bc053   ONLINE       0     0     0
      raidz2-10             ONLINE       0     0     0
        35000c50095f92a8f1  ONLINE       0     0     0
        35000c50095f8ba23   ONLINE       0     0     0
        35000c500844be667   ONLINE       0     0     0
        35000c5009606ed9b1  ONLINE       0     0     0
        35000c500844bf083   ONLINE       0     0     0
        35000c50095f56c9f1  ONLINE       0     0     0
        35000c500844ad1bb1  ONLINE       0     0     0
      raidz2-11             ONLINE       0     0     0
        35000c50095e1a287   ONLINE       0     0     0
        35000c50095f8f597   ONLINE       0     0     0
        35000c500960f3df3   ONLINE       0     0     0
        35000c50095fb12b7   ONLINE       0     0     0
        35000c50095febecf1  ONLINE       0     0     0
        35000c500844b839f1  ONLINE       0     0     0
        35000c5009609287f1  ONLINE       0     0     0
    logs
      35000c5003023fe73     ONLINE       0     0     0
      35000c5003022e38f1    ONLINE       0     0     0
    cache
      35000c5003023feeb     ONLINE       0     0     0
      35000c5003023feef1    ONLINE       0     0     0
      35000c5003022e327     ONLINE       0     0     0
      35000c5003022e2f3     ONLINE       0     0     0
      35000c5003022e2c3     ONLINE       0     0     0
    spares
      35000c50095f7b9c3     AVAIL
      35000c50086636db3     AVAIL

errors: No known data errors

Here is my dataset:

[root@FKM2 ~]# zfs get all fkmmedium/images
NAME                            PROPERTY              VALUE                            SOURCE
fkmmedium/images  type                  filesystem                       -
fkmmedium/images  creation              Çrş Mar  7 18:55 2018          -
fkmmedium/images  used                  27,0T                            -
fkmmedium/images  available             44,0T                            -
fkmmedium/images  referenced            27,0T                            -
fkmmedium/images  compressratio         1.06x                            -
fkmmedium/images  mounted               yes                              -
fkmmedium/images  quota                 71T                              received
fkmmedium/images  reservation           none                             received
fkmmedium/images  recordsize            128K                             received
fkmmedium/images  mountpoint            /fkmmedium/images  local
fkmmedium/images  sharenfs              off                              default
fkmmedium/images  checksum              on                               default
fkmmedium/images  compression           lz4                              received
fkmmedium/images  atime                 off                              received
fkmmedium/images  devices               on                               default
fkmmedium/images  exec                  on                               default
fkmmedium/images  setuid                on                               default
fkmmedium/images  readonly              off                              received
fkmmedium/images  zoned                 off                              default
fkmmedium/images  snapdir               hidden                           default
fkmmedium/images  aclinherit            passthrough                      received
fkmmedium/images  createtxg             3253379                          -
fkmmedium/images  canmount              on                               default
fkmmedium/images  xattr                 sa                               received
fkmmedium/images  copies                1                                default
fkmmedium/images  version               5                                -
fkmmedium/images  utf8only              off                              -
fkmmedium/images  normalization         none                             -
fkmmedium/images  casesensitivity       mixed                            -
fkmmedium/images  vscan                 off                              default
fkmmedium/images  nbmand                off                              default
fkmmedium/images  sharesmb              off                              default
fkmmedium/images  refquota              none                             default
fkmmedium/images  refreservation        none                             default
fkmmedium/images  guid                  8661600337962025303              -
fkmmedium/images  primarycache          all                              received
fkmmedium/images  secondarycache        all                              received
fkmmedium/images  usedbysnapshots       0B                               -
fkmmedium/images  usedbydataset         27,0T                            -
fkmmedium/images  usedbychildren        0B                               -
fkmmedium/images  usedbyrefreservation  0B                               -
fkmmedium/images  logbias               latency                          received
fkmmedium/images  dedup                 off                              received
fkmmedium/images  mlslabel              none                             default
fkmmedium/images  sync                  standard                         default
fkmmedium/images  dnodesize             legacy                           local
fkmmedium/images  refcompressratio      1.06x                            -
fkmmedium/images  written               0                                -
fkmmedium/images  logicalused           28,4T                            -
fkmmedium/images  logicalreferenced     28,4T                            -
fkmmedium/images  volmode               default                          default
fkmmedium/images  filesystem_limit      none                             default
fkmmedium/images  snapshot_limit        none                             default
fkmmedium/images  filesystem_count      none                             default
fkmmedium/images  snapshot_count        none                             default
fkmmedium/images  snapdev               hidden                           default
fkmmedium/images  acltype               posixacl                         received
fkmmedium/images  context               none                             default
fkmmedium/images  fscontext             none                             default
fkmmedium/images  defcontext            none                             default
fkmmedium/images  rootcontext           none                             default
fkmmedium/images  relatime              off                              received
fkmmedium/images  redundant_metadata    all                              default
fkmmedium/images  overlay               off                              default
fkmmedium/images  encryption            off                              default
fkmmedium/images  keylocation           none                             default
fkmmedium/images  keyformat             none                             default
fkmmedium/images  pbkdf2iters           0                                default

Here is more info about the dataset
zfs list -t all -o name,used,available,usedsnap,usedds,usedrefreserv,usedbychildren,userrefs,creation -s creation -r fkmmedium/images

NAME                                               USED  AVAIL  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD  USERREFS  CREATION
fkmmedium/images@1516182586920REPL     0B      -         -       -              -          -         0  Çrş Oca 17 12:49 2018
fkmmedium/images                    27,0T  44,0T        0B   27,0T             0B         0B         -  Çrş Mar  7 18:55 2018

zfs get receive_resume_token fkmmedium/images

Resume Token
NAME                            PROPERTY              VALUE      SOURCE
fkmmedium/images  receive_resume_token  -          -

Finally I have a couple of things with DEBUG enabled latest bits:

1. Enabled encryption & project_quota features
   Crash log:

Same as features disabled.

2. Disabled prefetch:
   Crash log:

Nis 06 03:12:52 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 03:12:52 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dm_service_time ip6t_rpfilter bonding ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 03:12:52 FKM2 kernel:  lpc_ich i2c_i801 mfd_core mei_me mei ioatdma ses enclosure sg shpchp ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter wmi acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mdio mpt3sas ptp raid_class pps_core i2c_algo_bit dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 03:12:52 FKM2 kernel: CPU: 13 PID: 2517 Comm: receive_writer Tainted: P           O     4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 03:12:52 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 03:12:52 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 03:12:52 FKM2 kernel: RSP: 0018:ffffc9002e84fbe8 EFLAGS: 00010286
Nis 06 03:12:52 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 000000000000edb4
Nis 06 03:12:52 FKM2 kernel: RDX: 000000000000edb3 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 03:12:52 FKM2 kernel: RBP: ffffc9002e84fbe8 R08: ffff887f7f365f60 R09: ffff887f61ceb400
Nis 06 03:12:52 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa069a70f R12: 000000000140c200
Nis 06 03:12:52 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 06 03:12:52 FKM2 kernel: FS:  0000000000000000(0000) GS:ffff887f7f340000(0000) knlGS:0000000000000000
Nis 06 03:12:52 FKM2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 06 03:12:52 FKM2 kernel: CR2: 00007f38b4514000 CR3: 000000000220a003 CR4: 00000000003606e0
Nis 06 03:12:52 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000

3. Disabled prefetch and removed all logs & l2arc disks from pool.
   Crash log:

Nis 06 03:41:36 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 03:41:36 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dm_service_time bonding ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 03:41:36 FKM2 kernel:  i2c_i801 ses enclosure mei_me sg lpc_ich mfd_core mei ioatdma ipmi_si ipmi_devintf acpi_power_meter ipmi_msghandler shpchp wmi acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mdio mpt3sas ptp i2c_algo_bit pps_core raid_class dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 03:41:36 FKM2 kernel: CPU: 27 PID: 22463 Comm: receive_writer Tainted: P           O     4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 03:41:36 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 03:41:36 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 03:41:36 FKM2 kernel: RSP: 0018:ffffc9003411bbe8 EFLAGS: 00010286
Nis 06 03:41:36 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 00000000000272aa
Nis 06 03:41:36 FKM2 kernel: RDX: 00000000000272a9 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 03:41:36 FKM2 kernel: RBP: ffffc9003411bbe8 R08: ffff887f7f4e5f60 R09: ffff887f5fbeec00

This issue is perfectly reproduciable. Please let me know if I can provide more information. Also as this is an isoloted test environment I can provide private access if needed.

[tcaputi]

@morphinz thank you very much for the detailed report. If access to the machine is available, that would probably be the most convenient thing for me to work with, since this bug is probably related to something about the send files you already have. You can email credentials to me at tcaputi@datto.com and I will look into it today. Otherwise I can reproduce this myself tomorrow.