OpenZFS 2.4.1rc4 Sequioa Intel Core i7 15.7.4 Send/Receive between pools panics

[nexusgoblin]

Sending between to local pools panics hard (2 out of 2 attempts):

mp_kdp_enter() timed-out on cpu 0, NMI-ing
mp_kdp_enter() NMI pending on cpus: 0 4
mp_kdp_enter() timed-out during locked wait after NMI;expected 12 acks but received 11 after 5732267 loops in 1596000000 ticks
panic(cpu 0 caller 0xffffff7faecd0236): Tub: blkptr at 0xffffffe16aae7a40x has no valid DVAs
Panicked task 0xffffff90523c2980: 999 threads: pid 0: kernel_task
Backtrace (CPU 0), panicked thread: 0xffffff90515bd598, Frame : Return Address
0xffffffdcc0b2b180 : 0xffffff80185d6761 
0xffffffdcc0b2b1d0 : 0xffffff801874c846 
0xffffffdcc0b2b210 : 0xffffff801873b99c 
0xffffffdcc0b2b2d0 : 0xffffff801856b971 
0xffffffdcc0b2b2f0 : 0xffffff80185d6a57 
0xffffffdcc0b2b3f0 : 0xffffff80185d60db 
0xffffffdcc0b2b460 : 0xffffff8018dd109d 
0xffffffdcc0b2b550 : 0xffffff7faecd0236 
0xffffffdcc0b2b740 : 0xffffff7faecd0165 
0xffffffdcc0b2b880 : 0xffffff7faedaa582 
0xffffffdcc0b2b960 : 0xffffff7faee20a1f 
0xffffffdcc0b2bb70 : 0xffffff7faee203f1 
0xffffffdcc0b2bbd0 : 0xffffff7faee20d35 
0xffffffdcc0b2bc00 : 0xffffff7faed49545 
0xffffffdcc0b2bc80 : 0xffffff7faed3e5a2 
0xffffffdcc0b2bce0 : 0xffffff7faed3e7a9 
0xffffffdcc0b2bd60 : 0xffffff7faed854f2 
0xffffffdcc0b2bdc0 : 0xffffff7faed3e039 
0xffffffdcc0b2be70 : 0xffffff7faed23d99 
0xffffffdcc0b2bed0 : 0xffffff7faece0fdc 
0xffffffdcc0b2bf60 : 0xffffff7faece1572 
0xffffffdcc0b2bfa0 : 0xffffff801856b19e 
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.4.1)[60674E7F-5CE9-3966-96F0-68804BE9CC9A]@0xffffff7faeccf000->0xffffff7faf039fff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[0F68373A-6845-334F-A3F9-1547BE96F804]@0xffffff801abc4000->0xffffff801abdafff

Process name corresponding to current thread (0xffffff90515bd598): kernel_task

Mac OS version:
24G517

Kernel version:
Darwin Kernel Version 24.6.0: Mon Jan 19 22:00:10 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_X86_64
Kernel UUID: A8F2DD73-2B65-3BF9-9B5A-BA2B753AA960
roots installed: 0
KernelCache slide: 0x0000000018200000
KernelCache base:  0xffffff8018400000
Kernel slide:      0x00000000182e8000
Kernel text base:  0xffffff80184e8000
__HIB  text base: 0xffffff8018300000
System model name: Macmini8,1 (Mac-7BA5B2DFE22DDD8C)
System shutdown begun: NO
Hibernation exit count: 0

System uptime in nanoseconds: 900002039469
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x000000d18c4d4c3f
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x00000018b5fab9f1 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff805038f000 - 0xffffffa05038f000
  . PGZ   : 0xffffff805038f000 - 0xffffff8054390000
  . VM    : 0xffffff8054390000 - 0xffffff85206c2000
  . RO    : 0xffffff85206c2000 - 0xffffff86b9d32000
  . GEN0  : 0xffffff86b9d32000 - 0xffffff8b86064000
  . GEN1  : 0xffffff8b86064000 - 0xffffff9052396000
  . GEN2  : 0xffffff9052396000 - 0xffffff951e6c8000
  . GEN3  : 0xffffff951e6c8000 - 0xffffff99ea9fb000
  . DATA  : 0xffffff99ea9fb000 - 0xffffffa05038f000
  Metadata: 0xffffffe0bfc21000 - 0xffffffe0dfc21000
  Bitmaps : 0xffffffe0dfc21000 - 0xffffffe0e5c21000
  Extra   : 0 - 0

last started kext at 10780323790: >!APlatformEnabler	2.7.0d0 (addr 0xffffff7fad461000, size 8192)
loaded kexts:
org.openzfsonosx.zfs	2.4.1
>!APlatformEnabler	2.7.0d0
>AGPM	137
>X86PlatformShim	1.0.0
@filesystems.autofs	3.0
>!AUpstreamUserClient	3.6.12
>!AMCCSControl	1.17
>!AGFXHDA	400.1
>!AGraphicsDevicePolicy	8.1.10
>!ABridgeAudio!C	530.2
@AGDCPluginDisplayMetrics	8.1.10
>!A!IKBLGraphics	23.0.7
>pmtelemetry	1
@filesystems.nfs	1
|IOUserEthernet	1.0.1
>usb.!UUserHCI	1
>!ARXE	1.0
>!AHV	1
>!AFIVRDriver	4.1.0
>BridgeAudioCommunication	530.2
>!AAVEBridge	6.1
>ACPI_SMC_PlatformPlugin	1.0.0
>!A!ICFLGraphicsFramebuffer	23.0.7
>!A!IPCHPMC	2.0.1
>!A!ISlowAdaptiveClocking	4.0.0
>!AThunderboltNHISpeedTest	0.0.1
>!AThunderboltIP	4.0.3
@!AEthernetAquantiaAqtionFirmware	1.0.36
|SCSITaskUserClient	500.120.2
>!UMass!SInterfaceNub	556
>BCMWLANFirmware4388.Hashstore	1
>BCMWLANFirmware4387.Hashstore	1
>BCMWLANFirmware4378.Hashstore	1
>BCMWLANFirmware4377.Hashstore	1
>BCMWLANFirmware4364.Hashstore	1
>!AFileSystemDriver	3.0.1
@filesystems.tmpfs	1
@filesystems.lifs	1
@filesystems.hfs.kext	683.120.3
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
>!AAHCIPort	384
@filesystems.apfs	2332.140.13
>!ABCMWLANBusInterfacePCIeMac	1
@private.KextAudit	1.0
>!AACPIButtons	6.1
>!ASMBIOS	2.1
>!AACPIEC	6.1
>!AAPIC	1.7
@!ASystemPolicy	2.0.0
@nke.applicationfirewall	406
|IOKitRegistryCompatibility	1
|EndpointSecurity	1
@Dont_Steal_Mac_OS_X	7.0.0
>!ADiskImages2	385.140.4
@kec.!AEncryptedArchive	1
>!UAudio	750.4
>!AAudioClockLibs	420.3
>!ASMBus!C	1.0.18d1
>usb.IOUSBHostHIDDevice	1.2
>!AGraphicsControl	8.1.10
>!AHDA!C	600.2
|IOHDA!F	600.2
|IOAudio!F	600.2
@vecLib.kext	1.2.0
>!A!ILpssUARTv1	3.0.60
>!A!ILpssUARTCommon	3.0.60
>!AOnboardSerial	1.0
>!ASMBusPCI	1.0.14d1
@kext.triggers	1.0
>IOHIDPowerSource	1
>!ACallbackPowerSource	1
|IOAVB!F	1320.3
>!AUIO	1
|IORDMA!F	1.0
>!ARSMChannel	1
|IORSM!F	1
>!AIPAppender	1.0
|IONDRVSupport	599
>driverkit.serial	6.0.0
>IOPlatformPluginLegacy	1.0.0
|IOAccelerator!F2	485
@!AGPUWrangler	8.1.10
@!AGraphicsDeviceControl	8.1.10
|IOGraphics!F	599
@plugin.IOgPTPPlugin	1340.12
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
|IOSlowAdaptiveClocking!F	1.0.0
>!AThunderboltEDMSink	5.0.3
|IOAHCIBlock!S	362.100.1
>!AThunderboltDPOutAdapter	8.5.1
>usb.cdc.ecm	5.0.0
>usb.cdc.ncm	5.0.0
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!AThunderboltDPInAdapter	8.5.1
>!AThunderboltDPAdapter!F	8.5.1
>!AThunderboltPCIUpAdapter	4.1.1
>!AThunderboltPCIDownAdapter	4.1.1
>!AHPM	3.4.4
>!A!ILpssI2C!C	3.0.60
>!A!ILpssI2C	3.0.60
>!A!ILpssDmac	3.0.60
>!ABSDKextStarter	3
|IOSurface	372.5.2
@filesystems.hfs.encodings.kext	1
>!ASyntheticGame!C	12.6.1
|IOAHCI!F	308
>!AXsanScheme	3
>!AThunderboltNHI	7.2.81
|IOThunderbolt!F	9.3.3
>usb.!UVHCIBCE	1.2
>usb.!UVHCICommonBCE	1.0
>usb.!UVHCI	1.2
>usb.!UVHCICommon	1.0
>!AEffaceableNOR	1.0
|IONVMe!F	2.1.0
|IOBufferCopy!C	1.1.0
|IOBufferCopyEngine!F	1
>!AEthernetAquantiaAqtion	1.0.64
>!ABCMWLANCoreMac	1.0.0
|IO80211!F	1200.13.1
>IOImageLoader	1.0.0
>!AOLYHALMac	1
|IOSerial!F	11
>corecapture	1.0.4
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
>!AEFINVRAM	2.1
>!ASMCRTC	1.0
>!AEFIRuntime	2.1
|IOSMBus!F	1.1
|IOHID!F	2.0.0
|IOTimeSync!F	1340.12
|IOSkywalk!F	1.0
>mDNSOffloadUserClient	1.0.1b8
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	9.0.0
|IOReport!F	47
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
>!ASSE	1.0
>!ALockdownMode	1
>!AKeyStore	2
>!UTDM	556
|IOUSBMass!SDriver	259.100.1
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
|IOSCSIBlockCommandsDevice	500.120.2
|IOSCSIArchitectureModel!F	500.120.2
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
|IO!S!F	2.1
>!ACyrus	1
>!AMobileFileIntegrity	1.0.5
$!AImage4	7.0.0
@kext.CoreTrust	1
>!ACredentialManager	1.0
|CoreAnalytics!F	1
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
>!AACPIPlatform	6.1
>!ASMC	3.1.9
|IOPCI!F	2.9
|IOACPI!F	1.4
>watchdog	1
@kec.pthread	1
@kec.XrtHostedXnu	1
@kec.Libm	1
@kec.Compression	1.0
@kec.corecrypto	14.0

[nexusgoblin]

Pool has nested datasets if that is relevant. I have not "upgraded" the pools with new 2.4.1 features (still on 2.3.1).

[nexusgoblin]

Eeek! I downgraded to 2.3.1 and still got a panic. Maybe this isn't related to my upgrade to 2.4.1? Hmm... panic is a bit different too:

panic(cpu 6 caller 0xffffff801d13bffa): Kernel trap at 0xffffff7fb36dc33d, type = 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0x00000000000039f0, CR3: 0x0000000030fc6000, CR4: 0x00000000003626e0
RAX: 0x0000000008000080, RBX: 0x00000000000039f0, RCX: 0xffffff7fb3bd02ec, RDX: 0x0000000000000217
RSP: 0xffffffd2f851b840, RBP: 0xffffffd2f851b860, RSI: 0xffffff7fb39d189d, RDI: 0x00000000000039f0
R8:  0x2936a36c8c9a537c, R9:  0x449f8ab8ed2f203e, R10: 0x0004f4cc955d59fe, R11: 0xa5da24260354d800
R12: 0x0000000001000000, R13: 0x0000000005f00000, R14: 0xffffffd2f851b878, R15: 0xffffff7fb3a1f040
RFL: 0x0000000000010206, RIP: 0xffffff7fb36dc33d, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x00000000000039f0, Error code: 0x0000000000000000, Fault CPU: 0x6, PL: 0, VF: 1

Panicked task 0xffffff9f6e4f1980: 885 threads: pid 0: kernel_task
Backtrace (CPU 6), panicked thread: 0xffffff95d5468b30, Frame : Return Address
0xffffffd2f851b0e0 : 0xffffff801cfd6761 
0xffffffd2f851b130 : 0xffffff801d14c846 
0xffffffd2f851b170 : 0xffffff801d13b99c 
0xffffffd2f851b230 : 0xffffff801cf6b971 
0xffffffd2f851b250 : 0xffffff801cfd6a57 
0xffffffd2f851b350 : 0xffffff801cfd60db 
0xffffffd2f851b3c0 : 0xffffff801d7d109d 
0xffffffd2f851b4b0 : 0xffffff801d13bffa 
0xffffffd2f851b670 : 0xffffff801d13bba9 
0xffffffd2f851b730 : 0xffffff801cf6b971 
0xffffffd2f851b750 : 0xffffff7fb36dc33d 
0xffffffd2f851b860 : 0xffffff7fb381d5c4 
0xffffffd2f851b8a0 : 0xffffff7fb3817d5d 
0xffffffd2f851b8e0 : 0xffffff7fb381d019 
0xffffffd2f851b920 : 0xffffff7fb381746d 
0xffffffd2f851b960 : 0xffffff7fb381af68 
0xffffffd2f851ba30 : 0xffffff7fb381746d 
0xffffffd2f851ba70 : 0xffffff7fb36effa9 
0xffffffd2f851bb20 : 0xffffff7fb3708a85 
0xffffffd2f851bc50 : 0xffffff7fb370a579 
0xffffffd2f851bca0 : 0xffffff7fb373af61 
0xffffffd2f851bd10 : 0xffffff7fb371a3e0 
0xffffffd2f851bda0 : 0xffffff7fb372aa0e 
0xffffffd2f851be00 : 0xffffff7fb3729d50 
0xffffffd2f851bef0 : 0xffffff7fb3727ebd 
0xffffffd2f851bf60 : 0xffffff7fb36e1543 
0xffffffd2f851bfa0 : 0xffffff801cf6b19e 
      Kernel Extensions in backtrace:
         org.openzfsonosx.zfs(2.3.1)[66F37767-FB22-3798-872B-F0AA81854AB2]@0xffffff7fb36cf000->0xffffff7fb3a1afff
            dependency: com.apple.iokit.IOStorageFamily(2.1)[0F68373A-6845-334F-A3F9-1547BE96F804]@0xffffff801f5c4000->0xffffff801f5dafff

Process name corresponding to current thread (0xffffff95d5468b30): kernel_task

Mac OS version:
24G517

Kernel version:
Darwin Kernel Version 24.6.0: Mon Jan 19 22:00:10 PST 2026; root:xnu-11417.140.69.708.3~1/RELEASE_X86_64
Kernel UUID: A8F2DD73-2B65-3BF9-9B5A-BA2B753AA960
roots installed: 0
KernelCache slide: 0x000000001cc00000
KernelCache base:  0xffffff801ce00000
Kernel slide:      0x000000001cce8000
Kernel text base:  0xffffff801cee8000
__HIB  text base: 0xffffff801cd00000
System model name: Macmini8,1 (Mac-7BA5B2DFE22DDD8C)
System shutdown begun: NO
Hibernation exit count: 0

System uptime in nanoseconds: 303153894749
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x0000004695615892
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000000534700e9d 0x0000000000000000
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Zone info:
  Zone map: 0xffffff8aa0169000 - 0xffffffaaa0169000
  . PGZ   : 0xffffff8aa0169000 - 0xffffff8aa416a000
  . VM    : 0xffffff8aa416a000 - 0xffffff8f7049c000
  . RO    : 0xffffff8f7049c000 - 0xffffff9109b0c000
  . GEN0  : 0xffffff9109b0c000 - 0xffffff95d5e3e000
  . GEN1  : 0xffffff95d5e3e000 - 0xffffff9aa2170000
  . GEN2  : 0xffffff9aa2170000 - 0xffffff9f6e4a2000
  . GEN3  : 0xffffff9f6e4a2000 - 0xffffffa43a7d5000
  . DATA  : 0xffffffa43a7d5000 - 0xffffffaaa0169000
  Metadata: 0xffffff8a7a159000 - 0xffffff8a9a159000
  Bitmaps : 0xffffff8a9a159000 - 0xffffff8aa0159000
  Extra   : 0 - 0

last started kext at 111828132287: |SCSITaskUserClient	500.120.2 (addr 0xffffff801f542000, size 20480)
loaded kexts:
org.openzfsonosx.zfs	2.3.1
|SCSITaskUserClient	500.120.2
>!UMass!SInterfaceNub	556
>!AUpstreamUserClient	3.6.12
>!AMCCSControl	1.17
>AGPM	137
>!APlatformEnabler	2.7.0d0
>X86PlatformShim	1.0.0
@filesystems.autofs	3.0
>!AGFXHDA	400.1
>!AGraphicsDevicePolicy	8.1.10
>!ABridgeAudio!C	530.2
@AGDCPluginDisplayMetrics	8.1.10
>pmtelemetry	1
@filesystems.nfs	1
|IOUserEthernet	1.0.1
>usb.!UUserHCI	1
>!A!IKBLGraphics	23.0.7
>!ARXE	1.0
>!AHV	1
>BridgeAudioCommunication	530.2
>!AFIVRDriver	4.1.0
>!A!ISlowAdaptiveClocking	4.0.0
>ACPI_SMC_PlatformPlugin	1.0.0
>!AAVEBridge	6.1
>!A!IPCHPMC	2.0.1
>!AThunderboltNHISpeedTest	0.0.1
>!AThunderboltIP	4.0.3
>!A!ICFLGraphicsFramebuffer	23.0.7
@!AEthernetAquantiaAqtionFirmware	1.0.36
>BCMWLANFirmware4388.Hashstore	1
>BCMWLANFirmware4387.Hashstore	1
>BCMWLANFirmware4378.Hashstore	1
>BCMWLANFirmware4377.Hashstore	1
>BCMWLANFirmware4364.Hashstore	1
>!AFileSystemDriver	3.0.1
@filesystems.tmpfs	1
@filesystems.lifs	1
@filesystems.hfs.kext	683.120.3
@BootCache	40
@!AFSCompression.!AFSCompressionTypeZlib	1.0.0
@!AFSCompression.!AFSCompressionTypeDataless	1.0.0d1
@filesystems.apfs	2332.140.13
>!AAHCIPort	384
>!ABCMWLANBusInterfacePCIeMac	1
@private.KextAudit	1.0
>!AACPIButtons	6.1
>!ASMBIOS	2.1
>!AACPIEC	6.1
>!AAPIC	1.7
@!ASystemPolicy	2.0.0
@nke.applicationfirewall	406
|IOKitRegistryCompatibility	1
|EndpointSecurity	1
@Dont_Steal_Mac_OS_X	7.0.0
>!ADiskImages2	385.140.4
@kec.!AEncryptedArchive	1
|IOAVB!F	1320.3
|IOAHCIBlock!S	362.100.1
>!ASMBus!C	1.0.18d1
>!UAudio	750.4
>!AAudioClockLibs	420.3
>usb.IOUSBHostHIDDevice	1.2
>!AGraphicsControl	8.1.10
>!AHDA!C	600.2
|IOHDA!F	600.2
|IOAudio!F	600.2
@vecLib.kext	1.2.0
>!A!ILpssUARTv1	3.0.60
>!A!ILpssUARTCommon	3.0.60
>!AOnboardSerial	1.0
>!ASMBusPCI	1.0.14d1
@kext.triggers	1.0
>IOHIDPowerSource	1
>!ACallbackPowerSource	1
>!AUIO	1
|IORDMA!F	1.0
>!ARSMChannel	1
|IORSM!F	1
>!AIPAppender	1.0
|IONDRVSupport	599
@!AGPUWrangler	8.1.10
|IOSlowAdaptiveClocking!F	1.0.0
>IOPlatformPluginLegacy	1.0.0
>X86PlatformPlugin	1.0.0
>IOPlatformPlugin!F	6.0.0d8
@plugin.IOgPTPPlugin	1340.12
|IOAccelerator!F2	485
@!AGraphicsDeviceControl	8.1.10
|IOGraphics!F	599
>driverkit.serial	6.0.0
>!AThunderboltEDMSink	5.0.3
>!AThunderboltDPOutAdapter	8.5.1
>usb.cdc.ecm	5.0.0
>usb.cdc.ncm	5.0.0
>usb.cdc	5.0.0
>usb.networking	5.0.0
>usb.!UHostCompositeDevice	1.2
>!AThunderboltDPInAdapter	8.5.1
>!AThunderboltDPAdapter!F	8.5.1
>!AThunderboltPCIUpAdapter	4.1.1
>!AThunderboltPCIDownAdapter	4.1.1
>!AHPM	3.4.4
>!A!ILpssI2C!C	3.0.60
>!A!ILpssI2C	3.0.60
>!A!ILpssDmac	3.0.60
>!ABSDKextStarter	3
|IOSurface	372.5.2
@filesystems.hfs.encodings.kext	1
>!ASyntheticGame!C	12.6.1
>!AXsanScheme	3
|IOAHCI!F	308
>!AThunderboltNHI	7.2.81
|IOThunderbolt!F	9.3.3
>usb.!UVHCIBCE	1.2
>usb.!UVHCICommonBCE	1.0
>usb.!UVHCI	1.2
>usb.!UVHCICommon	1.0
>!AEffaceableNOR	1.0
|IONVMe!F	2.1.0
|IOBufferCopy!C	1.1.0
|IOBufferCopyEngine!F	1
>!ABCMWLANCoreMac	1.0.0
|IO80211!F	1200.13.1
>IOImageLoader	1.0.0
>!AOLYHALMac	1
|IOSerial!F	11
>corecapture	1.0.4
>!AEthernetAquantiaAqtion	1.0.64
>usb.!UXHCIPCI	1.2
>usb.!UXHCI	1.2
>usb.!UHostPacketFilter	1.0
|IOUSB!F	900.4.2
>!AEFINVRAM	2.1
>!AEFIRuntime	2.1
>!ASMCRTC	1.0
|IOSMBus!F	1.1
|IOHID!F	2.0.0
|IOTimeSync!F	1340.12
|IOSkywalk!F	1.0
>mDNSOffloadUserClient	1.0.1b8
|IONetworking!F	3.4
>DiskImages	493.0.0
|IO!B!F	9.0.0
|IOReport!F	47
$quarantine	4
$sandbox	300.0
@kext.!AMatch	1.0.0d1
>!ASSE	1.0
>!ALockdownMode	1
>!AKeyStore	2
>!UTDM	556
|IOUSBMass!SDriver	259.100.1
|IOUSBHost!F	1.2
>!UHostMergeProperties	1.2
>usb.!UCommon	1.0
>!ABusPower!C	1.0
|IOSCSIBlockCommandsDevice	500.120.2
|IOSCSIArchitectureModel!F	500.120.2
>!AFDEKeyStore	28.30
>!AEffaceable!S	1.0
|IO!S!F	2.1
>!ACyrus	1
>!AMobileFileIntegrity	1.0.5
$!AImage4	7.0.0
@kext.CoreTrust	1
>!ACredentialManager	1.0
|CoreAnalytics!F	1
>!ASEPManager	1.0.1
>IOSlaveProcessor	1
>!AACPIPlatform	6.1
>!ASMC	3.1.9
|IOPCI!F	2.9
|IOACPI!F	1.4
>watchdog	1
@kec.pthread	1
@kec.XrtHostedXnu	1
@kec.Libm	1
@kec.Compression	1.0
@kec.corecrypto	14.0

[lundman]

blkptr at 0xffffffe16aae7a40x has no valid DVAs is coming from:

        } else if (unlikely(!DVA_IS_VALID(&bp->blk_dva[0]))) {
                /* Non-hole, non-embedded BPs _must_ have at least one DVA */
                errors += zfs_blkptr_verify_log(spa, bp, blk_verify,
                    "blkptr at %px has no valid DVAs", bp);
        }

which is a ZFS panic, not a macOS one. It is concerning though. You can tell it to not panic in this case, for best-effort-recover of the data though.

[nexusgoblin]

Yeah, I was just looking around and came across this: openzfs#17307

I have the same failure on send to a different pool (I have a backup of the backups). On the second pool though, the send failure completely toasted the pool (unable to import). I've recreated both target pools and am about to retry some sends. Its distrubing that this has appeared on both destination pools at "the same time" which makes me suspicious about the source pool, however, that seems fine (scrubbed ok).

Will continue to futz about.

[lundman]

Yeah, set zfs_recover=1 and then try sends.

[nexusgoblin]

Target pools already recreated, but will try that if it occurs again. Is that a sysctl or environment variable?

[lundman]

sysctl on macOS

[nexusgoblin]

Is the sysctl zfs_recover or actually kstat.zfs.darwin.tunable.zfs.recover? I've set the latter, however the panics continued.

I'm guessing there is something in the source pool triggering... "something" in send/recv. The source pool has 3 data sets, one about 37Gb (send/recv with this one panics), another about 1.2T (send/recv with this one succeeds) and another ~4.5Tb (ZVOL actually; send/recv with this one panics). The 1.2T dataset probably hasn't seen meaningful write operations in... a long time. The other two are hit hard weekly.

I've given up with send/recv from the source pool and I'm copying the data off to other storage to rebuild the source pool. The copying of the data off to other storage seems to be working fine - the 37GB that paniced all copied "fine" and so far, about half the data from the ZVOL has been copied off "fine".

Something seems to be leaking memory though (not sure what). A couple of times I've caught the box teetering, having wired 31+Gb of its 32Gb. This happens copying ~500Gb from dataset to dataset or dataset to EXFAT (as an example). I've not seen this before and am still trying to get a handle on it. This is running 2.4.1rc4. I'll see about switching back to 2.3.1 and see if it still happens.

[Nazari]

I've also the same problem. Steps to reproduce:

• Uncompress tank-dev1.bin.gzip and tank-dev2.bin.gzip (both are 256MB size)
• create loopbacks in MacOS:

hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount tank-dev1.bin
hdiutil attach -imagekey diskimage-class=CRawDiskImage -nomount tank-dev2.bin

• Then

sudo zpool import -N tank1
sudo zpool import -N -F tank2

(tank2 has to be rewinded, actual status is a failed recv)

• Remove tank2/sds1

sudo zfs destroy -R tank2/sds1

If you execute send/recv, panic and reboot:

sudo zfs send -wLecR tank1/sds1@GSA-hourly-20260322-130000 | pv -trab | sudo zfs recv -Fus tank2/sds1

tank-dev1.bin.gz
tank-dev2.bin.gz

[nexusgoblin]

Something seems to be leaking memory though (not sure what). A couple of times I've caught the box teetering, having wired 31+Gb of its 32Gb. This happens copying ~500Gb from dataset to dataset or dataset to EXFAT (as an example). I've not seen this before and am still trying to get a handle on it. This is running 2.4.1rc4. I'll see about switching back to 2.3.1 and see if it still happens.

I rolled back to 2.3.1 (not 2.3.1p1) and memory issue seems to be gone. I've copied a ~1.1T sparse bundle from the ZVOL mentioned above to another (mirrored) pool and wired memory has stayed around 19Gb where as it would have hit 31Gb+ (and the box likely panic/crashed) before completing while running 2.4.1rc4.