Npcap causes corruption in system's network stack

[haitetra]

Hello Npcap developers,

After installing an in-house utility and npcap 0.94, a system running Windows Server 2012 R2 was found to be un-responding, and non of the applications were able to use the networking functionalities.

We took a memory dump and found a potential issue caused by Npcap:

4: kd> !analyze -v
 
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000170, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff88000f27a83, address which referenced memory

STACK_COMMAND:  kb
 
FOLLOWUP_IP:
NETIO!NetioDereferenceNetBufferListChain+132
fffff880`00e60872 4c8bb42490000000 mov     r14,qword ptr [rsp+90h]
 
SYMBOL_STACK_INDEX:  7
 
SYMBOL_NAME:  NETIO!NetioDereferenceNetBufferListChain+132
 
FOLLOWUP_NAME:  MachineOwner
 
MODULE_NAME: NETIO
 
IMAGE_NAME:  NETIO.SYS
 
DEBUG_FLR_IMAGE_TIMESTAMP:  5b708e7f
 
FAILURE_BUCKET_ID:  X64_0xD1_NETIO!NetioDereferenceNetBufferListChain+132
 
BUCKET_ID:  X64_0xD1_NETIO!NetioDereferenceNetBufferListChain+132
 
Followup: MachineOwner
---------

4: kd>
Child-SP          RetAddr           Call Site
fffff880`0239a4c8 fffff800`024aad69 nt!KeBugCheckEx
fffff880`0239a4d0 fffff800`024a8b88 nt!KiBugCheckDispatch+0x69
fffff880`0239a610 fffff880`00f27a83 nt!KiPageFault+0x448
fffff880`0239a7a0 fffff880`00f7d8d4 NDIS!NdisFReturnNetBufferLists+0x23
fffff880`0239a7d0 fffff880`00fb417b NDIS!ndisReturnNetBufferListsInternal+0x94
fffff880`0239a810 fffff880`01c56c06 NDIS!NdisReturnNetBufferLists+0x3b
fffff880`0239a850 fffff880`00e60872 tcpip!FlpReturnNetBufferListChain+0x96
fffff880`0239a8a0 fffff880`01c8c82f NETIO!NetioDereferenceNetBufferListChain+0x132
fffff880`0239a970 fffff880`01c636e5 tcpip!TcpFlushDelay+0x13f
fffff880`0239aa50 fffff880`01c444bc tcpip!TcpPreValidatedReceive+0x3e5
fffff880`0239ab20 fffff880`01c563f2 tcpip!IpFlcReceivePreValidatedPackets+0x5bc
fffff880`0239ac80 fffff800`02433969 tcpip!FlReceiveNetBufferListChainCalloutRoutine+0xa2
fffff880`0239acd0 fffff880`01c56b22 nt!KeExpandKernelStackAndCalloutEx+0x2c9
fffff880`0239adc0 fffff880`00fb40eb tcpip!FlReceiveNetBufferListChain+0xb2
fffff880`0239ae30 fffff880`00f7dad6 NDIS!ndisMIndicateNetBufferListsToOpen+0xdb
fffff880`0239aea0 fffff880`00f007a4 NDIS!ndisMDispatchReceiveNetBufferLists+0x1d6
fffff880`0239b320 fffff880`00f00719 NDIS!ndisMTopReceiveNetBufferLists+0x24
fffff880`0239b360 fffff880`00f006b0 NDIS!ndisFilterIndicateReceiveNetBufferLists+0x29
fffff880`0239b3a0 fffff880`20876b43 NDIS!NdisFIndicateReceiveNetBufferLists+0x50
fffff880`0239b3e0 fffff880`00f18c24 npcap+0x5b43

Could you please take a look and advice if this is a known issue. Note that we have been installing the same combination of utility + Npcap 0.94, but this is the 1st instance it happened, so it's not easily reproducible. Rebooting the system seems to have fixed the issue (at least we no longer observed it).

Thanks !

[geraldcombs]

Have you tried the current release (0.99r7)? According to https://github.com/nmap/npcap/releases there have been a ton of bugs fixed since 0.94 was released.

[haitetra]

@geraldcombs we are planning to upgrade to latest release, however since the problem is really hard to reproduce I was wondering if we have really fixed the problem. I saw a couple of bug fixes that handle similar crash but I didn't see the same stack trace. It would be great if we could confirm or investigate this crash.

[dmiller-nmap]

@haitetra We are unaware of a crash with a similar backtrace, but as Gerald pointed out, there have been extensive changes since 0.94. It would be best to test with Npcap 0.99-r7 to see if the issue has been fixed by some other code change. If there is still a crash, or if you are completely unable to test on a later version, send the crash minidump to dmiller@nmap.com and I will investigate.

[haitetra]

Thanks @dmiller-nmap for helping. I have emailed you the full dump and will be waiting for your investigation result. In the mean time we will try the latest release, but again this crash was not easily reproducible so it's hard to confirm if the crash is fixed just by testing. I hope you could pinpoint exactly the cause of the crash based on the collected dump.

[dmiller-nmap]

We're currently debugging the crash dump. I don't have anything solid to report, but I do note 2 other drivers in the call stack: be2nd62.sys (HPE Emulex 10GB NDIS miniport adapter) and cpqteam.sys (HP NIC teaming driver).

[dmiller-nmap]

We have not located a cause for this crash. If it happens again with the latest Npcap, please open a new bug report. Thanks!