Avoiding MmProbeAndLockPages overhead in ReadFile

[Ext3h]

Describe the bug
Technically a performance bug, in the combination of npcap with libpcap, when trying to tune npcap for 10GBit+ operation.

This assumes 8MB userspace and kernel side buffers for npcap, in order to get the rate of syscalls to a manageable level in the first place.

For context have a look at the opposing side:
https://github.com/the-tcpdump-group/libpcap/blob/fbcc461fbc2bd3b98de401cc04e6a4a10614e99f/pcap-npf.c#L542
https://github.com/the-tcpdump-group/libpcap/blob/fbcc461fbc2bd3b98de401cc04e6a4a10614e99f/pcap-npf.c#L432

When using PacketReceivePacket with above buffer sizes, a significant share of time (70%+) is spent in MmProbeAndLockPages on the level of NtReadFile prior to NPF_read respectively MmUnlockPages at IoCompleteRequest.

As a user of libpcap, the memory for the user space buffer is allocated privately within libpcap, and is never exposed in raw form to the user of the API. Only PacketInitPacket or PacketReceivePacket on the npcap end has the guarantee to see the raw buffer.

Given the design of these 2 APIs, as a user I can't do anything to speed up MmProbeAndLockPages from the outside. I can't choose large pages, I can't VirtualLock.

What's worse, is that the overhead scales linearly with the size of the user space buffer - not with the actual amount of data transferred.

Expected behavior

Specifying a large user space buffer on libpcap side doesn't result in excessive overhead on npcap side.

Either npcap optimizes the buffer for repeated use (e.g. by explicitly applying VirtualLock in PacketInitPacket), or libpcap turns smarter when allocating the memory.

[Ext3h]

Issue appears to have been exacerbated by not activating the blocking mode - and thereby calling PacketReceivePacket with a large buffer excessively often. The overhead is significantly lower in blocking mode when the majority of calls does actual work.

[dmiller-nmap]

This is an interesting idea, and I'm reopening it for further investigation. I would be interested in what your bottlenecks turn out to be once you've optimized transfers using the existing tools: pcap_setmintocopy() and pcap_getevent() or pcap_set_timeout(), as well as managing buffer sizes with pcap_setuserbuffer() and pcap_set_buffer_size().

I do think we could avoid some of this overhead, but VirtualLock may not be the right tool. Here are some links I found so far on the topic of locking a buffer to be used for DMA between user-mode and kernel-mode:

• https://www.osr.com/nt-insider/2014-issue3/windows-real-time/
• https://community.osr.com/discussion/265341/shared-memory-between-kernel-and-user-mode

[dmiller-nmap]

We are giving more thought to this issue and hope to have a plan for some improvement soon. Here are some current thoughts:

1. As @Ext3h noted when they closed the issue, libpcap and Npcap already have mechanisms available to reduce the number of system calls and increase the utilization of those calls, reducing some of the overhead. Specifically, the packet buffer timeout (pcap_set_timeout() and avoiding pcap_set_nonblock()) is used by Packet.dll to throttle calls to ReadFile() to allow each one to retrieve more packets at a time. The MinToCopy value (pcap_setmintocopy()) sets a desired minimum amount of packet data for each ReadFile() call. The Npcap driver signals Packet.dll when this amount is available, interrupting the timeout wait. The result is that a call to pcap_dispatch() will return when the timeout expires or the minimum data is available, whichever happens first. See Npcap buffer sizes are not well-documented #30 for discussion about improving Npcap's documentation on buffer sizes and tuning.
2. Assuming VirtualLock() is an appropriate solution for this issue, libpcap currently does not use PacketAllocatePacket()/PacketFreePacket(), so that is not a suitable place to insert the call. PacketInitPacket() does not have a corresponding de-initialization function in which to call VirtualUnlock(), so that is also not suitable; also it is used for packet injection as well, which may use many different buffers. It seems more appropriate, then, to add it to libpcap.
3. As discussed at libpcap by default sets a snaplen that will not fit in the user buffer (NPF) the-tcpdump-group/libpcap#975, a more drastic change could be the way ahead: doing away with ReadFile() and instead using a shared buffer the way that memory-mapped capture on Linux does. This would be a big departure from the current Packet API, so it makes sense to discuss that at Potential enhancement: alternative to Packet.dll #569 instead.

[dmiller-nmap]

VirtualLock is not an appropriate solution, and the existing overhead-reducing mechanisms are sufficient for now. As we do further documentation for buffer sizing (#30), it is important to note that readfile calls incur overhead proportional to the user buffer size even if only a small amount of data is transferred. Closing this issue in favor of better documentation and existing performance suggestions.