non-blocking libpcap + select = lost packets

[guyharris]

Converted from SourceForge issue 3548076, submitted by None

It seems that libpcap is losing response packets when the come back to fast (when using a non-blocking pcap handler with select). For more details, please check http://stackoverflow.com/questions/11637063/asynchronous-libpcap-losing-packets.

[guyharris]

Submitted by guy_harris

If you really are dropping packets, then, as I indicated in the stackoverflow question:

If the FD for the pcap_t is reported as readable by select() (or poll() or whatever call/mechanism you're using), there is no guarantee that this means that only one packet can be read without blocking.

If you use pcap_next_ex(), you will read only one packet; if there's more than one packet available to be read, then, if you do another select(), it should immediately return, reporting the FD as being readable again, in which case you'll presumably call pcap_next_ex() again, and so on. This means at least one system call per packet (the select()), and possibly more calls, depending on what version of what OS you're doing and what version of libpcap you have.

If, instead, you were to call pcap_dispatch(), with a packet-count argument of -1, that call will return all the packets that can be obtained with a single read operation and process all of them, so, on most platforms, you may get multiple packets with one or two system calls if there are multiple packets available (which, with high network traffic, as you might get if you're testing your program with a SYN flood, is likely to be the case).

In addition, on Linux systems that support memory-mapped packet capture (I think all 2.6 and later kernels do, and most if not all 2.4 kernels do), and with newer versions of libpcap, pcap_next_ex() has to make a copy of the packet to avoid having the kernel change the packet out from under the code processing the packet and to avoid "locking up" a slot in the ring buffer for an indefinite period of time, so there's an extra copy involved.

[guyharris]

Submitted by None

After further investigation, I realized that even without select libpcap is losing packets. Please see this new SO question: http://stackoverflow.com/questions/11637063/asynchronous-libpcap-losing-packets. The workaround is to recompile libpcap without memory mapping support.

[guyharris]

The workaround is to recompile libpcap without memory mapping support.

Then the problem is probably due to TPACKET_V1's and TPACKET_V2's fixed-size packet slots.

libpcap 1.0.x and 1.1.x used the snapshot length as the slot size; the default snapshot length of 65535 in newer versions of tcpdump and all but ancient versions of Wireshark (so ancient that it was still called Ethereal) means you have only a small number of those large packet slots, so the kernel can easily run out of packet slots and drop packets.

libpcap 1.2 and later attempt, on Ethernet, to pick a slot size based on the Ethernet MTU, if no segmentation/reassembly offloading is being done. That makes things somewhat better, but....

The upcoming libpcap 1.5 will use TPACKET_V3 on kernels that support it (3.2 and later); it does not have fixed-size packet slots, and should drop a lot fewer packets (some testing done at Ericsson shows it dropped fewer packets).

[infrastation]

libpcap 1.5.0 had introduced support for TPACKET_V3 in 2013. So long as the original questions have been answered and the issue does not have a reporter anymore and it does not describe a problem that can be worked on I am closing it.