[BUG] UDP socket RX loss-free throughput is not reproducible under paced cross-host load

[dleshchev]

Describe the bug

On DAQIRI commit 263f4b5c3aedf25698c6404309fac6158359b6ef (main at measurement time), paced cross-host UDP receive throughput is not reliably loss-free under repeated runs, particularly with multiple receiver workers.

The benchmark pins its consumer worker with socket_bench_*.cpu_core, but the UDP socket engine runs recvmmsg() on a separate scheduler-selected I/O thread. In addition, each datagram returned by recvmmsg() becomes a separate one-packet DAQIRI burst and receive-queue handoff. Under load, the host can fall behind draining the UDP socket and lose packets after they have crossed the wire.

This makes PR #276-style UDP core-scaling measurements sensitive to incidental scheduler placement rather than only the configured benchmark-worker topology.

Steps/Code to reproduce bug

1. Build DAQIRI commit 263f4b5c3aedf25698c6404309fac6158359b6ef in Release mode and run daqiri_bench_socket in privileged Docker containers on two DGX Spark hosts connected by one physical 100 GbE link.
2. Use 8 KB UDP datagrams, one-way traffic, a 30-second paced client transfer, and a 70-second server lifetime.
3. Keep the benchmark worker cores fixed and non-overlapping. On main, do not configure UDP I/O-thread affinity; it is not supported.
4. Run each point three times: one RX worker at 27.5 Gb/s; two RX workers at 40 Gb/s aggregate; and four RX workers at 75 Gb/s aggregate.
5. While the server socket is still alive, compare sender tx_packets_phy / tx_bytes_phy with receiver PHY counters, application packet counts, receiver /proc/net/snmp UdpInErrors / RcvbufErrors, and NIC receive-buffer discard counters.

Expected behavior

At a stated loss-free operating point, repeated runs with the same physical topology and configured worker placement should remain free of kernel socket-buffer and NIC receive-buffer drops. UDP I/O placement should be explicit rather than depending on scheduler placement.

Environment overview (please complete the following information)

• Environment location: Bare-metal hosts; benchmark runs in Docker
• Method of DAQIRI install: Source build in daqiri:main-5f1445f
• Representative invocation: docker run --rm --privileged --network host --gpus all ... daqiri:main-5f1445f

Environment details

• OS: Ubuntu 24.04, Linux 6.17
• DAQIRI version: 2026.7.0, commit 263f4b5c3aedf25698c6404309fac6158359b6ef
• Hardware: Two DGX Spark systems with ConnectX-7 NICs and one direct 100 GbE link
• Workload: 8 KB UDP payloads, no GPU post-processing

Additional context

Physical TX/RX counters matched on every run, so these losses are not cable loss. They were primarily accounted for by receiver RcvbufErrors; at the four-worker point, a smaller portion was also visible in NIC rx_out_of_buffer.

Current-main results:

• One worker at 27.5 Gb/s was clean in 0/3 runs. RcvbufErrors increased by 1,048,087, 76,782, and 925,631 packets.
• Two workers at 40 Gb/s were clean in 2/3 runs; the failing run had 5,054 kernel socket-buffer drops.
• Four workers at 75 Gb/s were clean in 0/3 runs, losing about one million packets per run. Most were kernel drops; NIC receive-buffer drops also occurred.

The combined implementation in draft PR #280 was counter-clean in 3/3 runs at all three points. This is evidence for the combined change, not a precise per-change attribution:

• Receive batching makes one recvmmsg() return one DAQIRI burst, removing a burst allocation and receive-queue handoff per datagram. The unchanged standard harness independently showed higher application RX throughput with batching.
• socket_config.udp_rx_cpu_core pins the actual UDP I/O thread separately from the consumer worker, removing the scheduler-placement ambiguity.

The combined patch does not establish a universal ceiling: two-worker runs became unreliable at 45 Gb/s and above, while 40 Gb/s is the demonstrated 3/3 point.

Draft PR #280 is the proposed narrow fix. Drop-accounting and benchmark-lifecycle improvements should remain separate follow-up work.