High CPU utilization on RX side with UDP GRO (--gsro)

[kvasana]

Hi all,

I'm seeing high CPU utilization on the RX side with UDP GRO enabled (--gsro) when the iperf3 server is pinned to a single core using taskset. The pinned RX core becomes saturated.

While looking at the receive path, I noticed that nread_gro() / recv_msg_gro() use recvmsg() with MSG_DONTWAIT and parse the UDP_GRO cmsg. In iperf_udp_recv(), however, the returned segment size appears to be overwritten with blksize and not used afterwards.

As an experiment, I replaced the recvmsg() path with a plain recv() and observed lower CPU utilization with no noticeable throughput change in my testing.

Is this expected? Has anyone else observed similar behavior with --gsro enabled? I'm also interested in understanding whether the UDP_GRO cmsg information is required by iperf3's current receive path, and whether recvmsg()/cmsg handling (or the use of MSG_DONTWAIT) is expected to contribute measurable overhead in this case.

Thanks.

[davidBar-On]

I replaced the recvmsg() path with a plain recv() and observed lower CPU utilization with no noticeable throughput change

Hi, that may be because the throughput bottleneck is in the sending side (client) or in the network throughput. In these cases recvmsg() probably does not have any advantage.

I noticed that nread_gro() / recv_msg_gro() use recvmsg() with MSG_DONTWAIT

I am not sure about the reason for the MSG_DONTWAIT, but maybe the assumption is that GRO (and GSO) is used mainly when the receiver (sender) processing (e.g. CPU load) is the bottlneck, and not the network throughput. In this case, the recvmsg() will almost always return some data.

On the other hand, if the network throughput is the bottleneck, the recvmsg() with MSG_DONTWAIT will probably return no data most of the time, and the receiver (server) will be in a "CPU" load loop. Can it be that this is what you are seeing? What is the throughput you get in your tests? What is the network link throughput?

[kvasana]

Thanks - I don't suspect the network. On a 10G link I offer ~10 Gbps (1472B datagrams)
and achieve ~9.57 Gbps / ~812k pps - enough data is arriving. The issue is that the pinned
RX core (CPU0) saturates (~100%) processing it. So the CPU utilization is my concern, not
throughput.

Setup (server pinned to CPU0; GRO is negotiated to the server via the client's --gsro):
TX: iperf3 -c -p 6000 -u -b 10G -l 1472 -t 200 -i 5 --gsro
RX: taskset -c 0 iperf3 -s -p 6000

Switching to a blocking receive drops the RX core to ~27% idle, throughput unchanged.
Reproduced on two different Ethernet MAC configurations (hardware offload, GRO & GSO) and
multiple kernels.

Two factors:

1. MSG_DONTWAIT: between batches recvmsg() returns EAGAIN and the thread retries
   immediately (the busy-poll you described) - blocking alone lowers CPU.
2. recv_msg_gro() parses the UDP_GRO cmsg every call, but iperf_udp_recv() overwrites the
   segment size with blksize, so it's unused - paid on every receive that returns data;
   plain recv() removes it, on top of (1).

Also (separate from CPU): with stock recvmsg+MSG_DONTWAIT we couldn't reach line rate on
more than one port without raising socket buffers (net.core.rmem_max/rmem_default=64M);
with the change we hit line rate on all ports without tuning.

Questions:

• Is the MSG_DONTWAIT (non-blocking) behavior on the GRO path intentional?
• Is the UDP_GRO cmsg needed, given the size is overwritten with blksize?
• Is the stock path expected to need larger rmem to scale across ports, while recv() doesn't?

[davidBar-On]

On a 10G link I offer ~10 Gbps (1472B datagrams)
and achieve ~9.57 Gbps / ~812k pps - enough data is arriving.
....
Switching to a blocking receive drops the RX core to ~27% idle
...
Is the MSG_DONTWAIT (non-blocking) behavior on the GRO path intentional?

From the almost 10Gbps and the 27% idle it seems that your server (and probably client) machines are strong to process the data without the need for GSO/GRO. Therefore, as I wrote above, it may be that the current --gsro implementation with the MSG_DONTWAIT assumes it is used only in cases were the machines are not fast enough to fully load the network. Maybe @gegles that wrote the iperf3 GSRO implementation in PR #1925 can help with the question(s).

Is the UDP_GRO cmsg needed, given the size is overwritten with blksize?

My understanding is that the blksize does not overwrite the size. The blksize is only used to allow parsing the received packets into the separate UDP messages that were sent by the client.

[kvasana]

Thanks @davidBar-On , Sharing isolation results, then the cmsg point.
Setup: ARM host (2 cores), single UDP stream, -b 10G -l 1472, server pinned to CPU‑0. Builds: V1 stock, V2 blocking recvmsg (MSG_DONTWAIT→0), V3 blocking recv().

Findings: the MSG_DONTWAIT busy-poll is the dominant cost — blocking (V1→V2) recovers ~28% idle on the pinned core. Plain recv() (V2→V3) adds ~4–5% more. No loss penalty (all <0.02%).

Variant	Receive call	CPU‑0 %idle	Throughput	loss	RcvbufErr Δ
V1 stock	recvmsg(MSG_DONTWAIT)	~0.9%	9.57 Gbps	0.009%	275
V2 blocking	recvmsg(0) + cmsg	~28.6%	9.57 Gbps	0.014%	530
V3 recv	recv(0)	~33%	9.57 Gbps	0.007%	270

On cmsg: agreed blksize correctly drives the parse loop. My only point is that the cmsg value is discarded — in iperf_udp_recv():

r = Nread_gro(sp->socket, sp->buffer, size, Pudp, &dgram_sz);
dgram_sz = sp->settings->blksize;   // overwrites the cmsg gso_size on the next line

So recv_msg_gro()'s cmsg walk never reaches the parser, which is why plain recv() parsed identically with less CPU. The cmsg is still useful for validation / non-uniform datagrams — a safe approach: use gso_size when present, fall back to blksize, validate on mismatch.

On MSG_DONTWAIT: deferring to @gegles (#1925) on whether the non-blocking receive was intentional. On my setup blocking recovers ~27% of the RX core with no loss regression.

Happy to send a PR — blocking-only, or blocking + recv() with cmsg fallback?

[davidBar-On]

Happy to send a PR — blocking-only, or blocking + recv() with cmsg fallback?

I think that sending a blocking-only PR is a good idea (note that I am not from the iperf3 maintenance team). I don't suggest to include the recv() change, as that will reduce the chance that the PR will be merged, since the iperf3 team is sensitive about making changes.

A major potential issue with removing the MSG_DONTWAIT could be that in case of a network, etc. problem the server may be stuck forever waiting for the message. However, I tested and --rcv-timeout which is always active solves this problem. It is important to say this in the PR and it will help to describe how to test this case (I did it by adding sleep(100) in the send loop).

On cmsg: agreed blksize correctly drives the parse loop. My only point is that the cmsg value is discarded — in iperf_udp_recv():

OK. I understand now. Actually, I think that the main problem is that Nread_gro() should return -1 when *dgram_sz < 0 (or even when it is equal to 0, I am not sure). If this will be done, the dgram_sz = sp->settings->blksize may be redundant. I didn't test this so I not sure.

In any case, if you test this approach and have conclusion / proposed changes, I suggest to send them as a separate PR and not add them to the above non-blocking PR, to keep that PR as simple as possible.

[gegles]

Thanks for doing the isolation. The results make a good case that MSG_DONTWAIT plus immediate retry is the dominant CPU cost. A blocking-only PR is the right first step.

For design context, MSG_DONTWAIT was inherited from the earlier GSO/GRO implementation; it was not an intentional policy choice in my later work.

I agree that the cmsg question should be handled separately. One caution: the negotiated blksize stride was intentional because using kernel GRO hints had previously produced incorrect loss accounting. Also, a missing UDP_GRO cmsg can represent a valid uncoalesced datagram, so Nread_gro() should not return an error solely because dgram_sz is unset. A future cleanup could compare the cmsg value with blksize for validation/debugging while retaining blksize as the parsing stride.

For the blocking PR, please test normal termination, a short --rcv-timeout, interrupted input, reverse/bidirectional mode, and multiple streams. It would also be useful to repeat the measurements because the blocking-recvmsg() run shows a higher RcvbufErr delta; that may be noise, but it should be separated from the clear CPU improvement.

[kvasana]

Opened #2069 with the blocking-only change as discussed (removes MSG_DONTWAIT from the GRO receive path). Kept the cmsg cleanup out to keep it minimal, per the earlier feedback — happy to do that as a separate PR later.

[kvasana]

I also noticed that with --gsro enabled, iperf_time_now() is called many times in the RX path — effectively once per coalesced segment.

Would it be acceptable to timestamp once per GRO receive operation and reuse that timestamp for jitter processing, to reduce CPU cost? Since a coalesced batch is delivered as a single kernel event, the per-segment timestamps effectively reflect loop-iteration timing rather than true arrival.

Is per-segment timestamping required for jitter accuracy in iperf3's current model, or would a batched (per-receive) timestamp be an acceptable tradeoff? Happy to send a separate PR (independent of #2069) with before/after CPU numbers if this sounds reasonable.

[gegles]

Yes, I think a separate PR to timestamp once per GRO receive operation is reasonable.

Once GRO has coalesced multiple datagrams, iperf no longer has genuine per-segment arrival timestamps. Calling iperf_time_now() inside the parsing loop mostly measures user-space loop progress rather than distinct network arrival times. Taking one timestamp immediately after recvmsg() returns and reusing it for the segments in that batch is therefore a defensible model and should reduce overhead.

The main caveat is that this changes jitter semantics: the result becomes a batch-level approximation, not a more accurate per-datagram measurement. Please include before/after CPU profiling and compare the reported jitter with GRO disabled under controlled pacing/jitter. The behavior should also be documented so users do not interpret GRO-mode jitter as true wire-level per-segment timing.

Keeping this independent of #2069 is the right approach.

[kvasana]

Thanks — that matches our reasoning. We ran the measurements you asked for.

Setup: single UDP stream at ~9.57 Gbps, GRO on, receive core isolated with mpstat. Both builds include the blocking-receive change so the freed cycles are actually visible.

CPU (GRO on, 9.57 Gbps):

Timestamping	receive-core %idle	Jitter	Loss
per-segment (before)	3.90%	0.002 ms	0.0078%
one per batch (after)	7.40%	0.000 ms	0.0051%

Hoisting recovers ~3.5% of the receive core, with no throughput/loss regression (reproduced on a second platform).

Jitter vs GRO off (controlled pacing): GRO-off (true per-datagram) 0.022 ms vs GRO-on (hoisted) 0.000 ms — same sub-millisecond ballpark, so the batch-level value doesn't distort reported jitter.

Docs: the PR will note in docs/invoking.rst that under GRO the jitter is a batch-level approximation (one timestamp per coalesced batch), not per-segment wire timing.

I'll open this as a separate PR, independent of #2069.

[gegles]

Thanks—the CPU result is useful and supports continuing with a separate optimization PR. I don’t think the jitter result supports the “no distortion” conclusion yet, though.

The current receive loop updates the jitter EWMA once per segment. If one arrival timestamp is reused across every segment in a GRO batch, each batch applies many near-zero updates consecutively, which can drive the reported jitter toward zero and make it dependent on the coalesced batch size. The change from 0.022 ms to 0.000 ms may be evidence of that effect rather than equivalence.

Please report the raw JSON jitter_ms, since the normal output rounds to three decimal places. Also, controlled pacing does not introduce controlled network jitter; testing with a known injected delay variation, such as netem, would provide a stronger GRO-off/GRO-on comparison. It would also be useful to vary the GRO batch size or rate.

If the intended metric is batch-level jitter, I think the cleaner model is to update the jitter estimator once per GRO batch using a clearly defined representative send timestamp, rather than updating it once per segment with the same receive timestamp. The documentation should then describe that batch-level semantic explicitly.