`--bidir` TCP test deadlocks with "idle timeout for receiving data" under load

[AlexMoshkov]

Summary

When running iperf3 in bidirectional mode (--bidir) on a loaded host (e.g. many concurrent tests, virtualized environment), the test immediately deadlocks: both receiver streams report 0 bytes from the first second, and the server eventually exits with:

error - idle timeout for receiving data

Environment

• iperf3 version: 3.15
• OS: Linux 6.6 Ubuntu 20.04

Test configuration

The bug was observed in the following setup:

• 48 virtual machines (KVM/QEMU), each running two concurrent bidirectional iperf3 tests (96 total iperf3 processes)
• Each VM simultaneously acts as both server and client: every VM runs an iperf3 server and connects as a client to another VM, so all VMs are testing against each other.
• Each iperf3 is pinned to a single physical CPU core on VM (-A 0 / -A 1)
• IPv6 connectivity between VMs
• Test duration: 120 seconds

# Each VM runs a server (accepting connections from other VMs)
iperf3 -s -1 --idle-timeout 10 -J -A 0 -p 10000

# Each VM also runs as a client connecting to another VM
iperf3 --bidir -6 -t 120 -c <peer_vm> -J -A 1 -p 10000

Reproduces reliably under load. On an idle host the race window is small and the bug is rare.

Expected behavior

Each iperf3 --bidir test completes successfully, reporting non-zero throughput in both directions for the full test duration.

Actual behavior

Under load, some tests deadlock immediately: both receiver streams report 0 bytes transferred from the very first interval. The server eventually exits with:

error - idle timeout for receiving data

The client-side receiver stream also shows 0 bytes throughout. No data is transferred in either direction for the entire test duration.

Here logs that show this behavior:

Server intervals

{
        "streams":	[{
                "socket":	5,
                "start":	0,
                "end":	1.003693,
                "seconds":	1.0036929845809937,
                "bytes":	0,
                "bits_per_second":	0,
                "omitted":	false,
                "sender":	false
            }, {
                "socket":	8,
                "start":	0,
                "end":	1.003701,
                "seconds":	1.0037009716033936,
                "bytes":	12615680,
                "bits_per_second":	100553295.1101696,
                "retransmits":	1,
                "snd_cwnd":	449820,
                "snd_wnd":	0,
                "rtt":	5106,
                "rttvar":	1964,
                "pmtu":	1500,
                "omitted":	false,
                "sender":	true
            }],
    }, {
        "streams":	[{
                "socket":	5,
                "start":	1.003693,
                "end":	2.00106,
                "seconds":	0.997367024421692,
                "bytes":	0,
                "bits_per_second":	0,
                "omitted":	false,
                "sender":	false
            }, {
                "socket":	8,
                "start":	1.003701,
                "end":	2.001067,
                "seconds":	0.99736601114273071,
                "bytes":	0,
                "bits_per_second":	0,
                "retransmits":	0,
                "snd_cwnd":	449820,
                "snd_wnd":	0,
                "rtt":	5106,
                "rttvar":	1964,
                "pmtu":	1500,
                "omitted":	false,
                "sender":	true
            }],
    }, <intervals with 0 bytes up to test finish>

Client intervals

{
        "streams":	[{
                "socket":	5,
                "start":	0,
                "end":	1.003693,
                "seconds":	1.0036929845809937,
                "bytes":	0,
                "bits_per_second":	0,
                "omitted":	false,
                "sender":	false
            }, {
                "socket":	8,
                "start":	0,
                "end":	1.003701,
                "seconds":	1.0037009716033936,
                "bytes":	12615680,
                "bits_per_second":	100553295.1101696,
                "retransmits":	1,
                "snd_cwnd":	449820,
                "snd_wnd":	0,
                "rtt":	5106,
                "rttvar":	1964,
                "pmtu":	1500,
                "omitted":	false,
                "sender":	true
            }],
    }, {
        "streams":	[{
                "socket":	5,
                "start":	1.003693,
                "end":	2.00106,
                "seconds":	0.997367024421692,
                "bytes":	0,
                "bits_per_second":	0,
                "omitted":	false,
                "sender":	false
            }, {
                "socket":	8,
                "start":	1.003701,
                "end":	2.001067,
                "seconds":	0.99736601114273071,
                "bytes":	0,
                "bits_per_second":	0,
                "retransmits":	0,
                "snd_cwnd":	449820,
                "snd_wnd":	0,
                "rtt":	5106,
                "rttvar":	1964,
                "pmtu":	1500,
                "omitted":	false,
                "sender":	true
            }],
    }  <intervals with 0 bytes up to test finish>

Root cause

The bug: sender/receiver roles are assigned by accept() order

In bidirectional mode, iperf3 creates two TCP connections — one for each direction. The protocol for role assignment is:

Client (iperf_client_api.c):

if (iperf_create_streams(test, 1) < 0)  // sender first
    return -1;
if (iperf_create_streams(test, 0) < 0)  // receiver second
    return -1;

Server (iperf_server_api.c):

if (rec_streams_accepted != streams_to_rec) {
    flag = 0;   // first accepted -> receiver
} else if (send_streams_accepted != streams_to_send) {
    flag = 1;   // second accepted -> sender
}

The implicit contract is: "client connects sender first, so server will accept it first and assign it as receiver".

I also think that this bug can be reproduced on the latest version, but I have not checked that myself.

Why the order is non-deterministic

On a loaded host with virtualization and RSS (Receive Side Scaling), the kernel distributes incoming packets across CPUs. The two connections land on different CPUs and complete their handshakes in parallel — in unpredictable order. The server's accept() returns whichever completed first.

Logs also show this behavior:

Server connections:

"connected":	[{
        "socket":	5,  // sender=false
        "local_host":	"2a02:6b8:c02:901:0:fca5:0:242",
        "local_port":	10000,
        "remote_host":	"2a02:6b8:c02:901:0:fca5:0:1d9",
        "remote_port":	49180
    }, {
        "socket":	8,  // sender=true
        "local_host":	"2a02:6b8:c02:901:0:fca5:0:242",
        "local_port":	10000,
        "remote_host":	"2a02:6b8:c02:901:0:fca5:0:1d9",
        "remote_port":	49178
    }],

Client connections:

"connected":	[{
        "socket":	5,  // sender=true
        "local_host":	"2a02:6b8:c02:901:0:fca5:0:1d9",
        "local_port":	49178,
        "remote_host":	"2a02:6b8:c02:901:0:fca5:0:242",
        "remote_port":	10000
    }, {
        "socket":	7,  // sender=false
        "local_host":	"2a02:6b8:c02:901:0:fca5:0:1d9",
        "local_port":	49180,
        "remote_host":	"2a02:6b8:c02:901:0:fca5:0:242",
        "remote_port":	10000
    }],

You can see that server sender connect with client sender, and server receiver with client receiver. This cause deadlock.

Server sender (port 10000)   <---> client sender (port 49178)
Server receiver (port 10000) <---> client receiver (port 49180)

Possible solution

The root cause is that role assignment relies on accept() ordering, which is non-deterministic under load. The fix is to make the role explicit: after sending the cookie, the client sends a 1-byte flag indicating whether this connection is a sender or receiver. The server reads this flag and assigns the opposite role (client sender → server receiver, client receiver → server sender).

This eliminates the race entirely — the server no longer needs to guess the role from accept order, but this approach break compatibility with previous versions of iperf3. I would love to hear your thoughts!

[davidBar-On]

On a loaded host with virtualization and RSS (Receive Side Scaling), the kernel distributes incoming packets across CPUs. The two connections land on different CPUs and complete their handshakes in parallel — in unpredictable order.
...
The root cause is that role assignment relies on accept() ordering, which is non-deterministic under load.

Great analysis of the problem. I agree with the conclusion that RSS is allows the problem and that the root cause is the accept() order.

The fix is to make the role explicit: after sending the cookie, ...

However, I believe that there is no need to make the role explicit. I think that the the client's issue is in iperf_tcp_connect():

iperf/src/iperf_tcp.c

Line 592 in 3a25192

if (connect(s, (struct sockaddr *) server_res->ai_addr, server_res->ai_addrlen) < 0 && errno != EINPROGRESS) {

The first connect() (sender) returns before it completes and errno is set to EINPROGRESS. Since in this case the process continues, the client will perform the second connect() (receiver), which will cause the server's accept() ordering issue..

The fix may be to replace the connect() in iperf_tcp_connect() with timeout_connect():

iperf/src/net.c

Lines 90 to 91 in 3a25192

	timeout_connect(int s, const struct sockaddr *name, socklen_t namelen,
	int timeout)

timeout may be some predefined constant, e.g. the 2 minutes DEFAULT_NO_MSG_RCVD_TIMEOUT.

[AlexMoshkov]

Greetings @davidBar-On, thanks for reply! I opened the PR with the solution you suggested:
#2030