Server stops accepting clients: Too many open files

[bbczeuz]

Context

• Version of iperf3:
  iperf 3-CURRENT (cJSON 1.5.2)
  (same issue on git branch STABLE 3.2 and MASTER)
  (I minimally enhanced debugging output from commit 46cb4b4)

• Hardware:
  RHEL7 Linux VM on KVM

• Operating system (and distribution, if any):
  RHEL7 Linux VM on KVM

Bug Report

We're using iperf3 to perform scheduled TCP BW checks from 6 nodes to a central server (every 10m for 10s per probe)

• Expected Behavior
  iperf3 server continues to accept clients forever

• Actual Behavior
  iperf3 server stops accepting clients after a few hours/days. New clients trigger the following msg (server side log):

### Error IESTREAMCONNECT on accept:iperf3: error - unable to connect stream: Too many open files

(I added the first words to easier detect the source of the msg. It happens at src/iperf_tcp.c#L111, function iperf_tcp_accept(), return from accept() syscall)

The message seems valid:
(iperf3 -s -D -p 1234)

# lsof | grep iperf3 | wc -l       
1044
# netstat -n | grep 1234
[...]
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29839     TIME_WAIT  
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29770     TIME_WAIT  
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29857     TIME_WAIT  
tcp6       1      0 10.0.0.107:1234      10.0.1.213:62330     CLOSE_WAIT 
tcp6       1      0 10.0.0.107:1234      10.0.1.213:62331     CLOSE_WAIT 
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29849     TIME_WAIT  
tcp6       1      0 10.0.0.107:1234      10.0.1.213:62332     CLOSE_WAIT 
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29775     TIME_WAIT  
tcp6       0      0 10.0.0.107:1234      10.0.1.182:29786     TIME_WAIT  
[...]

• Steps to Reproduce
  • start iperf3 server daemon: iperf3 -s -D -p 1234
  • use 6 clients to connect to it. Every client connects once every 10m and performs 10s of measurement. If a client gets denied, it waits for a little more than 10s and tries again (up to 3m, then it gives up)
  • The clients disconnect from the network just after the test, so at least in theory, it could happen that the FIN gets lost.
  • issue typically happens after a few hours to a few days.

[bbczeuz]

The TIME_WAIT/CLOSE_WAIT seems not to be the issue as:

# netstat -n | grep 1234 | wc -l
66

It seems more that the socket handles just get lost:

# lsof | grep 24561 | wc -l
1044
# lsof
COMMAND     PID  TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
[...]
iperf3    24561                root 1012u     sock                0,7       0t0    1531130 protocol: TCPv6
iperf3    24561                root 1013u     sock                0,7       0t0    1531152 protocol: TCPv6
iperf3    24561                root 1014u     sock                0,7       0t0    1531132 protocol: TCPv6
iperf3    24561                root 1015u     sock                0,7       0t0    1531161 protocol: TCPv6
iperf3    24561                root 1016u     sock                0,7       0t0    1531134 protocol: TCPv6
iperf3    24561                root 1017u     sock                0,7       0t0    1531154 protocol: TCPv6
iperf3    24561                root 1018u     sock                0,7       0t0    1531136 protocol: TCPv6
iperf3    24561                root 1019u     sock                0,7       0t0    1531168 protocol: TCPv6
iperf3    24561                root 1020u     sock                0,7       0t0    1531138 protocol: TCPv6
iperf3    24561                root 1021u     sock                0,7       0t0    1531156 protocol: TCPv6
iperf3    24561                root 1022u     sock                0,7       0t0    1531140 protocol: TCPv6
iperf3    24561                root 1023u     sock                0,7       0t0    1531163 protocol: TCPv6

[bbczeuz]

• Hint: Clients are using 8 parallel TCP streams: Uplink iperf -c 10.0.0.107 -p 1234 -P8 -o 5 and downlink iperf -c 10.0.0.107 -p 1234 -P8 -R -o 5

• Workaround: cronjob every 1h: systemctl restart iperf3@1234 and kill client processes after a determined duration.

[bbczeuz]

Does anyone have success in reliably running an iperf3 server for a long time? The workaround I proposed keeps the server alive, but confuses the clients, so now they're stuck... (select() in iperf_run_client() ) Clients use iperf3 from Debian9 repo.

[bmah888]

Lots of good information here, thanks for an informative bug report. I agree it feels like something's wrong here although I have not yet had the time to try to recreate it or investigate.

On a related note, we at ESnet (as well as many others in the R&E space) use the perfSONAR tools pScheduler (preferred) or bwctl (deprecated) to manage iperf3 tests where multiple hosts might be wanting to run tests at the same time. In that setup, iperf3 server processes are forked on demand, and run exactly one test before being killed. That might be helpful for your use case, although I would not say say that's a solution to the problem you're seeing (i.e. there's still probably a bug that should be fixed).

[samtan6777]

Hi bbczeuz,
I have concur the issue does exist. I run just 2 clients to single server. However, I using a faster timing like each client run for 60 seconds and pause 5 seconds, and loop.
The server stopped and terminated with the same error just after slightly 3 hours.

Trying to do some research how to do a work around, if any.

[davidBar-On]

Hi, PR #1084 that was merged recently fix a problem that the prot-listener socket was not always closed at the end of a session. Since such problem may cause the too many files open problem, I suggest to to test with this fix.