Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UDP packets on dual-homed iperf3 server go out the wrong interface #637

Open
afpd opened this issue Sep 11, 2017 · 10 comments

Comments

@afpd
Copy link

commented Sep 11, 2017

Context

iperf3 3.1.3
Debian Linux 9.1 x86_64
Server has multiple IPs 192.168.112.61 and 192.168.93.141

Bug Report

Server replies with the first IPv4, while should with the one requested
Sample tcpdump:
11:44:49.307320 IP 192.168.93.141.5201 > 10.78.243.11.58002: Flags [P.], seq 2:3, ack 184, win 59, options [nop,nop,TS val 1013109869 ecr 3431431813], length 1
11:44:49.538440 IP 10.78.243.11.54513 > 192.168.93.141.5201: UDP, length 4
11:44:49.538593 IP 192.168.112.61.5201 > 10.78.243.11.54513: UDP, length 4

Server has IPs 192.168.93.141 and 192.168.112.61
Client IP 10.78.243.11

Client is running
iperf3 -c 192.168.93.141 -u -b150m -w32M -t60 --get-server-output
Session hangs, since UDP reply goes from 192.168.112.61

Expected behavior:
Server replies from 192.168.93.141

@dezenxi

This comment has been minimized.

Copy link

commented Sep 26, 2017

A possible way is to configure route to client (10.78.243.11) on server preferred source ip 192.168.93.141.
Try this. https://superuser.com/questions/376667/how-to-route-only-specific-subnet-source-ip-to-a-particular-interface

@bmah888 bmah888 added the bug label Sep 28, 2017

@bmah888

This comment has been minimized.

Copy link
Member

commented Sep 28, 2017

I had to draw myself a picture to see what's going on. But it appears the problem is that outbound UDP traffic from the server isn't going out through the same interface that it came in.

The OP didn't say anything about how routing is set up, but something along the lines of what @dezenxi suggested might be helpful as a workaround.

I'd like to understand why UDP packets went out a different interface. I haven't looked at the code yet, but this does seem wrong or at least non-intuitive. I'm going to say this is a real bug, at least for now.

@afpd: TCP tests work for you, correct? Can you show me the output if netstat -rn on the server so I can see the kernel routing table? Also note that you're running an old version of iperf3. 3.2 is current as of this writing.

@bmah888 bmah888 changed the title IPerf3 reply IP for UDP UDP packets on dual-homed iperf3 server go out the wrong interface Sep 28, 2017

@afpd

This comment has been minimized.

Copy link
Author

commented Dec 2, 2017

Server has one NIC, although lo (loopback) has 192.168.93.141/32, while eth0 has 192.168.112.61/24.

@bmah888

This comment has been minimized.

Copy link
Member

commented Dec 4, 2017

@afpd: Thanks for the additional information, although having answers to the rest of my questions would be useful too.

@ks228

This comment has been minimized.

Copy link

commented Jan 12, 2018

Is this bug resolved in the latest version? For some reason, I can't install the latest right now. But information on whether it is fixed or not would be helpful. Thanks

@bmah888

This comment has been minimized.

Copy link
Member

commented Jan 12, 2018

@ks228: No coding changes have been done towards fixing this issue. Indeed at this point I'm not sure exactly what the issue is, or whether it's a bug in iperf3 or in the OP's setup. Still waiting on answers to some of the follow-up questions I asked.

@ks228

This comment has been minimized.

Copy link

commented Jan 12, 2018

@bmah888 I actually tested it on my setup just a few minutes ago on versions 3.1.3 and the latest 3.3. It is an issue in my setup as well. My setup is simple. I have multiple addresses configured on my server's NIC. I second that it only happens with UDP. TCP works fine here as well.
There is this thread as well but that solution doesn't work #371 (comment) at least in my simple setup.

@rnwolfe

This comment has been minimized.

Copy link

commented Jan 23, 2018

I'm seeing this issue, as well.

On server

pi@wolfepi:~ $ iperf3 -s -V
iperf 3.0.7
Linux wolfepi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Tue, 23 Jan 2018 04:33:24 GMT
Accepted connection from 208.44.206.101, port 45776
      Cookie: jumpbox.1516682001.727054.3c8ab9de6a
[  5] local 10.2.0.70 port 5201 connected to 208.44.206.101 port 54888
Starting Test: protocol: UDP, 1 streams, 8192 byte blocks, omitting 0 seconds, 10 second test

Server hangs until I Ctrl-C

^C[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-542.39 sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (nan%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-542.39 sec  0.00 Bytes  0.00 bits/sec  0.000 ms  0/0 (nan%)  
CPU Utilization: local/receiver 0.0% (0.0%u/0.0%s), remote/sender 0.0% (0.0%u/0.0%s)
iperf3: interrupt - the server has terminated

On client

C:\Users\rwolfe\Downloads\iperf-3.1.3-win64>iperf3.exe -c xxxxx-iperf.ddns.net -
u -b 500000000
Connecting to host wolfe-iperf.ddns.net, port 5201

Client hangs until I Ctrl-C

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datag
rams
iperf3: interrupt - the client has terminated

Note that the client has terminated only results when I Ctrl-C the process.

More Info

Server's NIC Info

pi@wolfepi:~ $ ifconfig
eth0      Link encap:Ethernet  HWaddr b8:27:eb:0e:cf:10  
          inet addr:10.2.0.70  Bcast:10.2.0.255  Mask:255.255.255.0
          inet6 addr: fe80::470c:ff13:a6d0:afd8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:103276 errors:0 dropped:1 overruns:0 frame:0
          TX packets:56242 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:133280154 (127.1 MiB)  TX bytes:9376483 (8.9 MiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:200 errors:0 dropped:0 overruns:0 frame:0
          TX packets:200 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:16656 (16.2 KiB)  TX bytes:16656 (16.2 KiB)

wlan0     Link encap:Ethernet  HWaddr b8:27:eb:5b:9a:45  
          inet addr:10.2.0.71  Bcast:10.2.0.255  Mask:255.255.255.0
          inet6 addr: fe80::3c8f:1bd3:dad0:9802/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:586 errors:0 dropped:0 overruns:0 frame:0
          TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:106403 (103.9 KiB)  TX bytes:16860 (16.4 KiB)

Server's routing table

pi@wolfepi:~ $ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.2.0.1        0.0.0.0         UG        0 0          0 eth0
0.0.0.0         10.2.0.1        0.0.0.0         UG        0 0          0 wlan0
10.2.0.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.2.0.0        0.0.0.0         255.255.255.0   U         0 0          0 wlan0

The initial connection comes in through the wlan0 interface, but the UDP responses seem to go out of the eth0 interface which then hangs.

For what it's worth, TCP does work. Both sides are behind a firewall with NAT, and I own the network on both sides, so I can do some tweaking as needed.

Let me know if I can test anything out for you, @bmah888. I'm happy to help get this resolved.

@rnwolfe

This comment has been minimized.

Copy link

commented Jan 23, 2018

After writing my above comment, I realized that I was accidentally sending the iperf requests to the wlan0 interface. This was accidental. I fixed it to request to the eth0 interface and UDP began working. Presumably because the return flow was going back out of the same interface.

It doesn't necessarily fix the issue; however, it does seem to suggest that the OS is prioritizing NICs and could be impacting iperf UDP testing since the UDP test actually receives a request to the server, and then starts a new UDP stream to the client.

Somewhat similar to how Active mode FTP opens up a new data channel after the control channel is established.

Hopefully this sheds some light, but, regardless, let me know if I can test anything out for you.

@hkominos

This comment has been minimized.

Copy link

commented Jun 18, 2019

Workaround:

I have a single linux virtual NIC with multiple Ips assigned to it.
Iperf3 does not work when u try to generate traffic to multiple IPs on that NIC.
BUT you can start servers while binding those processes to the IP that you want and that seems to work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.