MPTCP uses only 4 interfaces out of 8

[matthiasATzema]

Hi experts,

i got two servers equipped with 2 quadport 1Gb ethernet adapters and 2 1Gb ports onboard. (sum of 10 ports a 1Gb per server).

i want to use 2 ports as local LAN (no multipath) and 8 ports bundled as a 8Gb link between both servers.

To check the connection speed i started "iperf -s" at the first machine and "iperf -c 10.250.240.20" at the second machine. In combination with "nload" i can see that only 4 interfaces are being used at the same time. What can i do to use all 8 interfaces at the same time?

How can in debug this issue?

Many thanks in advance!

[GinesGarcia]

Hi,
are you using "fullmesh" path-manager? have you configure all the routing tables needed by MPTCP (one per interface)?if so, can you provide more information about your configuration?

[matthiasATzema]

Yes, i use the fullmesh path-manager.

My hardware setup looks like this:
serverA & serverB:
HP ProLiant SE326M1R2 2x Intel Xeon L5640 Six Core 2.26 GHz (12 cores total per server)
2x Intel(R) PRO/1000 e1000e (quad port gigabit, 8x 1Gb direct connection between serverA and serverB)
Linux serverA 3.18.25.jessiemptcp #3 SMP Wed Jan 13 05:33:31 UTC 2016 x86_64 GNU/Linux

This i configured manually on serverA:
ip rule add from 10.250.240.20 table 1
ip rule add from 10.250.241.20 table 2
ip rule add from 10.250.242.20 table 3
ip rule add from 10.250.243.20 table 4
ip rule add from 10.250.244.20 table 5
ip rule add from 10.250.245.20 table 6
ip rule add from 10.250.246.20 table 7
ip rule add from 10.250.247.20 table 8
ip route add 10.250.240.0/24 dev enp5s0f0 scope link table 1
ip route add 10.250.241.0/24 dev enp5s0f1 scope link table 2
ip route add 10.250.242.0/24 dev enp6s0f0 scope link table 3
ip route add 10.250.243.0/24 dev enp6s0f1 scope link table 4
ip route add 10.250.244.0/24 dev enp12s0f0 scope link table 5
ip route add 10.250.245.0/24 dev enp12s0f1 scope link table 6
ip route add 10.250.246.0/24 dev enp13s0f0 scope link table 7
ip route add 10.250.247.0/24 dev enp13s0f1 scope link table 8

This on serverB:
ip rule add from 10.250.240.21 table 1
ip rule add from 10.250.241.21 table 2
ip rule add from 10.250.242.21 table 3
ip rule add from 10.250.243.21 table 4
ip rule add from 10.250.244.21 table 5
ip rule add from 10.250.245.21 table 6
ip rule add from 10.250.246.21 table 7
ip rule add from 10.250.247.21 table 8
ip route add 10.250.240.0/24 dev enp5s0f0 scope link table 1
ip route add 10.250.241.0/24 dev enp5s0f1 scope link table 2
ip route add 10.250.242.0/24 dev enp6s0f0 scope link table 3
ip route add 10.250.243.0/24 dev enp6s0f1 scope link table 4
ip route add 10.250.244.0/24 dev enp12s0f0 scope link table 5
ip route add 10.250.245.0/24 dev enp12s0f1 scope link table 6
ip route add 10.250.246.0/24 dev enp13s0f0 scope link table 7
ip route add 10.250.247.0/24 dev enp13s0f1 scope link table 8

according to /var/log/kern.log at serverA i can say that a subflow will be created from each ip address to each ip address. So this looks like i "fullmesh". But maybe here is the problem, because serverA and serverB are direct connected via 8 ethernet cables, so most of this subflows will be deadends.

[cpaasch]

As MPTCP creates a fullmesh across all IP-addresses, and additionally to that the maximum number of subflows is limited to 32, it can happen that the host ends up not using all interfaces in your case.

Because, with each host having 8 interfaces, a total of 64 subflows would be created (if there were not limit on the number of subflows).

You should install iptables-rules with target REJECT to prevent these unnecessary subflows.

[matthiasATzema]

My boss cancled the project so sadly i dont have any time left to test this. :(

serverA
iptables -A INPUT -s 10.250.240.20 -d 10.250.240.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.240.20 -j REJECT
iptables -A INPUT -s 10.250.241.20 -d 10.250.241.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.241.20 -j REJECT
iptables -A INPUT -s 10.250.242.20 -d 10.250.242.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.242.20 -j REJECT
iptables -A INPUT -s 10.250.243.20 -d 10.250.243.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.243.20 -j REJECT
iptables -A INPUT -s 10.250.244.20 -d 10.250.244.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.244.20 -j REJECT
iptables -A INPUT -s 10.250.245.20 -d 10.250.245.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.245.20 -j REJECT
iptables -A INPUT -s 10.250.246.20 -d 10.250.246.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.246.20 -j REJECT
iptables -A INPUT -s 10.250.247.20 -d 10.250.247.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.247.20 -j REJECT

serverB
iptables -A INPUT -s 10.250.240.21 -d 10.250.240.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.240.21 -j REJECT
iptables -A INPUT -s 10.250.241.21 -d 10.250.241.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.241.21 -j REJECT
iptables -A INPUT -s 10.250.242.21 -d 10.250.242.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.242.21 -j REJECT
iptables -A INPUT -s 10.250.243.21 -d 10.250.243.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.243.21 -j REJECT
iptables -A INPUT -s 10.250.244.21 -d 10.250.244.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.244.21 -j REJECT
iptables -A INPUT -s 10.250.245.21 -d 10.250.245.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.245.21 -j REJECT
iptables -A INPUT -s 10.250.246.21 -d 10.250.246.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.246.21 -j REJECT
iptables -A INPUT -s 10.250.247.21 -d 10.250.247.0/24 -j ACCEPT
iptables -A INPUT -s 10.250.247.21 -j REJECT