FS#3457 - ipq40xx: NBG6617/B-1300 using VLANs is incredibly slow (<10Mbit/s

[openwrt-bot]

spolack:

The listed config below is running fine on 19.07 till 19.07.3. From 19.07.4 on the throughput to peers on the tagged VLANs falls below 10Mbit/s. Seems there is some regression introduced on 19.07.4.

Sample config:

config interface 'ff_dhcp'
option proto 'static'
option ifname 'eth0.30'
option ipaddr '10.36.164.65/27'

config interface 'mgmt'
option proto 'static'
option ifname 'eth0.42'
option ipaddr '10.36.190.177/28'

config switch_vlan
option device 'switch0'
option vlan '30'
option ports '0t 3 4'

config switch_vlan
option device 'switch0'
option vlan '42'
option ports '0t 3t 4t'

* Note: As essedma has some bugs i cannot use switchport, which is hardcoded to VLAN2. But i'm fine with just leaving it as it is.

[openwrt-bot]

johnfzc:

I'm also affected by this issue. The issue appears to be with the reception of tagged traffic on the interface. I wonder if this may be related to this [[https://forum.openwrt.org/t/ipq40xx-low-performance-when-sending-traffic/74520/7|forum discussion]] but the file that PolynomialDivision modified doesn't appear to exist in the 19.07.4 tagged release.

Here are some iperf3 TCP tests receiving traffic, showing the impacted performance on 19.07.4 and close to line rate with 18.06.5.

Accepted connection from 172.16.2.1, port 58658 [ 5] local 172.16.2.20 port 5201 connected to 172.16.2.1 port 58660 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 130 KBytes 1.07 Mbits/sec [ 5] 1.00-2.00 sec 667 KBytes 5.47 Mbits/sec [ 5] 2.00-3.00 sec 1.16 MBytes 9.72 Mbits/sec [ 5] 3.00-4.00 sec 233 KBytes 1.91 Mbits/sec [ 5] 4.00-5.00 sec 214 KBytes 1.75 Mbits/sec [ 5] 5.00-6.00 sec 1.30 MBytes 10.9 Mbits/sec [ 5] 6.00-7.00 sec 392 KBytes 3.21 Mbits/sec [ 5] 7.00-8.00 sec 452 KBytes 3.71 Mbits/sec [ 5] 8.00-9.00 sec 624 KBytes 5.11 Mbits/sec [ 5] 9.00-10.00 sec 638 KBytes 5.22 Mbits/sec [ 5] 10.00-10.06 sec 21.2 KBytes 3.04 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.06 sec 5.75 MBytes 4.80 Mbits/sec receiver

root@hydrogen:~# iperf3 -c 172.16.2.20 -R
Connecting to host 172.16.2.20, port 5201
Reverse mode, remote host 172.16.2.20 is sending
[ 5] local 172.16.2.1 port 41928 connected to 172.16.2.20 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 109 MBytes 917 Mbits/sec
[ 5] 1.00-2.00 sec 110 MBytes 924 Mbits/sec
[ 5] 2.00-3.00 sec 113 MBytes 945 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 939 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec
[ 5] 5.00-6.00 sec 109 MBytes 911 Mbits/sec
[ 5] 6.00-7.00 sec 111 MBytes 930 Mbits/sec
[ 5] 7.00-8.00 sec 104 MBytes 876 Mbits/sec
[ 5] 8.00-9.00 sec 103 MBytes 864 Mbits/sec
[ 5] 9.00-10.00 sec 87.8 MBytes 736 Mbits/sec

---

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.04 sec 1.05 GBytes 897 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.05 GBytes 898 Mbits/sec receiver


Here are UDP tests first with 19.07.4 and then 18.06.5. The drop rate is a bit high with the UDP compared with the TCP throughput but I assume that the TCP algorithm does a better job of pacing the packets than iperf in UDP mode.

root@hydrogen:~# iperf3 -b 900M -u -c 172.16.2.20 -R Connecting to host 172.16.2.20, port 5201 Reverse mode, remote host 172.16.2.20 is sending [ 5] local 172.16.2.1 port 45202 connected to 172.16.2.20 port 5201 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 17.0 MBytes 142 Mbits/sec 0.273 ms 68461/80747 (85%) [ 5] 1.00-2.00 sec 49.4 MBytes 414 Mbits/sec 0.200 ms 41775/77551 (54%) [ 5] 2.00-3.00 sec 10.7 MBytes 89.8 Mbits/sec 0.307 ms 69805/77554 (90%) [ 5] 3.00-4.00 sec 50.3 MBytes 422 Mbits/sec 0.010 ms 41881/78278 (54%) [ 5] 4.00-5.00 sec 28.4 MBytes 238 Mbits/sec 0.022 ms 57140/77728 (74%) [ 5] 5.00-6.00 sec 61.6 MBytes 517 Mbits/sec 0.013 ms 33067/77702 (43%) [ 5] 6.00-7.00 sec 30.2 MBytes 253 Mbits/sec 0.325 ms 55186/77031 (72%) [ 5] 7.00-8.00 sec 54.1 MBytes 454 Mbits/sec 0.017 ms 39168/78356 (50%) [ 5] 8.00-9.00 sec 63.7 MBytes 535 Mbits/sec 0.019 ms 31518/77671 (41%) [ 5] 9.00-10.00 sec 64.0 MBytes 537 Mbits/sec 0.018 ms 31354/77673 (40%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.05 sec 1.05 GBytes 900 Mbits/sec 0.000 ms 0/780455 (0%) sender [ 5] 0.00-10.00 sec 429 MBytes 360 Mbits/sec 0.018 ms 469355/780291 (60%) receiver

root@hydrogen:~# iperf3 -b 900M -u -c 172.16.2.20 -R
Connecting to host 172.16.2.20, port 5201
Reverse mode, remote host 172.16.2.20 is sending
[ 5] local 172.16.2.1 port 44577 connected to 172.16.2.20 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 14.7 MBytes 123 Mbits/sec 0.190 ms 69885/80500 (87%)
[ 5] 1.00-2.00 sec 77.4 MBytes 649 Mbits/sec 0.013 ms 22079/78097 (28%)
[ 5] 2.00-3.00 sec 80.0 MBytes 671 Mbits/sec 0.007 ms 19754/77678 (25%)
[ 5] 3.00-4.00 sec 80.1 MBytes 672 Mbits/sec 0.009 ms 19702/77716 (25%)
[ 5] 4.00-5.00 sec 79.9 MBytes 670 Mbits/sec 0.011 ms 19863/77693 (26%)
[ 5] 5.00-6.00 sec 80.2 MBytes 672 Mbits/sec 0.013 ms 19649/77698 (25%)
[ 5] 6.00-7.00 sec 79.9 MBytes 670 Mbits/sec 0.009 ms 19791/77667 (25%)
[ 5] 7.00-8.00 sec 80.0 MBytes 671 Mbits/sec 0.009 ms 19788/77707 (25%)
[ 5] 8.00-9.00 sec 80.1 MBytes 672 Mbits/sec 0.009 ms 19713/77700 (25%)
[ 5] 9.00-10.00 sec 80.1 MBytes 672 Mbits/sec 0.006 ms 19692/77677 (25%)

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.04 sec 1.05 GBytes 900 Mbits/sec 0.000 ms 0/780249 (0%) sender
[ 5] 0.00-10.00 sec 732 MBytes 614 Mbits/sec 0.006 ms 249916/780133 (32%) receiver


[openwrt-bot]

johnfzc:

This [[https://github.com/openwrt/openwrt/commit/e73d866c6fc7dfad2fb7dd87345c90029bbda59d|commit]] is related, but shouldn't affect 19.07.4 as far as I can tell.

[openwrt-bot]

PolynomialDivision:

Please read my commit message and test if u can:
#3596

This should fix the issue. ;)

[openwrt-bot]

johnfzc:

Thanks for that Polynomdivision, but I don't believe that patch can be applied against the [[https://github.com/openwrt/openwrt/tree/openwrt-19.07|19.07]] tree; there's no edma_axi.c file.

[openwrt-bot]

rotanid:

John, as you can see on Github someone already created a 19.07 branch with adjusted patches.

[openwrt-bot]

johnfzc:

I looked on github and didn't see it, can you please provide us with a link to the commit, PR or branch you are talking about rotanid?

[openwrt-bot]

rotanid:

#3596 (comment)