GbE optimisations

[ThomasKaiser]

Tweaking RX/TX delay is an important step in the right direction. But assigning Ethernet IRQs to a dedicated CPU core might also be a good idea and also tweaking some tunables:

USB=usb2 ; MMC=sunxi-mmc ; GbE=1c30000.eth
echo 2 >/proc/irq/$(awk -F":" "/${USB}/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity
echo 4 >/proc/irq/$(awk -F":" "/${MMC}/ {print \$1}" </proc/interrupts | head -n1 | sed 's/\ //g')/smp_affinity
echo 8 >/proc/irq/$(awk -F":" "/${GbE}/ {print \$1}" </proc/interrupts | sed 's/\ //g')/smp_affinity

sysctl -w net/core/rmem_max=8738000
sysctl -w net/core/wmem_max=6553600
sysctl -w net/ipv4/tcp_rmem="8192 873800 8738000"
sysctl -w net/ipv4/tcp_wmem="4096 655360 6553600"
sysctl -w vm/min_free_kbytes=65536
ip link set eth0 txqueuelen 10000

This is OS X to Pine64:

-----------------------------------------------------------
Accepted connection from 192.168.83.119, port 64508
[  5] local 192.168.83.88 port 5201 connected to 192.168.83.119 port 64509
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   108 MBytes   901 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   931 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   932 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec                  
[  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   938 Mbits/sec                  
[  5]  10.00-10.01  sec   604 KBytes   937 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.01  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.01  sec  1.09 GBytes   934 Mbits/sec                  receiver

In the other direction currently only ~430 Mbits/sec :(

[ThomasKaiser]

And a small note: In the 1st released A64 BSP there was written

                    tx-delay = <7>;
                    rx-delay = <31>;

so obviously more combinations are possible now than with A20 back then. I wrote a script for A20 some time ago to create u-boot .debs with all possible RX/TX combinations (back then 8*8) and then set up an automated test environment (start, ping other machine, run iperf, replace u-boot BLOB, restart, ...)

Would this be interesting?

[longsleep]

Yes, though the speeds also depend on the other end. I played around with these values already and only very low values make 1000M work.

[ThomasKaiser]

Ok, good to know that you already tried different values out. From my understanding this is board specific so 'generic' values taken from Allwinner are worthless. But since I would believe they get their Android builds from there the engineers might already tried to look into this?

And yes, performance always depends on the other end. Am doing network testing for a living with a MacBook known to exceed 940 Mbits/sec (but good point: I exchanged the switch in the office -- will try again)

EDIT: Nope: different switch, same numbers

[longsleep]

I think

            tx-delay = <0x00000003>;
            rx-delay = <0x00000000>;

is the best and works stable on all switches i could find. Keeping this values until someone proposes better ones.