STM32H7 TCP server speed up

[IrekRM]

Hello! I need to send 32 KB from the NUCLEO-H743ZI2 board to a computer as quickly as possible. The Nucleo board is connected to a dedicated Ethernet adapter on the computer, so there’s no interference from the global Internet, and in theory I can use the full 100 Mbps speed. On the board the TCP server is running based on the MBed OS. The computer connects to it as a client, and when the data on the NUCLEO board is ready, it is sent from the server to the client. I send data buffer in a next manner:

client_sock[clnt]->set_blocking(true);
int scount = client_sock[clnt]->send(buf, len);
client_sock[clnt]->set_blocking(false);

With MBed 6.xx to speed up transmissions I used some increase of LWIP parameters
"lwip.tcp-mss": 2144, "lwip.tcpip-thread-stacksize": 9600, "lwip.mem-size": 10624
This sped up the transfer, but the connection was still unstable and the server would freeze frequently.

In the release notes of MBed CE 7.0 there is very attractive sentence: "Ethernet drivers for all STM32 devices completely rewritten from scratch to avoid the troubled HAL drivers, perform better, properly support multicast, and be compatible with the CPU cache."

So I’d like to know how the MBed CE experts would recommend handling this kind of relatively large and fast data transfer over TCP?

Currently, the default value of MSS is 536 bytes. Do I need to split my data into smaller packets and send them one by one?
Does my data buffer need to be aligned to any specific addresses?
Thank you in advance for your advice!

[multiplemonomials]

Hi! I'm not an expert with TCP tuning, but I did do some testing. Regarding the MSS, you might get better performance increasing it to around 1400, but I wouldn't recommend increasing it past there, as then you will run into the maximum size of an IP packet and the IP stack will start having to fragment your TCP packets. In my experience, the single most important variable is the TCP window size, lwip.tcp-wnd. Making that larger will allow more data to be "in flight" at the same time, but if you make it too big, data sent by the other end will be too big for the receive buffer space on the device and packets will be dropped.

Anyway, yeah I would expect network operations to be faster with the new driver, but it does depend a bit on your application. I'd love if you could give it a try, both with the default settings and with "lwip.tcp-mss": 1400, and let me know what the speed looks like!

[multiplemonomials]

btw, the primary reason that the old driver was slow was that it was completely synchronous: even though ethernet is designed to send packets in the background, the old H7 driver would block the CPU until the packet was completely finished sending. It was a massive waste of CPU time, so you should at least see a significant speedup from that.

[IrekRM]

Thank you for your response, Jamie!
After reading your response and lwIP Wiki https://lwip.fandom.com/wiki/LwIP_Wiki I played with lwip parameters in the mbed_app.json5 file a little bit. The best parameters set for now looks next:
"lwip.tcp-mss": 1460,
"lwip.tcp-wnd": 2920,
"lwip.tcp-snd-buf": 2920,
Unfortunately after any change in the mbed_app.json5 file the project is recompiling fully and this takes time. But visually, data transfer appears to be faster with these settings than with the default settings.

I decided to measure transfer time. I trigger the function rx_srq_func once per second. This function send two 32KB buffers (buf_re and buf_im) one after another and measure time which it take. The code looks like this:
`Timer rx_t;
int rx_tus;

rx_t.reset();
rx_t.start();

client_sock[clnt]->set_blocking(true);
client_sock[clnt]->send(buf_re, len);
client_sock[clnt]->set_blocking(false);

rx_t.stop();
rx_tus = rx_t.elapsed_time().count();
printf("Sent buf_re in %d us\r\n", rx_tus);
`
And I see the output in the putty:

rx_srq_func starts for 1 time:
Sent buf_re in 7934 us
Sent buf_im in 47462 us
rx_srq_func starts for 2 time:
Sent buf_re in 6229 us
Sent buf_im in 1290548 us
rx_srq_func starts for 3 time:
Sent buf_re in 5223 us
Sent buf_im in 53754 us
rx_srq_func starts for 4 time:
Sent buf_re in 6939 us
Sent buf_im in 45825 us
rx_srq_func starts for 5 time:
Sent buf_re in 1354880 us
Sent buf_im in 55484 us
rx_srq_func starts for 6 time:
Sent buf_re in 377104 us
Sent buf_im in 59182 us

Data arrives PC correctly, but as you can see the transfer time is varying a lot! From 5-6 ms to few seconds!
Do you have any ideas or suggestions about where I might be going wrong?
Thank you in adavance!

[multiplemonomials]

So it looks like the first packet was sent in sequence numbers 5 through 38, in about 4.5ms (wireshark timestamps agree with what you recorded). That means you're getting a bandwidth of about 60Mbps, which is pretty reasonable (though still some room for improvement). I suspect you are limited here by the TCP send buf (lwip.tcp-snd-buf), which defaults to 2 * max segment size (odd choice of default, that). I think this is only allowing LwIP to buffer two packets' worth of data, meaning that it can't send another one until it gets an ACK from the computer. You can see in the wireshark trace that there's always a pattern of two data packets, then an ACK, and I suspect this is why.

[multiplemonomials]

The second packet looks to be in sequence numbers 39 through 73, with a time of 57.44 ms (pretty similar to what you got). The odd part happened at packet #41, where LwIP sent a short packet (806 bytes instead of the max segment size), then just sat there for 52ms without doing anything. Only when the host PC sent an ACK (packet #42) did anything happen. I suspect this happened because the preceding ACK (packet #37) got lost somehow, leaving it with only 2266 bytes (1460 + 806) of send buffer space. That explains the short packet.

Normally a lost ACK like this would trigger a retransmission, but the problem is that LwIP's TCP timers are so slow that the situation ended up resolving itself before any retransmissions were made. Also seems like there may be some sort of HW or driver issue in play here causing intermittent packet drops. I have not seen this myself on my own H7 boards but at least 1 other person has reported something like this (disabling the D-cache fixed it for him).

So, for now, my recommendations are:

• Set lwip.tcp-snd-buf to a large value like 32768 (as long as you have the RAM). This will allow LwIP to have way more data "in flight" at the same time, improving bandwidth and making you more resilient against dropped packets.
• Add the following to the top level of mbed_app.json5:

"macros": [
    // Decrease TCP fast timer interval to 2ms for faster TCP retries
    "TCP_TMR_INTERVAL=2"
  ]

I've used this workaround in a past project, and it basically allows TCP retries to happen in ~10 ms rather than the default of like 1.5 seconds. This will add way more resilience against packet drops, driver issue or no.

[IrekRM]

Good morning Jamie,
Thank you very much for your detailed explanations! Very clear and easy to understand.

I tried to modify the mbed_app.json5 file. But project compilation fault down with pointing out to this string in lwip_init.c file:

#if !MEMP_MEM_MALLOC && (MEMP_NUM_TCP_SEG < TCP_SND_QUEUELEN)
#error "lwip_sanity_check: WARNING: MEMP_NUM_TCP_SEG should be at least as big as TCP_SND_QUEUELEN. If you know what you are doing, define LWIP_DISABLE_TCP_SANITY_CHECKS to 1 to disable this error."
#endif

The definitions are next:

#define MEMP_NUM_TCP_SEG            MBED_CONF_LWIP_MEMP_NUM_TCP_SEG

#define MBED_CONF_LWIP_MEMP_NUM_TCP_SEG 16

and

#define TCP_SND_QUEUELEN                ((4 * (TCP_SND_BUF) + (TCP_MSS - 1))/(TCP_MSS))

which is for TCP_SND_BUF=32768 and TCP_MSS=1460 equal 90

So is it possible to increase MBED_CONF_LWIP_MEMP_NUM_TCP_SEG ?
If yes, than how it it better to do without editing lwip_init.c ?

[IrekRM]

I tried to increase TCP_SND_BUF to the maximum value 5474 for TCP_MSS=1460 and MBED_CONF_LWIP_MEMP_NUM_TCP_SEG 16.
After that, H743 started to freeze when attempting to send the buffer.
So I decided to increase other two parameters:

            "lwip.tcpip-thread-stacksize": 32768,
            "lwip.mem-size": 64568

As result now with TCP_SND_BUF = 5474 transmission working, but transmission time increased:

rx_srq_func starts for 1 time:
Sent buf_re in 51669 us
Sent buf_im in 5247 us
rx_srq_func starts for 2 time:
Sent buf_re in 32440 us
Sent buf_im in 37099 us
QSPI read buffers in 2790 us
Sent buf_re in 50392 us
Sent buf_im in 38326 us

Here is corresponding Wireshark capture file
wcapture2.pcapng.gz

I see some TCP Retransmission events. What is wrong in my settings?

[multiplemonomials]

Agh yeah tuning these settings can be annoying because some have to be increased together. In this case, MBED_CONF_LWIP_MEMP_NUM_TCP_SEG maps to the lwip.memp-num-tcp-seg JSON config option (this is a standard relationship for any compile definition that starts with MBED_CONF_xx).

I suspect we are seeing retransmissions now because you increased the TCP timer rate, so the packet drops are now actually clearly showing up in the data instead of being masked by the long timer.

Could you try disabling the D-Cache as mentioned in that PR I linked earlier? Just put this as the first line in main():

SCB_DisableDCache();

Someone else reported that this fixed a similar issue for them, curious if it will work here.

[IrekRM]

Unfortunately disabling DCache doesn't help.
But at some moment I increased the TCP_TMR_INTERVAL to 10 and now I see something like this:

rx_srq_func starts for 1 time:
Sent buf_re in 4254 us
Sent buf_im in 4479 us
rx_srq_func starts for 2 time:
Sent buf_re in 103514 us
Sent buf_im in 4611 us

As you can see, sometimes both buffers are transmitted normally within 4.5 ms (very quickly!), but very often the first buffer now takes 50–100 ms to send. But transfers lasting 1 second or longer are now very rare.
So we are not far from working set of parameters! Tahnk you for your help!
Have a good weekend!

[IrekRM]

Hello Jamie,
Sorry for bothering you, but I spent some time playing with lwip parameters and still I do not understand some details.
May be my observations will help improve Mbed-CE.

The best set of parameters which I found is next:

    "macros": [
        // Decrease TCP fast timer interval to 2ms for faster TCP retries
        "TCP_TMR_INTERVAL=2"
    ],
    "target_overrides": {
        "*": {
            "lwip.socket-max": 4,
            "lwip.tcp-server-max": 4,
            "lwip.tcp-socket-max": 4,
            "lwip.memp-num-tcp-seg": 32,
            "lwip.tcp-mss": 1460,
            "lwip.tcp-wnd": 7300,
            "lwip.tcp-snd-buf": 7300,
            "lwip.tcpip-thread-stacksize": 32768,
            "lwip.mem-size": 65536

As result I have this type messages of my buffers transmission:

rx_srq_func starts for 1 time:
Sent buf_re in 14045 us
Sent buf_im in 3750 us
rx_srq_func starts for 2 time:
Sent buf_re in 14780 us
Sent buf_im in 3325 us

Here is corresponding wireshark capture file
wcapture3.pcapng.gz
The first buffer almost always takes longer to transfer than the second.

What I do not understand:

1. I set "lwip.memp-num-tcp-seg": 32, but I see that STM board sends one or two packets with Len=1460 and wait ACK packet back.
   See, e.g. sequence numbers 13-14. Why did that happen?
2. I am sending 4*8192+6=32774 bytes. It should be splitted to 22 packets of Len=1460 (MSS value) and last packet should be 654 bytes. But now I see the last packet with Len=1318 bytes - See, e.g. sequence numbers 81-82 or 153-154. Any idea what this mean?
   Just almost each time this packet need to be retransmitted.

Few other remarks:

• with disabled D-Cache the shortest transmission time increasing to ~4.5 ms.
• I tried to set MSS to larger values than 1460. The script modify the mbed-target-config.h file and set my value to MBED_CONF_LWIP_TCP_MSS, but after project compilation I see in wireshark that the packets size from the board still 1460 bytes. It seems that somewhere inside mbed-os my value is changing to the 1460 bytes.

Thank you in advance for any suggestions!

[multiplemonomials]

Ah yeah regarding the max segment size, you are going to be limited here by the Ethernet MTU of 1500 bytes. Anything over 1460 isn't going to work without significant Ethernet driver changes (enabling the Jumbo Frames extension, which is only available on some hardware).

[IrekRM]

Hello Jamie,
After doing a bit of googling, I realised that the MSS size is determined by the MTU size - Maximum-Transmission-Unit.
I tried to find out how it’s set in LWIP – and found a few places in the code and it seems it is hard-coded: MTU=1500 bytes.
So I give in. At least one of the two data blocks is being transferred at the maximum speed for a 100 Mbps network – that’s already a brilliant result compared to Mbed OS 6.xx.
I tested my system – over 200,000 data transfers from the FPGA to the PC through MBed CE were completed without a single error.
I’m very pleased!
Thank you very much your and all Mbed CE contributors - Mbed CE 7 is a big step forward !

[zhiyong-ft]

@IrekRM could you put in a PR to add some more optimized lwIP settings for STM targets? I ran into some issues that my STM32F767 devices would occasionally hang when making network requests. While troubleshooting, I noticed there was basically no tuning at all for STM targets. In v6.x the mem-size is left at 2300 for all STM targets, and in v7 mem-size is adjusted to 4000 by default. For the time being, I just copied the settings for MCU_PSOC6 to use on STM32F767, so far it seems to work better resilience wise. But I don't know lwIP enough to submit a PR myself.

[IrekRM]

Hello @zhiyong-ft,
Sorry – I’m not a professional programmer and there’s a lot I don’t understand about your area of expertise. For example, what is PR and how do I do it? The settings I used for my project are in the post above. I don’t want to delve into the intricacies of LWIP – it’s too far away from my project. But I’m really pleased that my system is running smoothly even with these sub-optimal settings. So I have great respect for, and am very grateful to, all the professionals who contribute to the development of Mbed CE!