Network performance drops with traffic volume

[icvidal]

Checklist

• Checked the issue tracker for similar issues to ensure this is not a duplicate
• Read the documentation to confirm the issue is not addressed there and your configuration is set correctly
• Tested with the latest version to ensure the issue hasn't been fixed

How often does this bug occurs?

always

Expected behavior

Irrespectful of network load or transfer volume, when the link is idle, network performance should remain stable with baseline latency.

Actual behavior (suspected bug)

On our custom device, with kernel 5.10 and using SPI, the baseline performance is good after boot.
Observations for baseline performance:

• Ping times are low, below 100ms (typically 25ms).
• No packet drop.
• Network responsive.

If iperf is run 3 - 5 times (or more), network performance degrades:

• Ping times are above 300ms, and often go around 2000-5000ms.
• Some packet drop (low).
• Connections time out.
  NOTE: This persists after iperf is stopped.

Error logs or terminal output

Steps to reproduce the behavior

1. Boot as normal, loading esp32_hosted_ng
2. Connect to WiFi
3. ping -c 10 8.8.8.8 # verify baseline response time
4. run iperf3 -c <server_hostname> more than 3 times (example: for i in seq 1 10;do iperf3 -c <server_hostname> ;done
5. ping -c 10 8.8.8.8 # verify response time after test

Project release version

latest

System architecture

MIPS - Custom device

Operating system

Linux

Operating system version

Linux kernel 5.10

Shell

ZSH

Setup Details

Transport: SPI
Slave: ESP32-S3
Reproductability: It does not occur on a raspberry pi.
Signal routing: Custom PCB
Frequency: Initial 10MHz, after negotiation 30MHz.

Additional context

1. Reloading the module seems to have some positive effect.
2. The issue does not reproduce over a raspberry pi (latest raspian). On the RPi, we also see good performance during the iperf3 test (ping response times around 20-50ms)
3. We worked around the issue with the following changes:

diff --git a/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/esp_cmd.c b/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/esp_cmd.c
index b32b577be..3da4ec994 100644
--- a/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/esp_cmd.c
+++ b/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/esp_cmd.c
@@ -485,7 +485,7 @@ static void esp_cmd_work(struct work_struct *work)
 
 static int create_cmd_wq(struct esp_adapter *adapter)
 {
-	adapter->cmd_wq = create_singlethread_workqueue("ESP_CMD_WORK_QUEUE");
+  adapter->cmd_wq = alloc_workqueue("ESP_CMD_WQ", WQ_HIGHPRI | WQ_MEM_RECLAIM, 1);
 
 	RET_ON_FAIL(!adapter->cmd_wq);
 
diff --git a/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/main.c b/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/main.c
index ebea4040d..492c91d5b 100644
--- a/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/main.c
+++ b/kernel/kernel-5.10/module_drivers/drivers/net/wireless/esp_hosted_ng/main.c
@@ -1039,7 +1039,7 @@ static struct esp_adapter *init_adapter(void)
 	memset(&adapter, 0, sizeof(adapter));
 
 	/* Prepare interface RX work */
-	adapter.if_rx_workqueue = alloc_workqueue("ESP_IF_RX_WORK_QUEUE", 0, 0);
+	adapter.if_rx_workqueue = alloc_workqueue("ESP_RX_WQ", WQ_HIGHPRI | WQ_MEM_RECLAIM, 1);
 
 	if (!adapter.if_rx_workqueue) {
 		deinit_adapter();

We are not sure if this will cause an issue over time. Please confirm.
4. We think this could be related to a defect on the SPI driver or SPI transaction. We would kindly like to add some resilience for faulty transfers to esp_hosted_ng.

[mantriyogesh]

Hello @icvidal ,

The changes seem to be fine.
You intend to use high priority for workqueue.

But I am still not very convinced if this would fix.

There are a lot of factors into play. Wi-Fi-Router, Wi-Fi interference, (network config, routing etc?), Hosted transport (is it jumper cables?), Frequency of operation, and many more.

I wish I would have reproduced this exact issue, which would have been then easier to track the issue down.

However, some understanding of the system would be really helpful to picture myself.

From prior history,
Transport I assume is SPI
slave used I assume to be ESP32-S3

Do you happen to have logic analyser with you? It would be good to see the capture, which would clear up the things.

If you do not have logic analyser, I might have to think why and what possible cause and rule out one by one. But this takes a lot time and interaction. Again, It would be much easier if I can reproduce at my end.

Do you observe this issue on some other hosts also?

[icvidal]

Hi @mantriyogesh! nice to see you again
Adding some data. (also editing the original comment).

Although running iperf did not affect the ping time at first, I've let my system run over night and repeated the test. Now the behavior is similar to the original. Looks like the changes only add some tolerance for some hours but after some time, it becomes the same.
Does this make any sense to you? I can repeat this test today and see results tomorrow.

Can you please tell me more why you think this won't fix? Evidently, you were right but it'd help me see where to look for to bring more information.

To answer your questions:
WiFi Router: seems not to have effect on this. My connection is not the best, but my colleagues experience the same issues in at least 3 other networks. Rebooting / reloading the module also seem to "reset" the behavior to normal, so I think that isolates the router/network variable.

Transport: SPI
Slave: ESP32-S3
Reproductability: It does not occur on a raspberry pi.
Frequency: Our DTS configures the SPI bus for up to 54MHz, the esp32_hosted_ng driver. The constant SPI_INITIAL_CLK_MHZ is left as 10MHz and the speed parameter to esp_init_interface_layer is kept as 0.
Later, when esp_adjust_spi_clock is called, the parameter is apparently 30MHz.
Signal routing: Custom PCB.
Note: I haven't seen this issue in a raspberry pi + dupont cable setup. That suggest it's likely not related to signal integrity.

We intend to prepare some boards to be analyzed, we can later submit logic analyzer captures. Is it enough to check MISO, MOSI, CLK, CS and HANDSHAKE ?

Note: We suspect it could be related to our host's SPI driver. If you have any idea on how to confirm or rule out this hypothesis feel free to share it. If there is a way to make the esp32_hosted_ng driver tolerant to this, we would be happy to try.

[mantriyogesh]

Hi @icvidal , sorry somehow i missed your ping. Apologies for the late reply.

Can you please tell me more why you think this won't fix? Evidently, you were right but it'd help me see where to look for to bring more information.

Not exactly that we won't fix, yet I am trying to find some correlation. it is very cloudy to make any statement.

Reproductability: It does not occur on a raspberry pi.

Yes I understood from your previous post, already. If it would have been reproducible, it would have been easier to fix.

WiFi Router: seems not to have effect on this. My connection is not the best, but my colleagues experience the same issues in at least 3 other networks. Rebooting / reloading the module also seem to "reset" the behavior to normal, so I think that isolates the router/network variable.

noted.

Note: I haven't seen this issue in a raspberry pi + dupont cable setup. That suggest it's likely not related to signal integrity.

Yes. I agree.

We intend to prepare some boards to be analyzed, we can later submit logic analyzer captures. Is it enough to check MISO, MOSI, CLK, CS and HANDSHAKE ?

It is better to capture
MISO
MOSI
CS
CLK
Handshake
DataReady
Resetpin (optional, but good to do)

The logic dump would actually clear off dust.

Note: We suspect it could be related to our host's SPI driver. If you have any idea on how to confirm or rule out this hypothesis feel free to share it. If there is a way to make the esp32_hosted_ng driver tolerant to this, we would be happy to try.

Can you please assess if FG driver also face this issue? The SPI is taken little difference in design for FG host driver.

[mantriyogesh]

Actually, I am not sure how deep you can go.

I would suspect if the GPIOs are really doing fine, specially Handshake and Data Ready.
Handshake is absolutely mandatory, you can bypass data ready for debugging from host driver, some changes would be needed.

But this is still not a focussed approach, which is why I do not wish to waste your time and panic about it later.

I think if FG driver works but Ng doesn't or any difference if you experience, that would at least give some direction ahead.

logic analyser dump would be the best to understand in times like this.