mbedtls DTLS Streaming (UDP) and ASyncUDP Receiving at the same time

[alorbach]

I am on ESP-IDF release 3.3 with arduino-esp32 master branch
Build with https://github.com/espressif/esp32-arduino-lib-builder

I have a sketch that receives about 60 UDP packages per seconds, with a data size of up to 1024 bytes per packet.
At the same time its streaming DTLS data (UDP) with about 50 packets per second with about 70 bytes erach packet.

I have more than enough memory free:
FreeMem ~ 140KB
MaxFreeHeap: 180KB

At a random point mbdedtls returns an error while sending:

error: [write():207]: (0XFFFFFFB2) UNKNOWN ERROR CODE (004E)

I then tear down the connection and reestablish DTLS Streaming. However from this point WIFI becomes very unstable.
Also incoming UDP packages start to stutter and fill up the internal queue. When I stop UDP streaming to the device, I can see that it still is processing UDP packages for 5-15 seconds.

Even if I stop sending / receiving UDP for a while, WIFI remains unstable. At some point it losses the connection completely.

Here is some debug output:

%debug: 0:492699: esp32debug: I (494326) wifi:
%debug: 0:492700: esp32debug: state: run -> init (2c0)
%debug: 0:492700: esp32debug: I (494327) wifi:
%debug: 0:492702: esp32debug: pm stop, total sleep time: 0 us / 490633554 us
%debug: 0:492708: esp32debug: I (494335) wifi:
%debug: 0:492712: esp32debug: new:<13,0>, old:<13,0>, ap:<255,255>, sta:<13,0>, prof:1
%debug: 1:492722: wpsWiFiEvent:Disconnected: Disconnected from station reason: 2 (auth expire), reconnect handling...
%debug: 1:492730: wpsWiFiEvent:Disconnected: isConnected true, force disconnect!
%debug: 0:492738: esp32debug: I (494364) wifi:
%debug: 0:492741: esp32debug: flush txq
%debug: 0:492745: esp32debug: I (494371) wifi:
%debug: 0:492749: esp32debug: stop sw txq
%debug: 0:492753: esp32debug: I (494380) wifi:
%debug: 0:492756: esp32debug: lmac stop hw txq
%debug: 0:492869: esp32debug: I (494495) wifi:
%debug: 0:492869: esp32debug: mode : sta (cc:50:e3:9a:03:8c)
%debug: 1:492869: wpsWiFiEvent: WiFi interface ready
%debug: 1:492872: wpsWiFiEvent: WiFi.mode set to WIFI_MODE_STA
%debug: 1:492877: wpsWiFiEvent: try WiFi.begin to wifisid/password...
%debug: 1:492984: wpsWiFiEvent: Station Mode Started
%debug: 1:492984: wpsWiFiEvent: Station Mode Started
%debug: 0:493194: esp32debug: I (494820) wifi:
%debug: 0:493196: esp32debug: Set ps type: 0
%debug: 0:493554: esp32debug: I (495180) wifi:
%debug: 0:493554: esp32debug: new:<13,0>, old:<1,0>, ap:<255,255>, sta:<13,0>, prof:1
%debug: 0:493555: esp32debug: I (495182) wifi:
%debug: 0:493559: esp32debug: state: init -> auth (b0)
%debug: 0:494564: esp32debug: I (496191) wifi:
%debug: 0:494564: esp32debug: state: auth -> init (200)
%debug: 0:494565: esp32debug: I (496191) wifi:
%debug: 0:494566: esp32debug: new:<13,0>, old:<13,0>, ap:<255,255>, sta:<13,0>, prof:1
%debug: 1:494575: wpsWiFiEvent:Disconnected: Disconnected from station reason: 2 (auth expire), reconnect handling...
%debug: 1:494584: wpsWiFiEvent: WiFi.mode set to WIFI_MODE_STA
%debug: 1:494589: wpsWiFiEvent: try WiFi.begin to wifisid/password...
...
%debug: 1:508587: wpsWiFiEvent:Disconnected: Disconnected from station reason: 2 (auth expire), reconnect handling...
%debug: 1:508596: wpsWiFiEvent: WiFi.mode set to WIFI_MODE_STA
%debug: 1:508601: wpsWiFiEvent: try WiFi.begin to wifisid/password ...
%debug: 0:508609: esp32debug: E (510235) wifi:
%debug: 0:508612: esp32debug: sta is connecting, return error
%debug: 0:508947: esp32debug: I (510574) wifi:
%debug: 0:508948: esp32debug: new:<13,0>, old:<13,0>, ap:<255,255>, sta:<13,0>, prof:1
%debug: 0:508948: esp32debug: I (510575) wifi:
%debug: 0:508952: esp32debug: state: init -> auth (b0)
%debug: 0:508969: esp32debug: I (510595) wifi:
%debug: 0:508969: esp32debug: state: auth -> assoc (0)
%debug: 0:508993: esp32debug: I (510620) wifi:
%debug: 0:508994: esp32debug: state: assoc -> run (10)
%debug: 0:510042: esp32debug: I (511669) wifi:
%debug: 0:510042: esp32debug: connected with wifisid, aid = 4, channel 13, BW20, bssid = f4:06:8d:56:6f:30
%debug: 0:510045: esp32debug: I (511672) wifi:
%debug: 0:510049: esp32debug: security type: 3, phy: bgn, rssi: -64
%debug: 0:510061: esp32debug: I (511688) wifi:
%debug: 0:510061: esp32debug: pm start, type: 0
%debug: 1:510066: wpsWiFiEvent: STA Connected to: wifisid

However WIFI remains unstable even after WIFI is reconnected until I reset the device.
It only happens when mbedtls and AsyncUDP are running at the same time.

Btw I already changed sdkconfig to higher the AsyncUDP Task, but this does not help.

CONFIG_ARDUINO_UDP_RUN_CORE0=y
CONFIG_ARDUINO_UDP_RUN_CORE1=
CONFIG_ARDUINO_UDP_RUN_NO_AFFINITY=
CONFIG_ARDUINO_UDP_TASK_PRIORITY=5
CONFIG_ARDUINO_UDP_RUNNING_CORE=0

The MBEDTLS Task is actually running on priority 3 and on Core 0.

[alorbach]

I digged little deeper and it actually fails in mbedtls_net_send with MBEDTLS_ERR_NET_SEND_FAILED.
I will try to output errno for the system error code.

EDIT: Errno is 12

EDIT": Ok this means
#define ENOMEM 12 /* Out of memory *

But I don't understand there is like 100 KB memory left.

[alorbach]

After weeks of getting mad because of this issue, I digged deep down into heap, lwip debugging and so on - nothing really was helping and showing that there is any memory shortage.

However today I most likely found the root cause of the issue.

So I have a Class that does the DTLS UDP Streaming which uses HTTPClient from arduino-esp32, and I was using new/delete to create HTTPClient objects dynamically after the ERR_MEM occurred.

I remembered that I had trouble (strange exceptions) when deleting dynamic HTTPClient objects before in my OTA Updatecode. So as a workaround, I made the HTTPClient Object static which fixed the problem at that time.

Now I applied the same to my Class that does the DTLS Streaming, I made the HTTPClient object static at the top of the Class code like this:
HTTPClient mHttp;

And it appears to have fixed the problem mostly - I am running DTLS Streaming now since about 60 minutes without any ERR_MEM till yet.

So I think HTTPClient is messing something badly up in the destructor, perhaps heap corruption?

All the samples I find are not using new/delete but either static HTTPClient objects or on Stack.

[stale]

[STALE_SET] This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[stale]

[STALE_DEL] This stale issue has been automatically closed. Thank you for your contributions.