mbedTLS memory allocation stragegy SPIRAM/default allocation mode breaks mbedTLS with gcc 8 (IDFGH-1336)

[PerMalmberg]

Environment

• Development Kit: ESP32-Wrover-Kit
• Kit version v3
• Module or chip used: ESP32-WROOM-32
• IDF version v4.0-dev-1136-g28f1cdf5e
• Build System: CMake
• Compiler version xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0
• Operating System: Linux
• Power Supply: USB

Problem Description

This bug relates to this question of the forum.

To make a long story short - forcing mbedTLS to use external memory breaks it in such a way that all TLS handshakes results in this error:

mbedtls_ssl_handshake returned -29056

meaning "SSL - Verification of the message MAC failed"

Expected Behavior

TLS to be functional even when using SPIRAM

Actual Behavior

I (15503) tcpip_adapter: sta ip: 192.168.10.94, mask: 255.255.255.0, gw: 192.168.10.254
I (15503) sslServer: Connected to AP
E (26173) sslServer: failed
! mbedtls_ssl_handshake returned -29056

Steps to repropduce

I've attached a zip with a modified https_server example in which I've replaced the actual server with one that uses mbedTLS (original source by @projectgus here, but modified to use the certs provided in the https_server project).

Simply change the hard-coded "CHANGE ME" ssid and password in main2.c, compile, run and browse to IP:9080 and you'll see your browser emit something like this, as well as the above log message.

An error occurred during a connection to 192.168.10.94:9080. SSL peer reports incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_ALERT

Changing mbedTLS memory allocation strategy to internal makes the handshake function again and the server will output the browsers request on screen.

I've intentionally forced most memory allocations to use SPIRAM by setting CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL to 256 to ensure the problem is shown in this example.

Unfortunately there is not enough internal memory for my project so I can't do without SPIRAM so any attempt to have mbedTLS only use internal memory results in memory allocation errors, with the side effects that brings.

mbedtls_broken_with_gcc8.zip

[negativekelvin]

In a browser it fails but it works in curl. Actually it works in a browser as long as you force https in the URL. The problem with chrome is that it will timeout on the first try and close the connection but it will succeed on the second try. However your example code returns on handshake failure instead of continuing the accept loop.

[PerMalmberg]

@negativekelvin Then you're getting other results than me. Just verified that if I browse to https://IP:9080, then I get the MAC-failure. Did you use the sdkconfig in the zip?

However your example code returns on handshake failure instead of continuing the accept loop.

It was the only example that uses mbedTLS I found that wasn't written by me I could find and in this case it doesn't matter - the MAC failure happens.

[negativekelvin]

Yes I only changed the Serial flasher settings.

Firefox:

I (1098) sslServer: Connected to AP
Result: [size: 339]
GET / HTTP/1.1
Host: 192.168.1.50:9080
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:66.0) Gecko/20100101 Fir
efox/66.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1

Chrome:

E (1775488) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592


Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.1.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/73.0.3683.103 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/a
png,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

[PerMalmberg]

Could the difference in result be due to different hardware? What device are you running on, @negativekelvin ?

[negativekelvin]

No same hardware but I am using make and gcc 5.

[PerMalmberg]

Ah, that might be the difference then. Could you please try with gcc 8 and same IDF just to see if it is reproducible by others than me?

[negativekelvin]

Yes switching to gcc 8 I can reproduce it

It does get pretty far into the handshake and then
mbedtls_cipher_auth_decrypt() returned -25344 (-0x6300)

[PerMalmberg]

@negativekelvin Thanks for confirming.

So it seems this is related to the gcc 8 tool chain and thus of interest for @igrr perhaps?

[PerMalmberg]

For the record: I tried the very latest IDF and xtensa-gcc 8 with the same result. OP updated with new versions.

Also, I tried to go back to gcc 5, but gcc 8 rc1 is the only one offered on the download page now so I'm totally blocked now.

[igrr]

For the GCC 5 link, you can still find it here: https://docs.espressif.com/projects/esp-idf/en/stable/get-started/linux-setup.html#toolchain-setup

[PerMalmberg]

Oh right. I always go /latest in the docs, not /stable Thank you @igrr

[PerMalmberg]

I forgot - with access to gcc 8 I've started using C++ 17, so going back to gcc 5 isn't an appealing option. Using bleeding edge tech is fun, but sometimes it hurts 💩 Then again, that's what early access is for.

[xiruilin]

Use "libc-psram-workaround.a" instead of "libc.a" to solve the problem,
espressif/arduino-esp32#2080
But now gcc 8.2.0 don't support "libc-psram-workaround.a", error message:

�[0;32mI (754) cpu_start: Starting app cpu, entry point is 0x400814a4�[0m
�[0;32mI (0) cpu_start: App cpu up.�[0m
�[0;32mI (1234) spiram: SPI SRAM memory test OK�[0m
�[0;32mI (1236) heap_init: Initializing. RAM available for dynamic allocation:�[0m
�[0;32mI (1236) heap_init: At 3FFAFF10 len 000000F0 (0 KiB): DRAM�[0m
�[0;32mI (1241) heap_init: At 3FFB6388 len 00001C78 (7 KiB): DRAM�[0m
�[0;32mI (1247) heap_init: At 3FFB9A20 len 00004108 (16 KiB): DRAM�[0m
�[0;32mI (1253) heap_init: At 3FFBDB5C len 00000004 (0 KiB): DRAM�[0m
�[0;32mI (1259) heap_init: At 3FFC4740 len 0001B8C0 (110 KiB): DRAM�[0m
�[0;32mI (1266) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM�[0m
�[0;32mI (1272) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM�[0m
�[0;32mI (1279) heap_init: At 40099E50 len 000061B0 (24 KiB): IRAM�[0m
�[0;32mI (1285) cpu_start: Pro cpu start user code�[0m
�[0;32mI (1290) spiram: Adding pool of 4070K of external SPI memory to heap allocator�[0m
�[0;32mI (1311) pm_esp32: Frequency switching config: CPU_MAX: 240, APB_MAX: 240, APB_MIN: 40, Light sleep: DISABLED�[0m
�[0;32mI (1311) cpu_start: Starting scheduler on PRO CPU.�[0m
�[0;32mI (0) cpu_start: Starting scheduler on APP CPU.�[0m
�[0;32mI (1325) spiram: Reserving pool of 32K of internal memory for DMA/internal allocations�[0m
GGuru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad
uGuru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad
Guru Meditation Error: Core 1 panic'ed (IllegalInstruction). Exception was unhandled.
Memory dump at 0x40206ffc: bad00bad bad00bad bad00bad

[negativekelvin]

@xiruilin suggestion works, overriding the newlib component with the gcc8 versions.

Actually I take it back it doesn't work 100% of the time. It might require the new psram fix.

[PerMalmberg]

It might require the new psram fix

You mean the outstanding issue in gcc 8?

[negativekelvin]

I mean this #2892

[PerMalmberg]

I mean this #2892

Yeah, that one. I wonder though - this MAC issue seems to only occur in gcc 8. Why is that you think? gcc8 just happens to generate code that triggers the PSRAM every time it runs the crypto code?

[negativekelvin]

Well I did not test enough to say it doesn't ever occur in gcc 5. I don't know if there are ABI changes that cause exactly the right trigger, but maybe.

[PerMalmberg]

Tried with the latest IDF (original post updated), and the issue remains.

[PerMalmberg]

It's been a month since this ticket was opened, would it be possible to get some feedback from Espressif, @igrr?

[enricop]

Hi all, we have the same issue in our project, where we have memory constraints.
Before the recent upgrade to mbedtls 2.16.2 the allocation into SPI RAM was working. But now error occurs in tls handshake. (We are using W-Rover-B and GCC8)

[igrr]

@PerMalmberg with the latest master branch (842432f) I can not reproduce the issue using the project attached to the OP. I see a handshake error -0x7780 with Chrome (as @negativekelvin has reported) and both GCC 5.2 and 8.2 based toolchains. There is no error with curl and Safari in both GCC versions.

However, while looking into this issue, I have found a potential problem related to the way psram-workaround version of libc.a was built for the GCC 8 toolchain. Could you please try replacing /path-to/xtensa-esp32-elf/xtensa-esp32-elf/lib/esp32-psram/libc.a with the attached copy, and see if the issue persists?

libc.a.zip
(the archive was stripped of debug symbols to make it fit into an attachment)

[PerMalmberg]

• v4.0-dev-1275-gfdab15dc7, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0

  • Problem exists

• v4.0-dev-1338-g842432fb6, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0

  • Problem still exists

• v4.0-dev-1338-g842432fb6, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0, using @igrr 's libc.a

  • Linker error as follows

[827/828] Linking CXX executable https_server.elf
FAILED: https_server.elf 
: && /home/permal/esp/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++  -mlongcalls -Wno-frame-address  -nostdlib CMakeFiles/https_server.elf.dir/project_elf_src.c.obj  -o https_server.elf  esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/driver/libdriver.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/efuse/libefuse.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/app_update/libapp_update.a esp-idf/spi_flash/libspi_flash.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/lwip/liblwip.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/esp_event/libesp_event.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/esp32/libesp32.a esp-idf/xtensa/libxtensa.a esp-idf/esp_common/libesp_common.a esp-idf/esp_rom/libesp_rom.a esp-idf/soc/libsoc.a esp-idf/log/liblog.a esp-idf/heap/libheap.a esp-idf/freertos/libfreertos.a esp-idf/vfs/libvfs.a esp-idf/newlib/libnewlib.a esp-idf/cxx/libcxx.a esp-idf/app_trace/libapp_trace.a esp-idf/asio/libasio.a esp-idf/coap/libcoap.a esp-idf/console/libconsole.a esp-idf/nghttp/libnghttp.a esp-idf/esp-tls/libesp-tls.a esp-idf/esp_adc_cal/libesp_adc_cal.a esp-idf/esp_gdbstub/libesp_gdbstub.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/esp_https_ota/libesp_https_ota.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/protocomm/libprotocomm.a esp-idf/mdns/libmdns.a esp-idf/esp_local_ctrl/libesp_local_ctrl.a esp-idf/esp_websocket_client/libesp_websocket_client.a esp-idf/expat/libexpat.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/fatfs/libfatfs.a esp-idf/freemodbus/libfreemodbus.a esp-idf/jsmn/libjsmn.a esp-idf/json/libjson.a esp-idf/libsodium/liblibsodium.a esp-idf/mqtt/libmqtt.a esp-idf/openssl/libopenssl.a esp-idf/spiffs/libspiffs.a esp-idf/ulp/libulp.a esp-idf/unity/libunity.a esp-idf/wifi_provisioning/libwifi_provisioning.a esp-idf/main/libmain.a esp-idf/protocol_examples_common/libprotocol_examples_common.a -Wl,--cref -Wl,--Map=/home/permal/tmp/build/https_server.map esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/driver/libdriver.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/efuse/libefuse.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/app_update/libapp_update.a esp-idf/spi_flash/libspi_flash.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/lwip/liblwip.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/esp_event/libesp_event.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/esp32/libesp32.a esp-idf/xtensa/libxtensa.a esp-idf/esp_common/libesp_common.a esp-idf/esp_rom/libesp_rom.a esp-idf/soc/libsoc.a esp-idf/log/liblog.a esp-idf/heap/libheap.a esp-idf/freertos/libfreertos.a esp-idf/vfs/libvfs.a esp-idf/newlib/libnewlib.a esp-idf/cxx/libcxx.a esp-idf/app_trace/libapp_trace.a esp-idf/asio/libasio.a esp-idf/coap/libcoap.a esp-idf/console/libconsole.a esp-idf/nghttp/libnghttp.a esp-idf/esp-tls/libesp-tls.a esp-idf/esp_adc_cal/libesp_adc_cal.a esp-idf/esp_gdbstub/libesp_gdbstub.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/esp_https_ota/libesp_https_ota.a esp-idf/esp_http_client/libesp_http_client.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/protocomm/libprotocomm.a esp-idf/mdns/libmdns.a esp-idf/esp_local_ctrl/libesp_local_ctrl.a esp-idf/esp_websocket_client/libesp_websocket_client.a esp-idf/expat/libexpat.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/fatfs/libfatfs.a esp-idf/wear_levelling/libwear_levelling.a esp-idf/sdmmc/libsdmmc.a esp-idf/freemodbus/libfreemodbus.a esp-idf/jsmn/libjsmn.a esp-idf/json/libjson.a esp-idf/libsodium/liblibsodium.a esp-idf/mqtt/libmqtt.a esp-idf/tcp_transport/libtcp_transport.a esp-idf/esp-tls/libesp-tls.a esp-idf/openssl/libopenssl.a esp-idf/spiffs/libspiffs.a esp-idf/ulp/libulp.a esp-idf/unity/libunity.a esp-idf/wifi_provisioning/libwifi_provisioning.a esp-idf/protocomm/libprotocomm.a esp-idf/esp_http_server/libesp_http_server.a esp-idf/nghttp/libnghttp.a esp-idf/protobuf-c/libprotobuf-c.a esp-idf/mdns/libmdns.a esp-idf/console/libconsole.a esp-idf/json/libjson.a esp-idf/cxx/libcxx.a esp-idf/newlib/libnewlib.a esp-idf/freertos/libfreertos.a esp-idf/heap/libheap.a esp-idf/log/liblog.a esp-idf/soc/libsoc.a esp-idf/esp_rom/libesp_rom.a esp-idf/esp_common/libesp_common.a esp-idf/xtensa/libxtensa.a esp-idf/esp32/libesp32.a esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/lwip/liblwip.a esp-idf/mbedtls/mbedtls/library/libmbedtls.a esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a esp-idf/mbedtls/mbedtls/library/libmbedx509.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/spi_flash/libspi_flash.a esp-idf/efuse/libefuse.a esp-idf/app_update/libapp_update.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcore.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libespnow.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libmesh.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libphy.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libpp.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/librtc.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a esp-idf/esp_event/libesp_event.a esp-idf/driver/libdriver.a esp-idf/vfs/libvfs.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/app_trace/libapp_trace.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a esp-idf/cxx/libcxx.a esp-idf/newlib/libnewlib.a esp-idf/freertos/libfreertos.a esp-idf/heap/libheap.a esp-idf/log/liblog.a esp-idf/soc/libsoc.a esp-idf/esp_rom/libesp_rom.a esp-idf/esp_common/libesp_common.a esp-idf/xtensa/libxtensa.a esp-idf/esp32/libesp32.a esp-idf/esp_ringbuf/libesp_ringbuf.a esp-idf/lwip/liblwip.a esp-idf/mbedtls/mbedtls/library/libmbedtls.a esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a esp-idf/mbedtls/mbedtls/library/libmbedx509.a esp-idf/bootloader_support/libbootloader_support.a esp-idf/spi_flash/libspi_flash.a esp-idf/efuse/libefuse.a esp-idf/app_update/libapp_update.a esp-idf/tcpip_adapter/libtcpip_adapter.a esp-idf/wpa_supplicant/libwpa_supplicant.a esp-idf/nvs_flash/libnvs_flash.a esp-idf/smartconfig_ack/libsmartconfig_ack.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libcore.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libespnow.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libmesh.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libphy.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libpp.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/librtc.a /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a esp-idf/esp_event/libesp_event.a esp-idf/driver/libdriver.a esp-idf/vfs/libvfs.a esp-idf/esp_wifi/libesp_wifi.a esp-idf/esp_eth/libesp_eth.a esp-idf/app_trace/libapp_trace.a esp-idf/pthread/libpthread.a esp-idf/espcoredump/libespcoredump.a -lstdc++ -u __cxa_guard_dummy -u __cxx_fatal_exception esp-idf/newlib/libnewlib.a -u newlib_include_locks_impl -u newlib_include_heap_impl -u newlib_include_syscalls_impl -u newlib_include_pthread_impl -Wl,--undefined=uxTopUsedPriority -L /home/permal/esp/esp-idf/components/esp_rom/esp32/ld -T esp32.rom.ld -T esp32.rom.libgcc.ld -T esp32.rom.syscalls.ld -T esp32.rom.newlib-data.ld -Wl,--gc-sections /home/permal/esp/esp-idf/components/xtensa/esp32/libhal.a -L /home/permal/tmp/build/esp-idf/esp32 -T esp32_out.ld -u app_main -L /home/permal/esp/esp-idf/components/esp32/ld -T esp32.extram.bss.ld -L /home/permal/tmp/build/esp-idf/esp32/ld -T esp32.project.ld -T esp32.peripherals.ld -u call_user_start_cpu0 -u ld_include_panic_highint_hdl -mfix-esp32-psram-cache-issue -u esp_app_desc -u vfs_include_syscalls_impl -L /home/permal/esp/esp-idf/components/esp_wifi/lib_esp32 -lgcov -lc -lm -lgcc && :
/home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/bin/ld: /home/permal/esp/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.2.0/../../../../xtensa-esp32-elf/lib/esp32-psram/libc.a: error adding symbols: archive has no index; run ranlib to add one
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed

Running ar -t libc.a yields a list of several object files:

ar -t libc.a                                                                                                                                     
_atexit.o
clibrary_init.o
creat.o
isatty.o
lib_a-_Exit.o
lib_a-__adjust.o
lib_a-__atexit.o
lib_a-__call_atexit.o
lib_a-__dprintf.o
...continued...

[igrr]

Sorry, stripping the static library was obviously a bad idea. The full version is at https://dl.espressif.com/dl/misc/IDFGH-1336-libc.a.zip, please give a try.

[PerMalmberg]

Full test results:

• 4.0-dev-1275-gfdab15dc7, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0
  • Problem exists
• v4.0-dev-1338-g842432fb6, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0
  • Problem still exists
• v4.0-dev-1338-g842432fb6, xtensa-esp32-elf-gcc (crosstool-NG esp32-2019r1) 8.2.0, using @igrr 's unstripped libc.a
  • Success!

I wonder why you can't reproduce it with the latest master and the original libc.a ?

[negativekelvin]

for me it is still not working consistently

D (4970) sslServer: >> waiting for accept
E (9360) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592


D (9370) sslServer: >> waiting for accept
E (11720) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592


D (11730) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9


D (15780) sslServer: >> waiting for accept
E (102280) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592


D (102280) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9


D (106310) sslServer: >> waiting for accept
E (113650) sslServer:  failed
  ! mbedtls_ssl_handshake returned -29312


D (113650) sslServer: >> waiting for accept
E (115690) sslServer:  failed
  ! mbedtls_ssl_handshake returned -78


D (115700) sslServer: >> waiting for accept
Result: [size: 449]
GET / HTTP/1.1
Host: 192.168.11.50:9080
Connection: keep-alive
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Saf
ari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exc
hange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9


D (119520) sslServer: >> waiting for accept
E (127220) sslServer:  failed
  ! mbedtls_ssl_handshake returned -30592


D (127230) sslServer: >> waiting for accept
E (130710) sslServer:  failed
  ! mbedtls_ssl_handshake returned -29056


D (130720) sslServer: >> waiting for accept
I (134830) sslServer: connection closed
Result: [size: 0]

[PerMalmberg]

@igrr Where are we on this? Any more changes coming, or will it be in the next release of the tool chain?

[PerMalmberg]

Another one and a half month later, status update, please @igrr ?

[igrr]

@PerMalmberg sorry, the fix should indeed be part of the new toolchain release. Unfortunately it got held up by a different issue, which we are working on. Hopefully it will get solved next week.

[enricop]

yes, all this seems related to a bigger issue concerning external ram allocation, which corrupts in some way.
We wait for this fix for our project too!
thanks for your support

[jeromeDms]

We also wait for this fix :)

[AndroidBaaaby]

Hi @igrr , I was just wondering when the new toolchain will be released.

[igrr]

Unfortunately we ran into an issue with enabling RTTI support in this new toolchain build. We are working on some alternative solution. I will update this issue when we have some progress!

[igrr]

The toolchain has been updated in c2db6a1#diff-0633a6582c888c6fc5bc6a3c00dba496, this version includes the correctly built libc.a. Please update the toolchain and give it a try. This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

[mcilloni]

The toolchain has been updated in c2db6a1#diff-0633a6582c888c6fc5bc6a3c00dba496, this version includes the correctly built libc.a. Please update the toolchain and give it a try. This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

Will release/4.0 also support this new toolchain? We're currently using the betas from 4.0 on our projects and it would be nice to avoid having to swap libc.a with the patched one.

[PerMalmberg]

This version uses the original version of the PSRAM workaround, not the new one mentioned in #2892.

Which, apart from possibly solving the issue reported in this thread, still leaves us with only unicore being a stable mode of operation. Can we expect another updated toolchain before the final release of v4.0 ?

[igrr]

release/v4.0 has also been updated to this toolchain release, it will get synced to GitHub once we fix some of the failing tests on our side.

Regarding the new PSRAM workaround version, I think it is unlikely that it will be finished in time for the 4.0 release. We will definitely include it into the subsequent patch release (4.0.x), once this toolchain version is ready. The linked issue will be updated when this happens.

[sebirdman]

@igrr using the latest release/v4.0 ( a3f3c7b ) with the 2020r1 psram fix toolchain release still results in this bug. Are there other steps i should be taking or should i to wait for release/4.0 to be updated?

edit:

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs) seem to make the problem go away. I'm using the idf_as_lib strategy with cmake for building. Is there some way i should be properly linking these? I have:

set(CMAKE_C_FLAGS "-mlongcalls -mfix-esp32-psram-cache-issue -Wno-frame-address" CACHE STRING "C Compiler Base Flags")
set(CMAKE_CXX_FLAGS "-mlongcalls -mfix-esp32-psram-cache-issue -Wno-frame-address" CACHE STRING "C++ Compiler Base Flags")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "shared ldflags")
set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "module linker flags")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS_INIT} -mfix-esp32-psram-cache-issue" CACHE STRING "exe linker flags")

in my toolchain.cmake file

[igrr]

Could you please do a verbose build, copy the final linker command line, and paste it here?

[sebirdman]

@igrr the final command is the one that generates the .elf file correct?

i notice that the espressif components have the -mfix-esp32-psram-cache flag after the .a blobs, but mine do not. Maybe that's part of the issue.

xtensa-esp32-elf-g++  -mlongcalls -Wno-frame-address  -mfix-esp32-psram-cache-issue CMakeFiles/app.elf.dir/main/app/main.c.obj  -o app.elf  source/libs/home_controller/libhome_controller.a  source/libs/global_state/libglobal_state.a  esp-idf/esp32/libesp32.a  -mfix-esp32-psram-cache-issue  -fno-rtti  -fno-lto  source/libs/monitors/libmonitors.a  source/libs/shopvac/libshopvac.a  source/libs/logdna_vacuum/liblogdna_vacuum.a  source/libs/file_ringbuffer/libfile_ringbuffer.a  source/libs/coredump_vacuum/libcoredump_vacuum.a  platform/esp32/port/coredump_platform/libcoredump_platform_esp32.a  source/libs/stats_vacuum/libstats_vacuum.a  source/libs/debug_shell/libdebug_shell.a  source/libs/plugin_manager/libplugin_manager.a  source/libs/diagnostics/libdiagnostics.a  source/libs/bluetooth_state/libbluetooth_state.a  source/libs/global_state/libglobal_state.a  source/hw/i2c/libi2c.a  source/libs/company_log/libcompany_log.a  source/libs/e2e/libe2e.a  source/libs/jobs/libjobs.a  source/libs/corewifi/libcorewifi.a  source/libs/internet_manager/libinternet_manager.a  source/libs/company_ui/libcompany_ui.a  source/libs/time_machine/libtime_machine.a  source/libs/aquaman/libaquaman.a  source/hw/hardware/libhardware.a  source/hw/kvs/libkvs.a  source/hw/storage/libstorage.a  source/libs/company_kvs/libcompany_kvs.a  source/libs/company_encryption/libcompany_encryption.a  source/hw/hwinfo/libhwinfo.a  source/libs/event_manager/libevent_manager.a  source/libs/ocelot/libocelot.a  source/hw/platform/libplatform.a  source/libs/sentry_client/libsentry_client.a  source/libs/enum_to_string/libenum_to_string.a  source/libs/profile/libprofile.a  source/libs/mqtt_setup/libmqtt_setup.a  source/libs/company_svcs/libcompany_svcs.a  source/libs/bt_handler/libbt_handler.a  source/libs/corebt/libcorebt.a  platform/esp32/port/bt_platform/libbt_platform_esp32.a  dependencies/generic/telnet_client/liblibtelnet.a  platform/esp32/port/wifi_platform/libwifi_platform.a  source/libs/mqtt_client/libmqtt_client.a  source/libs/local_api/liblocal_api.a  source/libs/company_packet_processor/libcompany_packet_processor.a  source/libs/lytebyte/liblytebyte.a  source/hw/gpio/libgpio.a  platform/esp32/port/kvs_platform/libkvs_platform.a  platform/esp32/port/storage_platform/libstorage_platform.a  source/libs/uuid4/libuuid4.a  platform/esp32/port/i2c_platform/libi2c_platform.a  platform/esp32/port/platform/libplatform_platform.a  source/libs/https_manager/libhttps_manager.a  source/libs/company_json/libcompany_json.a  source/libs/firmware_update/libfirmware_update.a  source/libs/wifi_provisioner/libwifi_provisioner.a  source/libs/bt_mesh_stack/libbt_mesh_stack.a  source/libs/jwt/libjwt.a  source/libs/mesh_handlers/libmesh_handlers.a  source/libs/device_info_service/libdevice_info_service.a  source/libs/linking_handler/liblinking_handler.a  source/libs/s3_utils/libs3_utils.a  platform/esp32/port/ping_platform/libping_platform.a  dependencies/generic/paho/liblibpaho.a  source/libs/sddp/libsddp.a  platform/esp32/port/rgbw_platform/librgbw_platform.a  platform/esp32/port/timer_platform/libtimer_platform.a  source/libs/b64url/libb64url.a  platform/esp32/port/https_requests_platform/libhttps_requests_platform_esp32.a  platform/esp32/port/ota_platform/libota_platform.a  source/libs/global_state/libglobal_state.a  source/hw/i2c/libi2c.a  source/libs/company_log/libcompany_log.a  source/libs/e2e/libe2e.a  source/libs/jobs/libjobs.a  source/libs/corewifi/libcorewifi.a  source/libs/internet_manager/libinternet_manager.a  source/libs/company_ui/libcompany_ui.a  source/libs/time_machine/libtime_machine.a  source/libs/aquaman/libaquaman.a  source/hw/hardware/libhardware.a  source/hw/kvs/libkvs.a  source/hw/storage/libstorage.a  source/libs/company_kvs/libcompany_kvs.a  source/libs/company_encryption/libcompany_encryption.a  source/hw/hwinfo/libhwinfo.a  source/libs/event_manager/libevent_manager.a  source/libs/ocelot/libocelot.a  source/hw/platform/libplatform.a  source/libs/sentry_client/libsentry_client.a  source/libs/enum_to_string/libenum_to_string.a  source/libs/profile/libprofile.a  source/libs/mqtt_setup/libmqtt_setup.a  source/libs/company_svcs/libcompany_svcs.a  source/libs/bt_handler/libbt_handler.a  source/libs/corebt/libcorebt.a  platform/esp32/port/bt_platform/libbt_platform_esp32.a  dependencies/generic/telnet_client/liblibtelnet.a  platform/esp32/port/wifi_platform/libwifi_platform.a  source/libs/mqtt_client/libmqtt_client.a  source/libs/local_api/liblocal_api.a  source/libs/company_packet_processor/libcompany_packet_processor.a  source/libs/lytebyte/liblytebyte.a  source/hw/gpio/libgpio.a  platform/esp32/port/kvs_platform/libkvs_platform.a  platform/esp32/port/storage_platform/libstorage_platform.a  source/libs/uuid4/libuuid4.a  platform/esp32/port/i2c_platform/libi2c_platform.a  platform/esp32/port/platform/libplatform_platform.a  source/libs/https_manager/libhttps_manager.a  source/libs/company_json/libcompany_json.a  source/libs/firmware_update/libfirmware_update.a  source/libs/wifi_provisioner/libwifi_provisioner.a  source/libs/bt_mesh_stack/libbt_mesh_stack.a  source/libs/jwt/libjwt.a  source/libs/mesh_handlers/libmesh_handlers.a  source/libs/device_info_service/libdevice_info_service.a  source/libs/linking_handler/liblinking_handler.a  source/libs/s3_utils/libs3_utils.a  platform/esp32/port/ping_platform/libping_platform.a  dependencies/generic/paho/liblibpaho.a  source/libs/sddp/libsddp.a  platform/esp32/port/rgbw_platform/librgbw_platform.a  platform/esp32/port/timer_platform/libtimer_platform.a  source/libs/b64url/libb64url.a  platform/esp32/port/https_requests_platform/libhttps_requests_platform_esp32.a  platform/esp32/port/ota_platform/libota_platform.a  platform/esp32/port/ntp_platform/libntp_platform.a  platform/esp32/port/ntp_platform/libntp_platform.a  dependencies/generic/addons/libfreertos_extras.a  source/libs/json_stream/libjson_stream.a  esp-idf/bt/libbt.a  -L/esp-idf/components/bt/controller/lib  -lbtdm_app  esp-idf/mdns/libmdns.a  esp-idf/console/libconsole.a  platform/esp32/port/gpio_platform/libgpio_platform.a  platform/esp32/port/gpio_platform/libgpio_platform.a  esp-idf/spiffs/libspiffs.a  source/libs/hex_utils/libhex_utils.a  esp-idf/jsmn/libjsmn.a  esp-idf/json/libjson.a  source/libs/crc/libcrc.a  esp-idf/esp_http_client/libesp_http_client.a  esp-idf/tcp_transport/libtcp_transport.a  esp-idf/esp-tls/libesp-tls.a  esp-idf/nghttp/libnghttp.a  dependencies/generic/safestringlib/libsafeclib.a  esp-idf/esp32/libesp32.a  esp-idf/cxx/libcxx.a  esp-idf/newlib/libnewlib.a  esp-idf/freertos/libfreertos.a  esp-idf/heap/libheap.a  esp-idf/log/liblog.a  esp-idf/soc/libsoc.a  esp-idf/esp_rom/libesp_rom.a  esp-idf/esp_common/libesp_common.a  esp-idf/xtensa/libxtensa.a  esp-idf/driver/libdriver.a  esp-idf/esp_event/libesp_event.a  esp-idf/efuse/libefuse.a  esp-idf/app_trace/libapp_trace.a  esp-idf/app_update/libapp_update.a  esp-idf/bootloader_support/libbootloader_support.a  esp-idf/nvs_flash/libnvs_flash.a  esp-idf/pthread/libpthread.a  esp-idf/spi_flash/libspi_flash.a  esp-idf/vfs/libvfs.a  esp-idf/wpa_supplicant/libwpa_supplicant.a  esp-idf/espcoredump/libespcoredump.a  esp-idf/esp_wifi/libesp_wifi.a  esp-idf/esp_ringbuf/libesp_ringbuf.a  esp-idf/tcpip_adapter/libtcpip_adapter.a  esp-idf/esp_eth/libesp_eth.a  esp-idf/lwip/liblwip.a  esp-idf/mbedtls/mbedtls/library/libmbedtls.a  esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a  esp-idf/mbedtls/mbedtls/library/libmbedx509.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcore.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libespnow.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libmesh.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libphy.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libpp.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/librtc.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a  esp-idf/esp32/libesp32.a  esp-idf/cxx/libcxx.a  esp-idf/newlib/libnewlib.a  esp-idf/freertos/libfreertos.a  esp-idf/heap/libheap.a  esp-idf/log/liblog.a  esp-idf/soc/libsoc.a  esp-idf/esp_rom/libesp_rom.a  esp-idf/esp_common/libesp_common.a  esp-idf/xtensa/libxtensa.a  esp-idf/driver/libdriver.a  esp-idf/esp_event/libesp_event.a  esp-idf/efuse/libefuse.a  esp-idf/app_trace/libapp_trace.a  esp-idf/app_update/libapp_update.a  esp-idf/bootloader_support/libbootloader_support.a  esp-idf/nvs_flash/libnvs_flash.a  esp-idf/pthread/libpthread.a  esp-idf/spi_flash/libspi_flash.a  esp-idf/vfs/libvfs.a  esp-idf/wpa_supplicant/libwpa_supplicant.a  esp-idf/espcoredump/libespcoredump.a  esp-idf/esp_wifi/libesp_wifi.a  esp-idf/esp_ringbuf/libesp_ringbuf.a  esp-idf/tcpip_adapter/libtcpip_adapter.a  esp-idf/esp_eth/libesp_eth.a  esp-idf/lwip/liblwip.a  esp-idf/mbedtls/mbedtls/library/libmbedtls.a  esp-idf/mbedtls/mbedtls/library/libmbedcrypto.a  esp-idf/mbedtls/mbedtls/library/libmbedx509.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcoexist.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libcore.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libespnow.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libmesh.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libnet80211.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libphy.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libpp.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/librtc.a  ../dependencies/espressif/esp-idf/components/esp_wifi/lib_esp32/libsmartconfig.a  -mfix-esp32-psram-cache-issue  -L /build_esp32_app/esp-idf/esp32  -T esp32_out.ld  -u app_main  -L /build_esp32_app/esp-idf/esp32/ld  -T esp32.project.ld  -L /esp-idf/components/esp32/ld  -T esp32.peripherals.ld  -u call_user_start_cpu0  -u ld_include_panic_highint_hdl  -u __cxa_guard_dummy  -lstdc++  esp-idf/pthread/libpthread.a  -u __cxx_fatal_exception  esp-idf/newlib/libnewlib.a  -u newlib_include_locks_impl  -u newlib_include_heap_impl  -u newlib_include_syscalls_impl  -u newlib_include_pthread_impl  -Wl,--undefined=uxTopUsedPriority  -L /esp-idf/components/esp_rom/esp32/ld  -T esp32.rom.ld  -T esp32.rom.libgcc.ld  -T esp32.rom.syscalls.ld  -T esp32.rom.newlib-data.ld  -T esp32.rom.newlib-nano.ld  -Wl,--gc-sections  ../dependencies/espressif/esp-idf/components/xtensa/esp32/libhal.a  -lgcov  -lc_nano  -lm  -lgcc  -u esp_app_desc  -u pthread_include_pthread_impl  -u pthread_include_pthread_cond_impl  -u pthread_include_pthread_local_storage_impl  -u vfs_include_syscalls_impl  -L /esp-idf/components/esp_wifi/lib_esp32

[igrr]

Thanks, this looks correct. Could you please run xtensa-esp32-elf-g++ -mfix-esp32-psram-cache-issue -print-search-dirs and check if the paths under libraries include esp32-psram subdirectories?

Also, if you have switched from esp-2019r2 to esp-2020r1 toolchain, have you removed your CMakeCache.txt? CMake caches the path to the compiler, so it might be possible that the older toolchain is still used.

One other thing, could you please explain what you mean by

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs)

Which exact files have you copied, and to what location?

[igrr]

Oh, i think i see the reason — you are using "nano" C library, which is not compatible with PSRAM. Please check if the issue persists after disabling "nano" option.

[sebirdman]

Removing the nano option seems to have done it, i've successfully ota'd my device from an https endpoint 3 times now. Thanks @igrr

One other thing, could you please explain what you mean by

Forcefully copying the psram fix blobs in the 2020r1 toolchain (and using them to replace the standard blobs)

I copied xtensa-esp32-elf/lib/esp32-psram files out into the main xtensa-esp32-elf/lib folder, replacing the files there. This least helped me debug that i was doing something wrong and the the toolchain was working.

[baldhead69]

Hi,

I tested with firefox, chrome, opera and edge and not work.

With Chrome:
I (764165) esp_https_server: performing session handshake
E (765545) esp-tls-mbedtls: mbedtls_ssl_handshake returned -30592
E (765546) esp_https_server: esp_tls_create_server_session failed
W (765551) httpd: httpd_accept_conn: session creation failed
W (765555) httpd: httpd_server: error accepting new connection
I (765562) esp_https_server: performing session handshake
E (766929) esp-tls-mbedtls: mbedtls_ssl_handshake returned -30592
E (766930) esp_https_server: esp_tls_create_server_session failed
W (766934) httpd: httpd_accept_conn: session creation failed
W (766939) httpd: httpd_server: error accepting new connection
I (766946) esp_https_server: performing session handshake

esp-idf version: v4.2-dev-1235-g71dc5eb27
ESP32-DevKitC-VB
Windows 10

[tom-borcin]

Hello @PerMalmberg, we are closing this issue. It was fixed in 4.0.2: #2892 (comment)

If you think this issue should be open, please let me know. Thank you!