Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1211

brentru · 2023-10-18T20:31:47Z

@maxgerhardt @ivankravets Hi, I was wondering if there are any plans to synchronize the development branch of this repository with the IDF v5.1 alpha releases? I understand that these releases are in the alpha stage, but I believe that updating the PIO package to be compatible with IDF v5+ could greatly benefit the community.

If this isn't in the roadmap, could you please suggest any potential workarounds?

Arduino Alpha 2 Release v3.0.0 based on ESP-IDF v5.1

maxgerhardt · 2023-10-19T11:10:48Z

Can't speak for PIO Labs as a company since I'm not even an employee, but I've read multiple times that PIO's policy is to support stable versions, not alpha / beta versions. So if it's released as stable, support will probably follow.

Nevertheless, @Jason2866 has a Arduino/IDF5 branch in his https://github.com/Jason2866/platform-espressif32. So your first try should be to just use

platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5

in your platformio.ini and see where that takes you.

mhaberler · 2023-10-19T12:17:00Z

@maxgerhardt thanks!

I tried your suggestion with a small esp32s3 project with Arduino as an esp-idf component using PlatformIO 6.1.11

platform = https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5
framework = arduino, espidf

build and running works fine

what does not work any more is JTAG debugging as gdb is not found

FileNotFoundError: [Errno 2] No such file or directory: 'xtensa-esp32s3-elf-gdb'

I guess this has to do with the factoring out of esp32 gdb into its own repo, it is not present in .platformio/packages/toolchain-xtensa-esp32s3/bin/

I assume this would be available from https://docs.espressif.com/projects/esp-idf/en/v5.1.1/esp32s3/api-guides/tools/idf-tools.html?highlight=xtensa%20esp32s3%20elf%20gdb#xtensa-esp-elf-gdb

but I'm at loss on how to add this to my installation/project?

edit: found #1126 - will try if this works now

edit2: well the replace_gdb.py trick worked so far: #1126 (comment)

maxgerhardt · 2023-10-19T12:31:39Z

FileNotFoundError: [Errno 2] No such file or directory: 'xtensa-esp32s3-elf-gdb'

You might be able to counteract this by explicitly enabling the package like this does by in the platformio.ini with

platform_packages = tool-xtensa-esp-elf-gdb

If that does not work you might need to patch the ~/.platformio/platforms/espressif32/platform.json to include the declaration of these packages like mainline.

Jason2866 · 2023-10-19T13:14:28Z

The gdb are listed in the platform.json. I will have a look why the are not installed.
My Python Build Scripts differ from the official ones. Probably overlooked to install them...

Jason2866 · 2023-10-19T13:41:08Z

@mhaberler @maxgerhardt Fixed. Thx for the issue info. The gdb packages are now installed.

mhaberler · 2023-10-19T14:06:57Z

thanks, yes this did have the same effect and got me the proper gdb version:

Processing N8R8 (board: waveshare-ESP32-S3-DEV-KIT-N8R8; platform: https://github.com/Jason2866/platform-espressif32.git#Arduino/IDF5; framework: arduino, espidf)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/waveshare-ESP32-S3-DEV-KIT-N8R8.html
PLATFORM: Espressif 32 (2023.8.1+sha.7c70aa1) > Espressif ESP32-S3-WROOM-1U-N8R8
HARDWARE: ESP32S3 240MHz, 512KB RAM, 8MB Flash
DEBUG: Current (esp-builtin) On-board (esp-builtin) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 @ 3.0.0 
 - framework-espidf @ 3.50101.0 (5.1.1) 
 - tool-cmake @ 3.21.3 
 - tool-esptoolpy @ 1.40700.0+dev (4.7.0) 
 - tool-mklittlefs @ 3.2.0 
 - tool-mkspiffs @ 2.230.0 (2.30) 
 - tool-ninja @ 1.9.0 
 - tool-openocd-esp32 @ 2.1100.20220706 (11.0) 
 - tool-xtensa-esp-elf-gdb @ 12.1.0+20221002 
 - toolchain-esp32ulp @ 1.23500.220830 (2.35.0) 
 - toolchain-riscv32-esp @ 12.2.0+20230208 
 - toolchain-xtensa-esp32s3 @ 12.2.0+20230208

overall this looks pretty good so far!

remaining nit: if I enable PSRAM:

CONFIG_ESPTOOLPY_HEADER_FLASHSIZE_UPDATE=y
CONFIG_AUTOSTART_ARDUINO=y
CONFIG_ARDUHAL_LOG_DEFAULT_LEVEL_VERBOSE=y
CONFIG_ARDUHAL_LOG_COLORS=y
CONFIG_ARDUHAL_ESP_LOG=y
CONFIG_COMPILER_STACK_CHECK_MODE_NORM=y
CONFIG_COMPILER_WARN_WRITE_STRINGS=y
CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_SPIRAM_IGNORE_NOTFOUND=y
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
# CONFIG_ESP_INT_WDT is not set
# CONFIG_ESP_TASK_WDT_INIT is not set
CONFIG_FREERTOS_HZ=1000
CONFIG_LOG_DEFAULT_LEVEL_DEBUG=y

the build fails with

Building in debug mode
Compiling .pio/build/N8R8/esp32-hal-psram.o
/Users/mah/.platformio/packages/framework-arduinoespressif32-src-3175c3f96db9cfefc7591c2cd26d1e5d/cores/esp32/esp32-hal-psram.c:22:10: fatal error: esp_psram.h: No such file or directory

*******************************************************************
* Looking for esp_psram.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:esp_psram.h"
* Web  > https://registry.platformio.org/search?q=header:esp_psram.h
*
*******************************************************************

   22 | #include "esp_psram.h"
      |          ^~~~~~~~~~~~~
compilation terminated.

could be a variant issue

I could work around this by -I/Users/mah/.platformio/packages/framework-arduinoespressif32-src-3175c3f96db9cfefc7591c2cd26d1e5d/tools/esp32-arduino-libs/esp32s3/include/esp_psram/include and PSRAM seems to come up:

=========== Before Setup Start ===========
Chip Info:
------------------------------------------
  Model             : ESP32-S3
  Package           : 0
  Revision          : 1
  Cores             : 2
  Frequency         : 240 MHz
  Embedded Flash    : No
  Embedded PSRAM    : No
  2.4GHz WiFi       : Yes
  Classic BT        : No
  BT Low Energy     : Yes
  IEEE 802.15.4     : No
------------------------------------------
INTERNAL Memory Info:
------------------------------------------
  Total Size        :   419843 B ( 410.0 KB)
  Free Bytes        :   355219 B ( 346.9 KB)
  Allocated Bytes   :    59972 B (  58.6 KB)
  Minimum Free Bytes:   350743 B ( 342.5 KB)
  Largest Free Block:   253952 B ( 248.0 KB)
------------------------------------------
SPIRAM Memory Info:
------------------------------------------
  Total Size        :  8388608 B (8192.0 KB)
  Free Bytes        :  8386308 B (8189.8 KB)
  Allocated Bytes   :        0 B (   0.0 KB)
  Minimum Free Bytes:  8386308 B (8189.8 KB)
  Largest Free Block:  8257536 B (8064.0 KB)
  Bus Mode          : OPI
------------------------------------------
Flash Info:
------------------------------------------
  Chip Size         :  8388608 B (8 MB)
  Block Size        :    65536 B (  64.0 KB)
  Sector Size       :     4096 B (   4.0 KB)
  Page Size         :      256 B (   0.2 KB)
  Bus Speed         : 80 MHz
  Bus Mode          : DIO
------------------------------------------
Partitions Info:
------------------------------------------
                nvs : addr: 0x00009000, size:    24.0 KB, type: DATA, subtype: NVS
           phy_init : addr: 0x0000F000, size:     4.0 KB, type: DATA, subtype: PHY
            factory : addr: 0x00010000, size:  1024.0 KB, type:  APP, subtype: FACTORY
------------------------------------------
Software Info:
------------------------------------------
  Compile Date/Time : Oct 19 2023 15:58:40
  ESP-IDF Version   : 5.1.1.231006
  Arduino Version   : 3.0.0
------------------------------------------
Board Info:
------------------------------------------
  Arduino Board     : ESP32S3_DEV
  Arduino Variant   : esp32s3
  Core Debug Level  : 5
  Arduino Runs Core : 1
  Arduino Events on : 1
  Arduino USB Mode  : 1
  CDC On Boot       : 1
============ Before Setup End ============

JTAG debugging (without uploading) works as well

I am impressed @Jason2866 !

thanks @maxgerhardt !

mhaberler · 2023-10-19T14:37:55Z

got curious what would happen if I slide this underneath m5stack/M5Unified@^0.1.10 and m5stack/M5GFX@^0.1.10 for a m5stack-coreS3 target @lovyan03

there is fallout, but pretty minimal - just two hiccups on 3.0 API changes:

.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp: In member function 'virtual bool lgfx::v1::Light_PWM::init(uint8_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp:43:5: error: 'ledcSetup' was not declared in this scope
   43 |     ledcSetup(_cfg.pwm_channel, _cfg.freq, PWM_BITS);
      |     ^~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Light_PWM.cpp:44:5: error: 'ledcAttachPin' was not declared in this scope; did you mean 'ledcAttach'?
   44 |     ledcAttachPin(_cfg.pin_bl, _cfg.pwm_channel);
      |     ^~~~~~~~~~~~~
      |     ledcAttach
... and ...
*** [.pio/build/coreS3/lib899/M5GFX/lgfx/v1/platforms/esp32/Light_PWM.cpp.o] Error 1
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp: In function 'void lgfx::v1::calcClockDiv(uint32_t*, uint32_t*, uint32_t*, uint32_t*, uint32_t, uint32_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp:167:28: warning: conversion from 'float' to 'uint32_t' {aka 'long unsigned int'} may change value [-Wfloat-conversion]
  167 |         uint32_t b = roundf(fdiv * a);
      |                      ~~~~~~^~~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/common.cpp:171:35: warning: conversion from 'float' to 'uint32_t' {aka 'long unsigned int'} may change value [-Wfloat-conversion]
  171 |         uint32_t freq = baseClock / ((n * cnt) + (float)(b * cnt) / (float)a);
      |                         ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Bus_SPI.cpp: In function 'void lgfx::v1::gpio_reset(size_t)':
.pio/libdeps/coreS3/M5GFX/src/lgfx/v1/platforms/esp32/Bus_SPI.cpp:182:5: error: 'gpio_matrix_out' was not declared in this scope; did you mean 'gpio_iomux_out'?
  182 |     gpio_matrix_out((gpio_num_t)pin, SIG_GPIO_OUT_IDX, 0, 0);
      |     ^~~~~~~~~~~~~~~
      |     gpio_iomux_out
*** [.pio/build/coreS3/lib899/M5GFX/lgfx/v1/platforms/esp32/Bus_SPI.cpp.o] Error 1

maxgerhardt · 2023-10-19T14:42:21Z

I think the LEDC API have changed per this so you might be able to fork the library and correct their source code, with some

#if ESP_ARDUINO_VERSION_MAJOR >= 3
/* code */ 
#else 
/* code */
#endif

check around that code to still have backwards compat.

Jason2866 · 2023-10-19T15:07:49Z

@mhaberler Dont see any issue with S3 and PSRAM. The S3 Tasmota builds (S3 is a mess with OPI...) do compile and work fine with and without PSRAM.
We are using this Arduino 3.0 framework which is compiled with the IDF version used in this setup.
My guess the sdkconfig you use is not 100% correct.

brentru · 2023-10-19T15:42:12Z

@maxgerhardt Thank you for the initial comment, I didn't know you don't work for PIO Labs.

@Jason2866 Thanks for hosting a mirror of the latest BSP, works perfectly!

mhaberler · 2023-10-19T15:44:39Z

@Jason2866
I'll give M5GFX a try and then will post a minimal example

mhaberler · 2023-10-19T18:41:39Z

example ESP-IDF/Arduino project using M5GFX+M5Unified: https://github.com/mhaberler/M5GFX-arduino3

M5GFX fixes: https://github.com/mhaberler/M5GFX/tree/arduino-3.0 - hope these are correct, at least the demo works @lovyan03

@Jason2866 this is needed to get at esp_psram.h: https://github.com/mhaberler/M5GFX-arduino3/blob/main/platformio.ini#L12 - interested if you can reproduce

the stock board file fails to enable PSRAM (missing "memory_type": "qio_qspi")

debugging works fine
interestingly gdb is MUCH faster on a 2015 Intel MBP than on a current M1 Mac; it's not Rosetta, on the M1 gdb is a bona-fine arm64 binary

maxgerhardt · 2023-10-20T07:03:00Z

I don't see why you'd need framework = arduino, espidf specifically for the project though since nothing fundamental is changed in the sdkconfig / ESP-IDF base. Regular framework = arduino should work too and you can get rid of all CMake files, and possible have working OPI PSRAM out of the box without the -I fix?

Jason2866 · 2023-10-20T09:29:43Z

To have a valid framework with all files at the correct places you have to look at the Arduino Lib Builder how and where the files (especially for the flash types and PSRAM) are copied too. This is totally nonsense at all: -mfix-esp32-psram-cache-issue. Only needed for (a few) ESP32 boards with PSRAM. ALL other ESP32 MCUs don't have this bug.
Using Arduino as a Component of IDF is not always trivial. I do not recommend for Projects just running Arduino Code.

mhaberler · 2023-11-01T12:55:35Z

@Jason2866 is there a way to use your Arduino/IDF5 branch with Nimble and AsyncTCP?

mhaberler · 2023-11-01T13:48:11Z

I think the LEDC API have changed per this so you might be able to fork the library and correct their source code, with some
#if ESP_ARDUINO_VERSION_MAJOR >= 3
/* code */ 
#else 
/* code */
#endif
check around that code to still have backwards compat.

both issues have been fixed in the develop branch of https://github.com/m5stack/M5GFX/commits/develop

AFAICT there is a remaining IDF 5.1 problem with using M5Unfied with still using the legacy adc API, causing a crash during global ctor init

I've tried a brute-force adaptation to use the esp_adc/adc_oneshot includes and disabled the ADC code for now - which compiles and at least does not crash during startup

Jason2866 · 2023-11-01T15:32:19Z

@Jason2866 is there a way to use your Arduino/IDF5 branch with Nimble and AsyncTCP?

@mhaberler We use h2zero NimBLE for project Tasmota. It is way better than the Arduino NimBLE port. Since the IDF version from h2zero already supports C2 and C6 we use this variant.
Seen here https://github.com/arendst/Tasmota/tree/development/lib/libesp32_div/esp-nimble-cpp BLE works well with this version.
I don't know an well working AsyncTCP. If you find one you can simply add as a library

mhaberler · 2023-11-01T19:44:06Z

that was good advice, thanks!

esp-nimble-cpp running a continuous passive scan
stock web server with 2 clients, wifi AP client
an LVGL app running ontop of M5GFX and M5Unified for touch support

I get reasonably good WiFi/BLE coexistence, no crashes so far and some memory left:

free heap in loop(): Core2 79k, coreS3 195k

with just the barograph demo it’s core2 105k, coreS3 218k

so much better than anything I had before

edit: https://github.com/mhaberler/arduino3-playground/tree/loadtest

ivankravets · 2023-11-04T06:57:52Z

Please vote for this issue #1225

anthonyhoegberg · 2024-06-03T03:02:38Z

For the ESP_PSRAM.h not found problem, i fix to make it work
espressif/arduino-esp32#9756 (comment)

mhaberler pushed a commit to mhaberler/M5GFX-arduino3 that referenced this issue Oct 20, 2023

delete flag as per platformio/platform-espressif32#1211 (comment)

1e973e6

mhaberler pushed a commit to mhaberler/M5GFX-arduino3 that referenced this issue Oct 22, 2023

add gdb as per platformio/platform-espressif32#1211 (comment)

0c684a8

mhaberler pushed a commit to mhaberler/M5GFX-arduino3 that referenced this issue Oct 30, 2023

add gdb as per platformio/platform-espressif32#1211 (comment)

a573517

ivankravets changed the title ~~Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1~~ Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 Nov 4, 2023

ivankravets pinned this issue Nov 4, 2023

ivankravets unpinned this issue Nov 4, 2023

ivankravets changed the title ~~Support Arduino ESP32 v3.0 based on ESP-IDF v5.1~~ Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 Nov 4, 2023

ivankravets closed this as completed Nov 4, 2023

sgryphon mentioned this issue Jan 14, 2024

Support Arduino ESP32 alpha/RC v3.0.0 based on ESP-IDF v5.1 #1281

Open

This was referenced May 31, 2024

Zigbee/Thread support on the new upcoming ESP32-H2 variant? esphome/feature-requests#1397

Open

3.0.0 version Migration related issues espressif/arduino-esp32#8796

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1211

Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1211

brentru commented Oct 18, 2023 •

edited

Loading

maxgerhardt commented Oct 19, 2023 •

edited

Loading

mhaberler commented Oct 19, 2023 •

edited

Loading

maxgerhardt commented Oct 19, 2023 •

edited

Loading

Jason2866 commented Oct 19, 2023

Jason2866 commented Oct 19, 2023 •

edited

Loading

mhaberler commented Oct 19, 2023

mhaberler commented Oct 19, 2023

maxgerhardt commented Oct 19, 2023 •

edited

Loading

Jason2866 commented Oct 19, 2023

brentru commented Oct 19, 2023

mhaberler commented Oct 19, 2023

mhaberler commented Oct 19, 2023

maxgerhardt commented Oct 20, 2023

Jason2866 commented Oct 20, 2023 •

edited

Loading

mhaberler commented Nov 1, 2023

mhaberler commented Nov 1, 2023

Jason2866 commented Nov 1, 2023 •

edited

Loading

mhaberler commented Nov 1, 2023 •

edited

Loading

ivankravets commented Nov 4, 2023

anthonyhoegberg commented Jun 3, 2024

Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1211

Track Arduino ESP32 Alpha Release v3.0.0 based on ESP-IDF v5.1 Support Arduino ESP32 v3.0 based on ESP-IDF v5.1 #1211

Comments

brentru commented Oct 18, 2023 • edited Loading

maxgerhardt commented Oct 19, 2023 • edited Loading

mhaberler commented Oct 19, 2023 • edited Loading

maxgerhardt commented Oct 19, 2023 • edited Loading

Jason2866 commented Oct 19, 2023

Jason2866 commented Oct 19, 2023 • edited Loading

mhaberler commented Oct 19, 2023

mhaberler commented Oct 19, 2023

maxgerhardt commented Oct 19, 2023 • edited Loading

Jason2866 commented Oct 19, 2023

brentru commented Oct 19, 2023

mhaberler commented Oct 19, 2023

mhaberler commented Oct 19, 2023

maxgerhardt commented Oct 20, 2023

Jason2866 commented Oct 20, 2023 • edited Loading

mhaberler commented Nov 1, 2023

mhaberler commented Nov 1, 2023

Jason2866 commented Nov 1, 2023 • edited Loading

mhaberler commented Nov 1, 2023 • edited Loading

ivankravets commented Nov 4, 2023

anthonyhoegberg commented Jun 3, 2024

brentru commented Oct 18, 2023 •

edited

Loading

maxgerhardt commented Oct 19, 2023 •

edited

Loading

mhaberler commented Oct 19, 2023 •

edited

Loading

maxgerhardt commented Oct 19, 2023 •

edited

Loading

Jason2866 commented Oct 19, 2023 •

edited

Loading

maxgerhardt commented Oct 19, 2023 •

edited

Loading

Jason2866 commented Oct 20, 2023 •

edited

Loading

Jason2866 commented Nov 1, 2023 •

edited

Loading

mhaberler commented Nov 1, 2023 •

edited

Loading