Unable to perform OTA rollback from custom panic handler

[eraetsky]

Board

ESP32 Devkit V1

Device Description

Ordinary ESP32 Devkit V1

Hardware Configuration

Version

v2.0.14

IDE Name

VSCode, PlatformIO

Operating System

macOS

Flash frequency

40 Mhz

PSRAM enabled

no

Upload speed

115200

Description

I want to perform a rollback from a custom panic handler. I am using PlatformIO and tried with both frameworks arduino and espidf, and only arduino. When my overrided panic handler function reaches esp_ota_mark_app_invalid_rollback_and_reboot(); it gets stuck and nothing happens. I know that on arduino implementation of this function is precompiled, but I tested with espidf and found out that it freezes deep inside function call chain, on
esp_err_t spi_flash_mmap(size_t src_addr, size_t size, spi_flash_mmap_memory_t memory, const void** out_ptr, spi_flash_mmap_handle_t* out_handle)

I tested versions 2.0.14 and 2.0.6, also 2.0.6 with esp-idf version 4.4.5. On 2.0.14 I am able to call esp_restart() from the panic handler, on 2.0.6 - not. But on bare esp-idf 4.4.5 rollback from custom panic handler works fine (tested with native_ota_example on PlatformIO).

Also I tried overriding verifyRollbackLater(), setting it to return true. In this case, when esp_restart() is done from the custom panic handler, rollback goes on, but I want app received by OTA to be valid unless esp_ota_mark_app_invalid_rollback_and_reboot() is called. I understand that in this case rollback is performed because app has an unconfirmed status and reboot from any part of the code or by pressing the button will cause rollback, and it does not suit me.

My PlatformIO configurations:

1. arduino + espidf
   platformio.ini
   [env:esp32dev] platform = espressif32 board = esp32dev framework = arduino, espidf build_flags = -std=c++17 -std=gnu++17 -DVERSION=${this.custom_prog_version} -DCONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=1 -Wl,--wrap=esp_panic_handler

2. arduino
   platformio.ini
   [env:esp32dev] platform = espressif32 @ 5.4.0 board = esp32dev framework = arduino build_flags = -std=c++17 -std=gnu++17 -Wl,--wrap=esp_panic_handler -DVERSION=${this.custom_prog_version} -DCONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=1

I set platform = espressif32 @ 5.4.0 in order to switch to another versions.

3. espidf only, where rollback from panic handler is possible
   platformio.ini
   `
   [platformio]
   src_dir = main

[env:esp32dev]

platform = espressif32 @ 5.4.0
board = esp32dev
framework = espidf`

Thank you in advance.

Sketch

extern "C" bool verifyRollbackLater()
{
  esp_rom_printf("Rollback verification on Arduino init disabled!\n");
  return true;
}
extern "C" void __real_esp_panic_handler(void *);
extern "C" void __wrap_esp_panic_handler(void *info)
{
  esp_rom_printf("PANIC HANDLER TRYING TO ROLLBACK\n");
  esp_ota_mark_app_invalid_rollback_and_reboot();
}

Debug Message

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13104
load:0x40080400,len:3036
entry 0x400805e4
RK����verification on Arduino init disabled!
Setting WiFiClient as a network client
WiFi Creds: SSID [my ssid], PSWD: [my pswd]
[WiFi] WiFi is disconnected
Retry...
[WiFi] WiFi is disconnected
Retry...
[WiFi] SSID not found
Retry...
[WiFi] SSID not found
Retry...
[WiFi] SSID not found
Retry...
Network manager could not connect to any network :(
[MQTT] Creds: Broker URL [my broker]
PANIC HANDLER TRYING TO ROLLBACK

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[atanisoft]

At the point where the panic handler is running virtually all sub-systems have been suspended and not much can be done other than print a backtrace and reboot (which the default handler does).

It would be better to hook into the OTA validation callback to validate the new binary and rollback at that point upon validation failure.

[eraetsky]

At the point where the panic handler is running virtually all sub-systems have been suspended and not much can be done other than print a backtrace and reboot (which the default handler does).

It would be better to hook into the OTA validation callback to validate the new binary and rollback at that point upon validation failure.

Thank you for your response.
I understand that some system API could be unavailable in that point, but the thing is that it works on bare esp-idf (at least 4.4.5, 4.4.7).

I (538) heap_init: Initializing. RAM available for dynamic allocation:
I (545) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (551) heap_init: At 3FFB7738 len 000288C8 (162 KiB): DRAM
I (558) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (564) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (570) heap_init: At 400948C8 len 0000B738 (45 KiB): IRAM
I (578) spi_flash: detected chip: generic
I (581) spi_flash: flash io: dio
I (586) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
PANIC HANDLER TRYING TO ROLLBACK
ets Jul 29 2019 12:21:46

rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6608
load:0x40078000,len:15060
ho 0 tail 12 room 4
load:0x40080400,len:3816
entry 0x40080698
I (58) boot: ESP-IDF 4.4.5 2nd stage bootloader
I (58) boot: compile time 15:20:21
I (58) boot: chip revision: v3.0
I (61) boot.esp32: SPI Speed      : 40MHz
I (66) boot.esp32: SPI Mode       : DIO
I (70) boot.esp32: SPI Flash Size : 4MB
W (75) boot.esp32: PRO CPU has been reset by WDT.
W (80) boot.esp32: WDT reset info: PRO CPU PC=0x4008b94e
W (86) boot.esp32: WDT reset info: APP CPU PC=0x4015f82e (waiti mode)
I (93) boot: Enabling RNG early entropy source...
I (99) boot: Partition Table:
I (102) boot: ## Label            Usage          Type ST Offset   Length
I (110) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (117) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (125) boot:  2 factory          factory app      00 00 00010000 00100000
I (133) boot: End of partition table
I (137) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=1ce4ch (118348) map
I (188) esp_image: segment 1: paddr=0002ce74 vaddr=3ffb0000 size=030b0h ( 12464) load
I (193) esp_image: segment 2: paddr=0002ff2c vaddr=40080000 size=000ech (   236) load
I (194) esp_image: segment 3: paddr=00030020 vaddr=400d0020 size=92ce4h (601316) map
I (420) esp_image: segment 4: paddr=000c2d0c vaddr=400800ec size=147dch ( 83932) load
I (465) boot: Loaded app from partition at offset 0x10000
I (465) boot: Disabling RNG early entropy source...
I (476) cpu_start: Pro cpu up.
I (477) cpu_start: Starting app cpu, entry point is 0x40081af0
I (0) cpu_start: App cpu up.
I (493) cpu_start: Pro cpu start user code

In my case it is not clear how to validate if new binary is valid, the system is quite complicated and exception may occur not immediately. Rollback from the panic handler would be nice solution.

[atanisoft]

Can you try:

extern "C" void __wrap_esp_panic_handler(void *info)
{
  esp_rom_printf("PANIC HANDLER TRYING TO ROLLBACK\n");
  ESP_ERROR_CHECK(esp_ota_mark_app_invalid_rollback_and_reboot());
}

The esp_ota_mark_app_invalid_rollback_and_reboot function will return an error code if it fails to update, if it does update it will trigger a reboot.

[eraetsky]

This function does neither return any value nor prints any logs. It just freezes somewhere inside.
If I am not mistaken, this is its current implementation, it is called with valid = false.
I tried replacing esp_ota_mark_app_invalid_rollback_and_reboot invocation with its implementation and found out, that it stops in read_otadata, likely on line 76, esp_partition_mmap gets stuck and do not return.

[atanisoft]

It likely is hanging on accessing resources that are disabled by that point, it's not clear why esp-idf is passing at this point. You may need to enable logging for various functions to see where it is actually hanging or use gdb to trace into it.

Also note that referencing esp-idf master is not valid for any version of arduino-esp32. For the 2.x versions it should be on the release/v4.4 branch at whatever the esp-idf commit referenced in the arduino-esp32 code.

[eraetsky]

Thx for prompt response. May be I'll try to debug it further, but don't sure it make sense

[VojtechBartoska]

Hello @eraetsky, can this issue be closed?