esp32_exception_decoder: ESP32-C3 support; possibility for external configuration

[ivmarkov]

Dear all,

This PR is implementing the following set of changes to the filter_exception_decoder.py (a.k.a. esp32_exception_decoder filter):

• (Most important) Support for ESP32-C3 (and future RiscV based chips)

  • The stacktraces generated by the ESP RiscV chip family do not really start with a Backtrace: string prefix (these start with a Stack memory:\n\n string prefix instead and overall have a very different and much more verbose dump of hexadecimal integers)
  • Rather than trying to extract the PC counters from this set of hex numbers, I've switched the filter to operate in a way similar to how the standard ESP IDF monitor operates (references embedded in the code itself): The ESP IDF monitor scans for any hex numbers that start with 0x4xxxxxxx instead, as this is an indication of a memory address which is corresponding to a code location. While this results in hexadecimals outside of the stacktrace being decoded as program locations, it is (a) more robust and (b) it has the nice benefit precisely that PC counters anywhere in the log will be decoded. I.e. we are planning to dump backtraces on non-panic situations from our Rust code, so this type of decoding will decode these backtraces too

• As a consequence, the decoded output is served on a separate line (or lines) following the line where PC counter address (or addresses) were detected. Looks quite OK IMO

• The decoded output is colored (with an ANSI sequence) in yellow, for better visibility. ESP-IDF logging uses ANSI sequences by default too

• Finally (but important for us) I've put provisions in the code to to take the firmware location, addr2line location and the project location via environment variables, as this is necessary for our Rust support (PlatformIO is used there as a build-support utility, but is not driving the whole build process, which is orchestrated by Rust Cargo). If these environment variables are not set, the filter will fall back to the standard logic of locating these items based on PlatformIO APIs.

[valeros]

Hi @Tasssadar ! Any chance you could take a look at this PR? Thanks in advance!

[Tasssadar]

Looks good to me, great work, thanks :)

Can you show us some screenshots how it looks like with ESP C3 and the normal stack trace from base ESP32?

Also, does the addr2line from rust have the same text output as the one from gcc? Might break the parsing in this filter.

[ivmarkov]

Nope it is not a great work, unfortunately.
I was looking into the detailed "stack-traces" produced in the ESP32-C3, as the decoding was producing decoded lines that are definitely not part of the stack and turns out the rabbit hole is much deeper than I thought.

First of all, on C3 they are producing not a backtrace, but stack dump it seems, which is then post-processed with GDB to get the actual "backtrace". If we don't do this as well in this filter (i.e. call GDB to get just the backtrace from the stack dump), I'm afraid we'll get decodings that simply do not belong to the backtrace and are there as left overs from previous calls.

(To make the rabbit hole even deeper, I'm seeing these fake things even on stock ESP32 with regular "backtrace" output, so this has to be checked as well - might be something Rust specific, related to calling conventions, etc.)

So what I would suggest is - I'll have to dig deeper first, and probably even talk to Espressif developers to see what the future plan is. They might be switching to "stack dumps" even on the Xtensa chips.

In the meantime, we should either close this PR (I'll open another one when I have some more visibility as to what is going on and what the overall plan is) or to turn this into draft.

[ivmarkov]

(re addr2line - the Rust toolchain is currently using the addr2line from GCC, as well as the linker from GCC. ESP-IDF itself is compiled with GCC, while the rust code is obviously compiled with Rust. They are linked then with GCC->LD)

[agners]

@ivmarkov FWIW, I noticed the missing backtrace on ESP32-C3/RISC-V too when adding support to ESPHome a while back:
esphome/esphome#2062 (comment)

You can "safely" decode the PC and LR to get at least the current function and the function before that (minus inlines, of course). Maybe this is start.
https://github.com/esphome/esphome/pull/2062/files#diff-dbf001f542fb85de707889ca75fa11637d2f7cbd08c826b8abe15ac512789c7d

I think this is a limitation of the toolchain and/or ESP-IDF's exception handling which cannot make sense of the stack currently and hence print a useable Backtrace.

[ivmarkov]

I think this is a limitation of the toolchain and/or ESP-IDF's exception handling which cannot make sense of the stack currently and hence print a useable Backtrace.

It is a RiscV-specific limitation in that the code on the ESP32-C3 is compiled without FP (frame pointer support), and therefore the stack trace cannot be reconstructed.

• Whether it would be possible to enable FPs in GCC (and four our case - in rustc as well), is yet to be seen, but that would be one avenue of exploration.
• Another option is to enable eh_frame support (DWARF tables) inside the ESP-IDF menuconfig with CONFIG_ESP_SYSTEM_USE_EH_FRAME . This will get us proper backtraces, but what I'm afraid of is that these tables include symbol (function) names so this option might be prohibitively expensive. Yet still worth exploring because some of this info must somehow be present even when CONFIG_ESP_SYSTEM_USE_EH_FRAME is disabled, because the C++ stack unwinding needs it anyway
• A third option is to just accept the "let's print the stack memory on RiscV instead of backtrace" status quo and try to decode it in the monitor using GDB, just like Espressif's own monitor does it, via GDB

More info here: esp-rs/rust#76 (comment)

[ivankravets]

What do mean these hacks?

[ivankravets]

@Tasssadar, sorry that disturb you. Does Espressif have an official decoder which can reuse without reinventing the wheels?

[ivmarkov]

It had been more than 6 months, and in the meantime things have changed a bit:

• Regarding support for Rust: this is no longer necessary. Rust ESP toolchain now has its own monitor and exception backtrace decoder (albeit for xtensa only, but if necessary, it can be extended for riscv as well). So this aspect of the patch (the "hacks" as you say) are no longer necessary
• As for decoding riscv backtraces, these are not backtraces at all, as compiling for riscv does not generate FP support. This is raw stack memory. To decode it, a simple lookup via addr2line is not enough. You should basically take the stack memory and hand it over to gdb as a core dump, together with the elf executable itself. You can look here for more details if you plan to reuse bits and pieces from the esp idf monitor

EDIT: More than 6 months, not an year, but a long time anyway :)

[pat1]

in documentation https://docs.platformio.org/en/latest/core/userguide/device/cmd_monitor.html#filters esp32_exception_decoder I found "Custom filter for Espressif 32 which decodes crash exception" so I understand it works for ESP32 -C3 too.
But it is not true.

[bmedici]

I can confirm it does not wirk with ESP32C3 for me, neither.

[bmedici]

Up ?