Implement setting interrupt handlers

[MabezDev]

Each xtensa lx6 core has 32 interrupts, 26 of which can be used for peripherals. The remaining 6 are for the on cpu peripherals.

Current understanding

My understanding is as follows:

There is a fixed table of interrupted with fixed priorities(levels), with some reserved, see https://github.com/espressif/esp-idf/blob/c1ab87b58057f0ddb8bad7e15a48a6697e8cda27/components/soc/esp32/include/soc/soc.h#L353-L388

Any unreserved interrupts can have a handler attached to them, provided their level is <= 3, anything greater requires asm.

This doesn't provide many interrupts (compared to armv7's 256 per core); I believe the esp-idf registers a global interrupt handler for a given interrupt, with the option of sharing that handler. (Edit: I've found some info online that indicates, the interrupt matrix was designed with interrupt sharing in mind, but need to follow up on how that actually works)

Questions

• The idf links against the xchal, a small runtime provided by cadence, which already has this table along with all the other vectors. Is there more info on the layout of vecbase? Will we have to link against this xchal too, or will we be able to write our own?

• Are the 'reserved' interrupts in that table really reserved (i.e via hardware) or are they just reserved in the IDF for specific usage.

[igrr]

First, slightly off-topic, I'd suggest not to focus on lx6 too much. From programmer's perspective, there is very little different between lx6 and lx7 or lx106; all these versions of Tensilica processors implement the same ISA. Newer versions of the processors may be slightly more efficient or may add some features. What does mainly change from one specific processor to another is how it was configured at design time. Tensilica processors have many configuration options which can be set when the chip designers generate the processor. These options affect the features present in the processor (instructions, registers, FPUs, coprocessors) and may change the ABI.

For Xtensa CPUs found in Espressif chips, you can find the list of configuration options in the overlay. For example, this is the configuration of the core used in the esp32: https://github.com/espressif/xtensa-overlays/blob/master/xtensa_esp32/newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h. This repository contains configuration files for cores in other chips as well.

The idf links against the xchal, a small runtime provided by cadence, which already has this table along with all the other vectors. Is there more info on the layout of vecbase? Will we have to link against this xchal too, or will we be able to write our own?

IDF does link to Xtensa libhal, however this is not required to set up interrupt handling. The layout of the exception vectors can be found... you've guessed it right, in the overlay: https://github.com/espressif/xtensa-overlays/blob/master/xtensa_esp32/newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h#L555-L599. The addresses mentioned there are given relative to the start of the ROM (0x40000000). However you can always point the CPU to your custom exception vectors using the VECBASE register. Here is the example how this is done in IDF:

1. Each exception vector is placed into its own section: https://github.com/espressif/esp-idf/blob/c1ab87b58057f0ddb8bad7e15a48a6697e8cda27/components/freertos/xtensa_vectors.S#L1189-L1197
2. Sections corresponding to the exception vectors are placed at appropriate offsets: https://github.com/espressif/esp-idf/blob/c1ab87b58057f0ddb8bad7e15a48a6697e8cda27/components/esp32/ld/esp32.project.ld.in#L121-L145
3. Startup code sets VECBASE to the first exception vector: https://github.com/espressif/esp-idf/blob/c1ab87b58057f0ddb8bad7e15a48a6697e8cda27/components/esp32/cpu_start.c#L133-L136

Are the 'reserved' interrupts in that table really reserved (i.e via hardware) or are they just reserved in the IDF for specific usage.

The type of each interrupt (external, timer, profiling, debug, software, NMI) is set at the time when the processor is generated, and as such can be found in the overlay: https://github.com/espressif/xtensa-overlays/blob/fe9a594027aa5a39cd1e653b786c42319a7afd90/xtensa_esp32/newlib/newlib/libc/sys/xtensa/include/xtensa/config/core-isa.h#L410-L442

Some of the interrupts in the table in soc.h are marked as "reserved" or have specific purpose assigned to them (such as Wi-Fi or BT). If you don't care about linking to the Wi-Fi or Bluetooth binary libraries at some point, then you can use these reserved interrupts. These libraries may use the designated interrupts for their own purpose. This is not something that is fixed in hardware, though.

[MabezDev]

From programmer's perspective, there is very little different between lx6 and lx7 or lx106

Thats good to know. How would you approach the differences in ABI? It seems like the esp8266 doesnt have the windowed ABI configured for example.

Each exception vector is placed into its own section:

So what does the layout of these vectors look like? The dispatch of the interrupts looks at a couple of special registers. The comment for that asm macro states that interrupts would be dispatched if _xtos_set_interrupt_handler had registered the handler. I couldn't find the source of _xtos_set_interrupt_handler anywhere though, only that it's equivilent to ets_isr_attach. I'd basically like to know what I need to do to attach a function (asm or rust) to an interrupt source with the right mask. Also where does _xtos_interrupt_table live and what is its layout? is it simply a block of function pointers like in armv7 chips?

Some of the interrupts in the table in soc.h are marked as "reserved" or have specific purpose assigned to them (such as Wi-Fi or BT). If you don't care about linking to the Wi-Fi or Bluetooth binary libraries at some point, then you can use these reserved interrupts. These libraries may use the designated interrupts for their own purpose. This is not something that is fixed in hardware, though.

Good to know. I think we will definitely want to use the wifi/ble blobs in the future, though I forsee that being quite a challege in itself. I'm guessing there are no plans (and probably licensing issues?) with open sourcing, or documenting the radio registers?

[igrr]

Thats good to know. How would you approach the differences in ABI? It seems like the esp8266 doesnt have the windowed ABI configured for example.

Typically in C code we use XCHAL_HAVE_xxx macros to check whether a feature is enabled. Example related to Call0/Window ABI: https://github.com/espressif/esp-idf/blob/600d542f53ab6540d277fca4716824d168244c8c/components/freertos/portasm.S#L509-L526.

So what does the layout of these vectors look like? The dispatch of the interrupts looks at a couple of special registers.

First, there is an important distinction between interrupts and exceptions. In Xtensa architecture, interrupts are a special case of exceptions. So in order to handle interrupts, first we need to be able to handle exceptions.

Each exception has an entry point where the processor jumps to (VECBASE + offset). There is some space (64 bytes) between entry points of adjacent exceptions. Simple exception handlers may be able to fit into this space. For more complex exception vectors, the entry point will simply jump to a different location. Some exception handlers may also call a user-provided handler, which can be set at run time. This is done by simply looking up the handler in a function pointer array, and calling it if the handler is not NULL. This is not used often though.

Here is the list of exception handlers, their offsets from VECBASE, along with some comments:

Window exception handlers

These exceptions exist to support the window register mechanism of Xtensa architecture. As far as I know, there is only one correct implementation of these exception handlers, it can be found here: https://github.com/espressif/esp-idf/blob/600d542f53ab6540d277fca4716824d168244c8c/components/freertos/xtensa_vectors.S#L1757-L1982. This is basically the equivalent of the code which saves/restores registers to/from the stack, on other architectures. If you do override VECBASE, you must provide correct implementation of these vectors, as they are required for function calls to work (except for very simple programs which use less than 64 hardware registers, or programs written for Call0 ABI).

Exception	Offset
WindowOverflow4	0x000
WindowUnderflow4	0x040
WindowOverflow8	0x080
WindowUnderflow8	0x0C0
WindowOverflow12	0x100
WindowUnderflow12	0x040

Actually useful exception handlers

UserException is the most useful/common one. It happens in multiple cases. They are distinguished by the value of EXCCAUSE register.

• Faults (illegal memory access, illegal instruction, misaligned access, and so on)
• Level 1 Interrupts (from external peripherals or CPU internal peripherals, we'll get to this later)
• Syscalls. For example, alloca and setjmp functions invoke these.

Level2Interrupt and Level3Interrupt exceptions get invoked for interrupts with corresponding levels.

In the first approximation, implementing a UserException handler along with Level 1 interrupt handler is probably sufficient, as most practical things can easily be done using these.

UserException handler is fairly complex (see the asm source linked above). It saves some registers, calls _xt_context_save to spill the register windows and save some more registers, then sets up stack frames so that a debugger can trace execution back from an exception, re-enables Window call mechanism, and finally jumps to the handler written in C.

Exception	Offset
UserException	0x340
Level2Interrupt	0x180
Level3Interrupt	0x1c0

Less useful exception handlers

For these exception handlers, trivial implementations are probably sufficient, initially. Level4 and Level5 interrupts are "high" level interrupts in ESP32 core configuration, and it is not recommended to handle them in C (or Rust) code. Typically these are reserved for fairly low-level handlers which do not rely on the existence of a stack. NMI Exception also falls into this category.

DebugException can be a simple infinite loop. Kernel and Double exceptions usually happen only if stuff went wrong in other exception handlers. Reset vector is only useful if you are implementing the on-chip ROM, otherwise the CPU doesn't have a chance of jumping there.

Exception	Offset
Level4Interrupt	0x200
Level5Interrupt	0x240
DebugException	0x2c0
NMIException	0x300
KernelException	0x340
DoubleException	0x3c0
Reset	0x400

Interrupt handling

Okay, let's assume that we have the basic set of exception handlers implemented, and UserException handler jumps to the Level 1 interrupt handler if EXCAUSE equals to EXCCAUSE_LEVEL1INTERRUPT.

At this point we can figure out which of the 32 interrupts has happened by reading the INTERRUPT register. Then we take the lowest bit set in this register, and look up the corresponding interrupt handler function in an _xt_interrupt_handler array. This array is defined here. There is some special handling for CPU internal timer interrupts and CPU internal software interrupts, plus the usual context saving and setting-up-frames-for-backtrace. I think the thing called _xtos_interrupt_table is the same as _xt_interrupt_handler, just the former is the name used in Xtensa SDK (which we mostly don't use) and the latter is the name from Xtensa FreeRTOS port (which we do use).

---

I think we will definitely want to use the wifi/ble blobs in the future, though I forsee that being quite a challege in itself. I'm guessing there are no plans (and probably licensing issues?) with open sourcing, or documenting the radio registers?

I think the amount of information necessary to convey all the details of hardware operation (both radio and the MAC) is just too large to make such a documentation and then re-implementation efforts practical. I am very sympathetic to open-sourcing these libraries, and it might make it slightly easier to build a Rust interface to them. The most recent one to be open sourced was the wpa_supplicant fork, but there are no concrete plans for the next step yet. So for now the best bet would be to try to interface with the closed-source libs. As @projectgus has mentioned when replying on this topic on Reddit, we can try to help from our side, if the Wi-Fi library OS abstraction layer ends up being hard to implement in Rust.

[MabezDev]

Thanks that really clears things up! I will try and get something simple working over the weekend.

Typically in C code we use XCHAL_HAVE_xxx macros to check whether a feature is enabled. Example related to Call0/Window ABI: https://github.com/espressif/esp-idf/blob/600d542f53ab6540d277fca4716824d168244c8c/components/freertos/portasm.S#L509-L526.

I think we should be able to do something similar with cargo features.

I think the amount of information necessary to convey all the details of hardware operation (both radio and the MAC) is just too large to make such a documentation and then re-implementation efforts practical. I am very sympathetic to open-sourcing these libraries, and it might make it slightly easier to build a Rust interface to them. The most recent one to be open sourced was the wpa_supplicant fork, but there are no concrete plans for the next step yet. So for now the best bet would be to try to interface with the closed-source libs. As @projectgus has mentioned when replying on this topic on Reddit, we can try to help from our side, if the Wi-Fi library OS abstraction layer ends up being hard to implement in Rust.

I was originally planning to use the unfortunately named rtfm rtos, mainly because I have experience with it, but also because it has multicore support out of the box. rtfm is simple to port and very performant, but it uses a run-to-completion scheduler (it basically uses the interrupt controller as a scheduler). My headaches started when @projectgus kindly pointed out the API abstraction layer to me; I realised the API spawns tasks and a lot more, which I wasn't expecting.

Since then I've been looking for other Rust rtos's and came across tockOS which looks like it fits the more traditional rtos model. I haven't settled on tockOS but it's looking the most likely at the moment.

Out of curiosity, how did the wifi libs work for the esp8266? I know there is a freertos port for it, but there is also a non os variant. Do the blobs for the esp8266 work without any OS?

[MabezDev]

So I've put together something that might do the job in #6 , notable differences are that I'm using Rust's #[naked] functions to avoid an extra build step (linking against some prebuilt asm). https://github.com/esp-rs/xtensa-lx6-rt/blob/cc621d832f7e64940b0beb688f45a203573441f4/src/exceptions.rs#L88-L92

(gdb) disassemble _UserExceptionVector
Dump of assembler code for function _UserExceptionVector:
   0x40080340 <+0>:     wsr.excsave1    a0
   0x40080343 <+3>:     call0   0x40081598 <_rust_user_exc>
   0x40080346 <+6>:     j       0x40080349 <_UserExceptionVector+9>
   0x40080349 <+9>:     j       0x40080349 <_UserExceptionVector+9>
End of assembler dump.

It seems to be working nicely (no entry instructions etc).

So I think the next step is to actually try and trigger some interrupts; Which level one interrupts are easiest to trigger? Will I have to play with the interrupt matrix peripheral?

[arjanmels]

I think this one is closed with the merge of #12 .