Xtensa vectored interrupts

[MabezDev]

This only adds support for Xtensa, RISCV will be in a follow-up PR. Quite a biggy, so best reviewed commit by each commit.

Summary

• Vectoring is enabled by default, as this is what most embedded Rust folks expect
• CPU interrupts are dispatched from xtensa-lx-rt see CPU Interrupts xtensa-lx-rt#45
• Only three priorities are available in Xtensa, and currently just one with this issue: Low Level interrupts cannot preempt eachother? xtensa-lx-rt#46. We may be able to supplement this with some software prioritization of interrupts (the Xtensa handbook eludes to this). Would like to hear some ideas on how to do this efficiently.
• For each level and for each edge interrupt, one CPU interrupt at that level is reserved for handling interrupts, these are currently Interrupt10EdgePriority1, Interrupt22EdgePriority3, Interrupt1LevelPriority1, Interrupt19LevelPriority2 & Interrupt23LevelPriority3.
• All handling code runs out of ram, vectored handlers can opt into running from ram with the #[ram] attribute

Issues

• Should enable(core: Cpu, interrupt: Interrupt, which: CpuInterrupt) & disable(core: Cpu, interrupt: Interrupt) be made unsafe? They may break vectoring unless you know what you're doing.
  • Maybe the CPU interrupts should be cfg'd away when vectoring is enabled?
  • It's quite hard to do this check actually, because vectoring is built on top of the basic interrupt stuff, black listing interrupts there makes vectoring fail too 🤔
• Needs a CS impl from Xtensa CS implementation #90 but we can probably use the xtensa_lx::mutex's for now
• Can this approach be used in esp-wifi, instead of raw level handlers? Note that the relevant WiFi and Bluetooth interrupts were added in the PACS in Add WiFi and bluetooth interrupts for each chip esp-pacs#26.
• Need to fix CI, which will involve carrying oversome changes from Xtensa CS implementation #90
• Wait until we release new xtensa-lx-rt version so we don't merge with patched deps
• Update examples to use vectored interrupt handlers
• Add examples of using non-vectoring? It will require some CI work, see smartled feature shouldn't be activated by default #105
  • Maybe we could add an example using the high-level interrupts (4-7) in another PR

[MabezDev]

This is now ready for review. @bjoernQ do you think you could try and use the vectored interrupts in esp-wifi and report back?

[bjoernQ]

This is now ready for review. @bjoernQ do you think you could try and use the vectored interrupts in esp-wifi and report back?

After looking into it I don't know how to do this

While the WiFi driver calls set_intr CPU 0 SRC 0 NUM 0 PRIO 1 - so we could map that interrupt but for BLE it seems the driver just maps the interrupt on it's own

Following interrupts are used by WiFi and BLE

• Interrupt0LevelPriority1
• Interrupt7SoftwarePriority1
• Interrupt8LevelPriority1
• Interrupt9LevelPriority1

[bjoernQ]

The GPIO interrupt example looks good but unfortunately timer_interrupt and serial_interrupts examples don't work anymore. For timer_interrupt it uses a reserved interrupt but even after changing it, it still doesn't work anymore

Probably we could change them to use vectored interrupts but we should keep one of the examples using the low-level way so we can always verify it is still working

Only tested on ESP32 so far

[bjoernQ]

It's quite hard to do this check actually, because vectoring is built on top of the basic interrupt stuff, black listing interrupts there makes vectoring fail too

Maybe the low-level function could be renamed and made private and we add a new public function with the old name and signature which does the check?

[bjoernQ]

For esp-rs/xtensa-lx-rt#46

We could try to change the interrupt handling to allow higher priority interrupt levels while executing the Rust ISR or the ISR could opt-in to get interrupted by lowering PS.INTLEVEL?

[MabezDev]

Thanks for the review!

While the WiFi driver calls set_intr CPU 0 SRC 0 NUM 0 PRIO 1 - so we could map that interrupt but for BLE it seems the driver just maps the interrupt on it's own
Following interrupts are used by WiFi and BLE
Interrupt0LevelPriority1
Interrupt7SoftwarePriority1
Interrupt8LevelPriority1
Interrupt9LevelPriority1

Hmm, this would be fine but we need an entry for the interrupt in the INTERRUPT_LEVELS static. It would be really nice to remove this, but AFAICT there is no way to see what (peripheral) interrupts are enabled at a given priority. I think we could get away with 'remapping' the interrupt with the rust code, just get an entry in INTERRUPT_LEVELS.

Probably we could change them to use vectored interrupts but we should keep one of the examples using the low-level way so we can always verify it is still working

I don't think it will be possible to mix vectoring interrupts & non-vectoring interrupts in their current state, as vectoring requires registering all the level handlers to go to one handler. One thing I can see is not registering levels 4 through 7, and only using vectoring for 'low-level' interrupts (1 through 3). Given that we have this limitation in xtensa-lx-rt already it sort of makes sense. Then high prio interrupts could be written in the low level way. To write an example using the low-level handlers would require no-default-features passed when building in CI though, as vectoring is enabled by default.

Maybe the low-level function could be renamed and made private and we add a new public function with the old name and signature which does the check?

This is a good idea, I'll try this out, thanks!

We could try to change the interrupt handling to allow higher priority interrupt levels while executing the Rust ISR or the ISR could opt-in to get interrupted by lowering PS.INTLEVEL?

This sounds good! Having at least a couple of levels of preemption will be helpful. Only three priorities may be an issue if we ever want an RTIC port or something, but by that time we could look into software interrupt priorities.

[MabezDev]

Hmm, this would be fine but we need an entry for the interrupt in the INTERRUPT_LEVELS static. It would be really nice to remove this, but AFAICT there is no way to see what (peripheral) interrupts are enabled at a given priority. I think we could get away with 'remapping' the interrupt with the rust code, just get an entry in INTERRUPT_LEVELS.

Actually, I guess I can just read the mapping registers, see what CPU interrupt it's assigned to a have a lookup table of priorities there... I'll give that a try

[bjoernQ]

I don't think it will be possible to mix vectoring interrupts & non-vectoring interrupts in their current state, as vectoring requires registering all the level handlers to go to one handler.

Ah sure - I probably need more coffee 😄

One thing we shouldn't forget: get rid of the patch sections in the TOMLs ... just mentioning that because those things often happen to me

[jessebraham]

Still need to dig into the interrupt/xtensa.rs file a bit more and do some local testing, but mostly looks fine to me. Left a few comments, not all necessarily require action.

[MabezDev]

Rebased on the main, a couple of things are left:

• New xtensa-lx-rt release, should be covered by allow interrupt level preemption xtensa-lx-rt#47
• Can this be used within eps-wifi? I've removed the limitation of having to call enable_with_priority so in theory the interrupts the blobs assign should now be serviced too.

[bjoernQ]

PR in xtensa-lx-rt is reviewed and merged!

[bjoernQ]

Can this be used within eps-wifi? I've removed the limitation of having to call enable_with_priority so in theory the interrupts the blobs assign should now be serviced too.

There is one thing missing unfortunately (before I can actually try it): The task-scheduler needs a mutable reference to the trap frame - we don't have that for vectored interrupts yet

Besides that, it looks quite nice and convenient - can't wait to get that for RISCV

Two last things:

• we need to adapt all the interrupt related examples - if I'm not wrong only the gpio example is adapted right now?
• the vectored feature needs to be exposed by the chip specific HALs

[MabezDev]

There is one thing missing, unfortunately (before I can actually try it): The task-scheduler needs a mutable reference to the trap frame - we don't have that for vectored interrupts yet

Hmm, we would need to change the handler generation in the pacs, right? Or maybe we can be cheeky and accept both handlers with args and without? We can (ab)use the linker to allow interrupt handlers to either have no arguments or the context. I don't think that will cause issues, provided we always call the handler with the arguments.

[bjoernQ]

We can (ab)use the linker to allow interrupt handlers to either have no arguments or the context. I don't think that will cause issues, provided we always call the handler with the arguments.

I also think that should work fine

[MabezDev]

With the latest commit it's now possible to pass the context into interrupt handlers:

.

#[interrupt]
fn GPIO(frame: &mut xtensa_lx_rt::exception::Context) {
    unsafe {
        esp_println::println!(
            "GPIO Interrupt with priority {}",
            xtensa_lx::interrupt::get_level()
        );
        esp_println::println!(
            "{:?}",
            frame
        );

        (&BUTTON).lock(|data| {
            let mut button = data.borrow_mut();
            let button = button.as_mut().unwrap();
            button.clear_interrupt();
        });
    }
}

The frame can be omitted if its not needed.

[bjoernQ]

Still trying to get esp-wifi working with it but this

esp-hal/esp-hal-procmacros/src/lib.rs

Line 184 in 0f1aaaf

crate::pac::Interrupt::#ident_s;

Creates problems since crate is then not the HAL ... gives weird errors like help: a similar path exists: esp32_hal::pac for the macro invocation

[bjoernQ]

🎉 I got at least the WiFi example working with the vectored interrupts ... need to look into BT-BLE but generally I'd say it should work

[bjoernQ]

Oh, not that easy for BLE ... it needs Interrupt7SoftwarePriority1 but I can only register peripheral interrupts?

Ah yes I see: cpu_interrupt_nr_to_cpu_interrupt_handler always returns None ... so, we need a way to register handlers for CPU interrupts 😢

[jessebraham]

I'm fine with pushing off examples to a later PR, if you'd like to get this wrapped up.

[bjoernQ]

BTLE is not yet fully working ... but I think it should and the reason it's not working is most probably not because of the vectored interrupts

[bjoernQ]

I'm fine with pushing off examples to a later PR, if you'd like to get this wrapped up.

I'm not so sure about that ... I mean the examples will just not work at all if they don't get adjusted - ideally we should never have such a situation on the main branch

[MabezDev]

I'm not so sure about that ... I mean the examples will just not work at all if they don't get adjusted - ideally we should never have such a situation on the main branch

I'm going to fix the old examples to use vectoring, I think Jesse is referring to my last comment about a raw example using levels 4-7.

[jessebraham]

Yes sorry I should have been more clear, I expect the existing examples to continue to work but adding new ones can be deferred for now.

[bjoernQ]

LGTM

[jessebraham]

Thanks for all your work on this!

[jrmoulton]

It seems that this PR broke the esp32c3-hal. The Hal won't build unless I revert back to the commit before this one. Is anyone else having the same issue?

[jrmoulton]

Never mind. The lock file was causing problems