esp32: Re-organize the interrupt handling process to correctly handle the SMP case

[Ouss4]

Summary

This PR has 3 major changes:

• Simplify the interrupt API exported for other drivers. Mainly, this hides the details of allocating and attaching an interrupt. A driver will just have to "setup" and "teardown" an interrupt, the details are handled internally (this also helps with the next parts).
• Merge the different interrupt implementation into one file. This simplifies sharing variables and functions and removes the necessity of global, extern'd, variables.
• When possible, use the Interrupt Matrix to disable/enable an interrupt. Unlike the in-core INTENABLE register, the Interrupt Matrix can be accessed by both CPUs. Hence, manipulating interrupts can be done without inter-CPU calls.

Impact

ESP32 chip.

Testing

All of the in-tree ESP32 defconfigs were tested.

[Ouss4]

@masayuki2009 this is a draft just because I might create separate PRs for the different changes if required. Otherwise, the change is ready. It implements enabling/disabling interrupts using the Interrupt Matrix.

[masayuki2009]

@Ouss4

Hmm, I tried esp32-devkitc:wapi_smp but I was not able to connect to my Wi-Fi access point.

nsh> uname -a
NuttX  3.6.1 eaed2aaa94-dirty Aug 22 2021 01:18:26 xtensa esp32-devkitc
nsh> ps
  PID CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND
    0   0   0 FIFO     Kthread N-- Assigned           00000000 002016 000736  36.5%  CPU0 IDLE
    1   1   0 FIFO     Kthread N-- Running            00000000 002016 000608  30.1%  CPU1 IDLE
    2 --- 100 RR       Kthread --- Waiting  Semaphore 00000000 002080 000944  45.3%  lpwork 0x3ffc02b8
    3   0 100 RR       Task    --- Running            00000000 003120 001744  55.8%  init
    4 --- 223 RR       Kthread --- Waiting  Semaphore 00000000 002080 000544  26.1%  rt_timer
    5 --- 253 RR       Kthread --- Waiting  MQ empty  00000000 006704 001728  25.7%  wifi
    6 --- 100 RR       Task    --- Waiting  Semaphore 00000000 002064 000528  25.5%  Telnet daemon 0x3ffe3ef0
nsh> free
                   total       used       free    largest  nused  nfree
        Umem:     241984      58256     183728     110000    144      4
nsh> mount
  /mnt/esp/wifi type spiffs
  /proc type procfs
nsh> ifconfig
wlan0	Link encap:Ethernet HWaddr ac:67:b2:71:65:2c at UP
	inet addr:10.0.0.2 DRaddr:10.0.0.1 Mask:255.255.255.0

             IPv4   TCP   UDP  ICMP
Received     0000  0000  0000  0000
Dropped      0000  0000  0000  0000
  IPv4        VHL: 0000   Frg: 0000
  Checksum   0000  0000  0000  ----
  TCP         ACK: 0000   SYN: 0000
              RST: 0000  0000
  Type       0000  ----  ----  0000
Sent         0000  0000  0000  0000
  Rexmit     ----  0000  ----  ----
nsh> wapi psk wlan0 wifi-test-24g 1
nsh> wapi essid wlan0 raspi3-g 1

[Ouss4]

Hmm, I tried esp32-devkitc:wapi_smp but I was not able to connect to my Wi-Fi access point.

@masayuki2009 thanks for testing, let me check.

[masayuki2009]

@Ouss4
esp32-devkitc:wapi has the same problems too.

nsh> uname -a
NuttX  3.6.1 eaed2aaa94-dirty Aug 22 2021 01:30:30 xtensa esp32-devkitc
nsh> ps
  PID PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK   STACK   USED  FILLED COMMAND
    0   0 FIFO     Kthread N-- Ready              00000000 003040 000624  20.5%  Idle Task
    1 100 RR       Kthread --- Waiting  Semaphore 00000000 004128 000448  10.8%  lpwork 0x3ffb983c
    2 100 RR       Task    --- Running            00000000 004144 001744  42.0%  init
    3 223 RR       Kthread --- Waiting  Semaphore 00000000 002080 000448  21.5%  rt_timer
    4 253 RR       Kthread --- Waiting  MQ empty  00000000 006704 001696  25.2%  wifi
    5 100 RR       Task    --- Waiting  Semaphore 00000000 004112 000528  12.8%  Telnet daemon 0x3ffbe950
nsh> free
                   total       used       free    largest  nused  nfree
        Umem:     273248      62480     210768     135232    136      4
nsh> mount
  /mnt/esp/wifi type spiffs
  /proc type procfs
nsh> ifconfig
wlan0	Link encap:Ethernet HWaddr ac:67:b2:71:65:2c at UP
	inet addr:10.0.0.2 DRaddr:10.0.0.1 Mask:255.255.255.0

             IPv4   TCP   UDP  ICMP
Received     0000  0000  0000  0000
Dropped      0000  0000  0000  0000
  IPv4        VHL: 0000   Frg: 0000
  Checksum   0000  0000  0000  ----
  TCP         ACK: 0000   SYN: 0000
              RST: 0000  0000
  Type       0000  ----  ----  0000
Sent         0000  0000  0000  0000
  Rexmit     ----  0000  ----  ----
nsh> wapi psk wlan0 wifi-test-24g 1
nsh> wapi essid wlan0 raspi3-g 1

[Ouss4]

@masayuki2009 enabling the Wi-fi interrupt at the core level was lost in the middle of my rebasing. I added it again. Please check.

Since now we control peripheral interrupt only based on the Interrupt Matrix, the in-core interrupts has to be enabled at start-up. This is done during the setup of an interrupt, but since Wi-fi is setup with the external library, it has to be handled separately.

[masayuki2009]

@Ouss4
Now, the Wi-Fi works both in non-SMP and in SMP mode.

[Ouss4]

@masayuki2009 @gustavonihei I'm keeping the commits as they are to ease reviewing. Once that's done, I'll squash some of them.

[gustavonihei]

LGTM

[masayuki2009]

@Ouss4
The code looks good but I think the following code should be atomic which means that we need to use enter_critical_section() or spin_lock_irqsave().

cpu = up_cpu_index();
esp32_setup_irq(cpu, ...);

For example, if the code is not atomic, the currently running CPU would be switched to another CPU after calling up_cpu_index(). So please be careful when calling up_cpu_index().

[Ouss4]

@Ouss4
The code looks good but I think the following code should be atomic which means that we need to use enter_critical_section() or spin_lock_irqsave().

cpu = up_cpu_index();
esp32_setup_irq(cpu, ...);

For example, if the code is not atomic, the currently running CPU would be switched to another CPU after calling up_cpu_index(). So please be careful when calling up_cpu_index().

@masayuki2009 if I'm not mistaken all of those calls are within a critical section. I'll check again tomorrow from my computer.

[masayuki2009]

@masayuki2009 if I'm not mistaken all of those calls are within a critical section. I'll check again tomorrow from my computer.

@Ouss4
No problem, I will wait for your investigation results. And if it is OK, I will merge this PR.

[Ouss4]

@masayuki2009 I have checked that all the calls are from within a critical section. There is only the emac driver initialization and the serial DMA initialization that don't explicitly call enter_critical_section but that part is initialized early in the bring-up process when we still didn't start the other CPUs.

I have also done some squashing.