[FEATURE] Implement RPi 4B SMP

[linguini1]

Is your feature request related to a problem? Please describe.

The RPi 4B support currently does not have verified SMP support.

Describe the solution you'd like

I would like SMP support that allows users to leverage all four cores of the BCM2711. This will need to be verified using the SMP tests. I would also like a defconfig to be added to the list of presets so users can start with an SMP base configuration.

Describe alternatives you've considered

No response

Verification

• I have verified before submitting the report.

[linguini1]

With the existing SMP logic matching other arm64 platforms and using the attached defconfig, a few issues are noticed.

1. 100% CPU usage for IDLE tasks

- Ready to Boot Primary CPU                                                                                             
- Boot from EL2                                                                                                         
- Boot from EL1                                                                                                         
- Boot to C runtime for OS Initialize                                                                                   
[CPU0] nx_start: Entry                                                                                                  
[CPU0] up_allocate_heap: heap_start=0x0x4e0000, heap_size=0xfbb20000                                                    
[CPU0] gic_validate_dist_version: GICv2 detected                                                                        
[CPU0] uart_register: Registering /dev/console                                                                          
[CPU0] work_start_highpri: Starting high-priority kernel worker thread(s)                                               
[CPU0] nxtask_activate: hpwork pid=4,TCB=0x4e05a8                                                                       
[CPU0] nx_start_application: Starting init thread                                                                       
[CPU0] task_spawn: name=nsh_main entry=0x48df5c file_actions=0 attr=0x4d9f70 argv=0x4d9f68                              
[CPU0] nxtask_activate: nsh_main pid=5,TCB=0x4e2750                                                                     
[CPU0] lib_cxx_initialize: _sinit: 0x4b4000 _einit: 0x4b4000                                                            
[CPU0] nx_start: CPU0: Beginning Idle Loop                                                                              
                                                                                                                        
NuttShell (NSH) NuttX-12.12.0                                                                                           
nsh> ps                                                                                                                 
  TID   PID  PPID CPU PRI POLICY   TYPE    NPX STATE    EVENT     SIGMASK            STACK    USED FILLED COMMAND       
    0     0     0   0   0 FIFO     Kthread   - Assigned           0000000000000000 0008144 0002032  24.9%  CPU0 IDLE    
    1     0     0   1   0 FIFO     Kthread   - Running            0000000000000000 0008144 0008144 100.0%! CPU1 IDLE    
    2     0     0   2   0 FIFO     Kthread   - Running            0000000000000000 0008144 0008144 100.0%! CPU2 IDLE    
    3     0     0   3   0 FIFO     Kthread   - Running            0000000000000000 0008144 0008144 100.0%! CPU3 IDLE    
    4     0     0 --- 192 RR       Kthread   - Waiting  Semaphore 0000000000000000 0008080 0001072  13.2%  hpwork 0x4c38
    5     5     0   0 100 RR       Task      - Running            0000000000000000 0008112 0003584  44.1%  nsh_main  

2. getprime hangs when completed

nsh> getprime                                                                                                           
[CPU0] task_spawn: name=getprime entry=0x4a3be0 file_actions=0x4e49f8 attr=0x4e4a00 argv=0x4e4ba0                       
[CPU0] spawn_execattrs: Setting policy=2 priority=50 for pid=6                                                          
[CPU0] nxtask_activate: getprime pid=6,TCB=0x4e5128                                                                     
[CPU0] nxtask_activate: getprime pid=7,TCB=0x4e7750                                                                     
Set thread priority to 10                                                                                               
Set thread policy to SCHED_RR                                                                                           
Start thread #0                                                                                                         
thread #0 started, looking for primes < 10000, doing 10 run(s)                                                          
thread #0[CPU0] nx_pthread_exit: exit_value=0                                                                           
[CPU0] pthread_completejoin: pid=7 exit_value=0                                                                         
[CPU0] pthread_join: Returning 0, exit_value 0                                                                          
[CPU0] nxtask_exit: getprime pid=6,TCB=0x4e5128                                                                         
[CPU0] group_drop: Keep group 0x4e5288 (waiters > 0)                                                                    
[CPU0] nxtask_exit: getprime pid=7,TCB=0x4e7750                                                                         
 finished, found 1230 primes, last one was 9973                                                                         
Done                                                                                                                    
getprime took 358 msec                                                                                                  

3. ostest hangs at the spinlock test

...
End of test memory usage:                                                                                               
VARIABLE  BEFORE   AFTER                                                                                                
======== ======== =[CPU0] nxtask_activate: ostest pid=331,TCB=0x4e9e50                                                  
=======                                                                                                                 
arena    fbb20000 fbb20000                                                                                              
ordblks         6        6                                                                                              
mxordblk fbb13ea0 fbb13ea0                                                                                              
uordblks     9f60     9f60                                                                                              
fordblks fbb160a0 fbb160a0                                                                                              
                                                                                                                        
user_main: spinlock test                                                                                                
Start Lock test:                                                                                                        
Thread num: 1, Loop times: 10000000                                                                                     
                                                                                                                        
Test type: spinlock                                                                                                     

SMP defconfig

Details

#
# This file is autogenerated: PLEASE DO NOT EDIT IT.
#
# You can use "make menuconfig" to make any modifications to the installed .config file.
# You can then do "make savedefconfig" to generate a new defconfig file that includes your
# modifications.
#
CONFIG_ARCH="arm64"
CONFIG_ARCH_ARM64=y
CONFIG_ARCH_BOARD="raspberrypi-4b"
CONFIG_ARCH_BOARD_RASPBERRYPI_4B=y
CONFIG_ARCH_CHIP="bcm2711"
CONFIG_ARCH_CHIP_BCM2711=y
CONFIG_ARCH_EARLY_PRINT=y
CONFIG_ARCH_INTERRUPTSTACK=4096
CONFIG_ARCH_IRQPRIO=y
CONFIG_BOARD_LOOPSPERMSEC=132954
CONFIG_BUILTIN=y
CONFIG_DEBUG_ASSERTIONS=y
CONFIG_DEBUG_FEATURES=y
CONFIG_DEBUG_FULLOPT=y
CONFIG_DEBUG_IPC=y
CONFIG_DEBUG_IPC_ERROR=y
CONFIG_DEBUG_IPC_WARN=y
CONFIG_DEBUG_IRQ=y
CONFIG_DEBUG_IRQ_ERROR=y
CONFIG_DEBUG_IRQ_WARN=y
CONFIG_DEBUG_MM=y
CONFIG_DEBUG_MM_ERROR=y
CONFIG_DEBUG_MM_WARN=y
CONFIG_DEBUG_SCHED=y
CONFIG_DEBUG_SCHED_ERROR=y
CONFIG_DEBUG_SCHED_WARN=y
CONFIG_DEBUG_SYMBOLS=y
CONFIG_DEFAULT_TASK_STACKSIZE=8192
CONFIG_EXAMPLES_HELLO=y
CONFIG_EXPERIMENTAL=y
CONFIG_FS_PROCFS=y
CONFIG_FS_ROMFS=y
CONFIG_HAVE_CXX=y
CONFIG_HAVE_CXXINITIALIZE=y
CONFIG_IDLETHREAD_STACKSIZE=8192
CONFIG_INIT_ENTRYPOINT="nsh_main"
CONFIG_INTELHEX_BINARY=y
CONFIG_NCPUS=1
CONFIG_NSH_ARCHINIT=y
CONFIG_NSH_BUILTIN_APPS=y
CONFIG_NSH_FILEIOSIZE=512
CONFIG_NSH_READLINE=y
CONFIG_PREALLOC_TIMERS=4
CONFIG_PTHREAD_STACK_MIN=8192
CONFIG_RAMLOG=y
CONFIG_RAM_SIZE=4227858432
CONFIG_RAM_START=0x00000000
CONFIG_RAW_BINARY=y
CONFIG_READLINE_CMD_HISTORY=y
CONFIG_RR_INTERVAL=200
CONFIG_SCHED_HPWORK=y
CONFIG_SCHED_HPWORKPRIORITY=192
CONFIG_SMP=y
CONFIG_STACK_COLORATION=y
CONFIG_START_MONTH=11
CONFIG_START_YEAR=2022
CONFIG_SYMTAB_ORDEREDBYNAME=y
CONFIG_SYSTEM_NSH=y
CONFIG_SYSTEM_SYSTEM=y
CONFIG_SYSTEM_TIME64=y
CONFIG_TESTING_GETPRIME=y
CONFIG_TESTING_OSTEST=y
CONFIG_TESTING_SMP=y
CONFIG_USEC_PER_TICK=1000
CONFIG_USERLED=y

[linguini1]

None of this is an issue with qemu-armv8a:nsh_smp, which uses pretty much the same SMP implementation as the Pi 4B right now. I also tried qemu-armv8a:nsh_gicv2 modified to have SMP=y with four cores. This must be a quirk of "real" hardware to track down...