Run RPi OS on qemu

[s-matyukevich]

No description provided.

[lizhuohua]

I'd like to share some of my experience.
The first thing is to install the latest QEMU (2.12.0), which supports raspi3 machine model. It only has limited support for BCM2837 peripherals, so some changes for RPi OS are needed.

1. QEMU doesn't support config.txt, so we need to follow the default booting procedure.
   • The kernel is loaded to address 0x80000 instead of 0x0. Simply modifying linker.ld should work.
   • The processor runs at EL2 instead of EL3. boot.S should be adapted to configure spsr_el2 and elr_el2.
   • Make an identify mapping to avoid "prefetch abort".
2. QEMU only supports UART0 (PL011), UART1 (mini UART) is not sent to stdio.
3. QEMU doesn't emulate the system timer. I use ARM generic timer to generate timer interrupts.

[s-matyukevich]

Thanks a lot @lizhuohua! Looks like it is more work than I originally envisioned. I'll probably add an optional exercise to port RPi OS on qemu to all of the lessons.

[cirosantilli]

Here is a QEMU / gem5 Buildroot non-RPI (-M virt) setup that might be of interest to people on this thread: https://github.com/cirosantilli/linux-kernel-module-cheat

Here is another repo of interest: https://github.com/s-matyukevich/raspberry-pi-os which just worked on QEMU, minimal getting started: https://raspberrypi.stackexchange.com/questions/34733/how-to-do-qemu-emulation-for-bare-metal-raspberry-pi-images/85135#85135

Step debugging the source code mentioned in this repo is something I will do later on.

[chishiro]

@lizhuohua QEMU (2.12.0) also supports UART1 (mini UART). Please check the following command in lesson02.
$ qemu-system-aarch64 -m 128 -M raspi3 -serial null -serial mon:stdio -nographic -kernel build/kernel7.elf

[lizhuohua]

@chishiro Great, it works! Many thanks!

[evopen]

I can't get qemu's four cores to work. There should be 4 cores enabled by default. I use the same code for pi plus additional modifications. But I only get one core's result. Is there anything I missed?

[s-matyukevich]

As far as I understand, for now quemu always run in a single core mode when emulating ARM in top of x86. https://lwn.net/Articles/697265

[evopen]

@s-matyukevich But I checked the original patch, there is a definition for default core number. And I tried smp option with no luck.

[bl4ckout31]

I did some testing with my solution for exercice 1.3.

Loading the kernel to address 0x0 works fine with kernel.img and made me able to boot with src/kernel.elf aswell wich was not possible with address 0x80000.
Like @evopen, I got only one core running when booting with kernel.img but all 4 cores were running when booting with src/kernel.elf.

I got similar results with @evopen solution too.

[s-matyukevich]

@evopen @bl4ckout31 I am not sure about the CPU issue. Maybe multiple CPUs are supported only when you run qemu in KVM mode using some ARM device? You can try to install qemu on Raspbian and test it from there. (I personally think that emulating Raspberry PI from Raspberry PI is definitely cool :) )
I will also take a look at it, but later.

[db47h]

I did not see it mentioned explicitly, but with qemu you need to use the kernel ELF file instead of the raw img. That way qemu will properly load your kernel at address 0. See upcoming comment...

Another solution for using the mini UART in qemu while keeping access to the monitor and GUI is this:

qemu-system-aarch64 -m 128 -M raspi3 -cpu cortex-a53 -kernel kernel8.img \
                    -serial null -serial vc

In the view menu you'll now have a serial1 (the mini UART). Like @chishiro did, I just disabled serial0, but you could leave it enabled by replacing the first -serial null by -serial vc.

Here's also a no GUI option that allows you to retain access to the qemu monitor. Since it makes use of multiple terminals, you'll want tu use tmux or screen obviously.

Start with a terminal that will serve as your serial terminal:

# create a pair of FIFOs for mini_uart (do this only once)
mkfifo uart1.{in,out}
# redirect the terminal IO to the FIFOs
cat <uart1.out >&1 & cat <&0 >uart.in

Then run qemu in another terminal:

qemu-system-aarch64 -m 128 -M raspi3 -cpu cortex-a53 -kernel kernel8.img -nographic \
                    -serial null -chardev pipe,id=uart1,path=uart1 -serial chardev:uart1

The qemu monitor will be accessible from this terminal.

[evopen]

I think we should not use elf file for qemu. Although it runs as expected, but it's actually running from 0x30 which is the entry point for elf file. And somehow it jump to 0x0.

EDIT: I gave up trying to find thre reason of multicore problem to img kernel. It seem qemu modifies my img kernel and add some startup code to put the other 3 cores to idle using WFE instruction. So the best solution is to use elf with ENTRY(_start) in ld script.

Maybe it's something to do with multiboot? idk

[NourElMenshawy]

Hello all
Before i saw this thread, my initial thought was to use QEMU for pi also combined it with CMake build system to make it more generic
You can find it here
https://github.com/NourElMenshawy/RaspberrypiOS.git
For sure this a very primitive approach so if anyone would like to highlight better approaches or more advance techniques, I would be grateful

[aaronjanse]

Hey, here's a way to run QEMU without needing mkfifo incantations:

qemu-system-aarch64 -m 128 -M raspi3 -cpu cortex-a53 -kernel kernel8.img -nographic \
                    -serial null -chardev stdio,id=uart1 -serial chardev:uart1 -monitor none

[stefanJi]

Hey, here's a way to run QEMU without needing mkfifo incantations:

qemu-system-aarch64 -m 128 -M raspi3 -cpu cortex-a53 -kernel kernel8.img -nographic \
                    -serial null -chardev stdio,id=uart1 -serial chardev:uart1 -monitor none

When I running your command in osx with lesson01-exercise1, I got this cpu statistic.(use top command). I am confused why all the CPUs of the computer be run.

COMMAND	%CPU
qemu-system	397.1

[AlMazyr]

Hello.
For those of you who are still interested in how to start all 4 cores with kernel.img.
First of all, with the kernel.img file qemu boots platform in a different way. It loads additional bootloaders to memory. You can verify this with the help of qemu monitor:

(qemu) info roms
addr=00000000000000d8 size=0x000020 mem=ram name="raspi_spintables"
addr=0000000000000300 size=0x00002c mem=ram name="raspi_smpboot"
addr=0000000000000000 size=0x000028 mem=ram name="bootloader"
addr=0000000000080000 size=0x000548 mem=ram name="kernel.img"

Qemu sets the main core to addr 0 and all the other cores to 0x300.

(gdb) info threads
* 1    Thread 1.1 (CPU#0 [running]) 0x0000000000000000 in ?? ()
  2    Thread 1.2 (CPU#1 [running]) 0x0000000000000300 in ?? ()
  3    Thread 1.3 (CPU#2 [running]) 0x0000000000000300 in ?? ()
  4    Thread 1.4 (CPU#3 [running]) 0x0000000000000300 in ?? ()

So the main core starts to execute "bootloader" code. The code of the "bootloader" can be found in qemu sources. It simply branches to our "kernel.img".
Other cores start to execute "raspi_smpboot" code. Its code can be found at hw/arm/raspi.c:

     static const uint32_t smpboot[] = {                                                                                                                                                           
         0xd2801b05, /*        mov     x5, 0xd8 *                                                                                                                                                    
        0xd53800a6, /*        mrs     x6, mpidr_el1 */                                                                                                                                            
        0x924004c6, /*        and     x6, x6, #0x3 */                                                                                                                                                 
        0xd503205f, /* spin:  wfe */                                                                                                                                                               
        0xf86678a4, /*        ldr     x4, [x5,x6,lsl #3] */                                                                                                                                         
        0xb4ffffc4, /*        cbz     x4, spin */                                                                                                                                                    
        0xd2800000, /*        mov     x0, #0x0 */                                                                                                                                                     
        0xd2800001, /*        mov     x1, #0x0 */                                                                                                                                                    
        0xd2800002, /*        mov     x2, #0x0 */                                                                                                                                                    
        0xd2800003, /*        mov     x3, #0x0 */                                                                                                                                                   
        0xd61f0080, /*        br      x4 */                                                                                                                                                         
     };       

Cores spin on "raspi_spintables". To wake up cores we need to do 2 things.

1. We have to load starting address at the correct place in the "raspi_spintables".
2. We have to use SEV instruction to wake up core.
   The address for each core must be calculated like this:
   addr = 0xd8 + mpidr_el1 * 8

I prepared function which wakes up core:

.globl wakeup_core
wakeup_core:
        mov x2, 0xd8
        str x1, [x2, x0, LSL #3]
        sev
        ret

It gets 2 arguments. x0 is for mpidr_el1 value and x1 is for starting address.
It works fine for me.

[AlMazyr]

Hello.
For those of you who are still interested in how to start all 4 cores with kernel.img.
First of all, with the kernel.img file qemu boots platform in a different way. It loads additional bootloaders to memory. You can verify this with the help of qemu monitor:

(qemu) info roms
addr=00000000000000d8 size=0x000020 mem=ram name="raspi_spintables"
addr=0000000000000300 size=0x00002c mem=ram name="raspi_smpboot"
addr=0000000000000000 size=0x000028 mem=ram name="bootloader"
addr=0000000000080000 size=0x000548 mem=ram name="kernel.img"

Qemu sets the main core to addr 0 and all the other cores to 0x300.

(gdb) info threads
* 1    Thread 1.1 (CPU#0 [running]) 0x0000000000000000 in ?? ()
  2    Thread 1.2 (CPU#1 [running]) 0x0000000000000300 in ?? ()
  3    Thread 1.3 (CPU#2 [running]) 0x0000000000000300 in ?? ()
  4    Thread 1.4 (CPU#3 [running]) 0x0000000000000300 in ?? ()

So the main core starts to execute "bootloader" code. The code of the "bootloader" can be found in qemu sources. It simply branches to our "kernel.img".
Other cores start to execute "raspi_smpboot" code. Its code can be found at hw/arm/raspi.c:

     static const uint32_t smpboot[] = {                                                                                                                                                           
         0xd2801b05, /*        mov     x5, 0xd8 *                                                                                                                                                    
        0xd53800a6, /*        mrs     x6, mpidr_el1 */                                                                                                                                            
        0x924004c6, /*        and     x6, x6, #0x3 */                                                                                                                                                 
        0xd503205f, /* spin:  wfe */                                                                                                                                                               
        0xf86678a4, /*        ldr     x4, [x5,x6,lsl #3] */                                                                                                                                         
        0xb4ffffc4, /*        cbz     x4, spin */                                                                                                                                                    
        0xd2800000, /*        mov     x0, #0x0 */                                                                                                                                                     
        0xd2800001, /*        mov     x1, #0x0 */                                                                                                                                                    
        0xd2800002, /*        mov     x2, #0x0 */                                                                                                                                                    
        0xd2800003, /*        mov     x3, #0x0 */                                                                                                                                                   
        0xd61f0080, /*        br      x4 */                                                                                                                                                         
     };       

Cores spin on "raspi_spintables". To wake up cores we need to do 2 things.

1. We have to load starting address at the correct place in the "raspi_spintables".

2. We have to use [SEV](https://developer.arm.com/docs/ddi0596/e/base-instructions-alphabetic-order/sev-send-event) instruction to wake up core.
   The address for each core must be calculated like this:
   **addr = 0xd8 + mpidr_el1 * 8**

I prepared function which wakes up core:

.globl wakeup_core
wakeup_core:
        mov x2, 0xd8
        str x1, [x2, x0, LSL #3]
        sev
        ret

It gets 2 arguments. x0 is for mpidr_el1 value and x1 is for starting address.
It works fine for me.

I tried your technique with the SMP cores flag but no luck, I'm pretty sure it didn't run my code.

You can check my solution for multicore boot. It works on both qemu and raspberry pi:
https://github.com/AlMazyr/raspberry-pi-os/tree/lesson01/src/lesson01

[rockytriton]

Thanks, I will try this soon, I was just noticing yesterday that the latest armstub code is putting the cores to sleep (at least that's what it seemed like it's doing to me): https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S

secondary_spin:
	wfe
	ldr x4, [x5, x6, lsl #3]
	cbz x4, secondary_spin
	mov x0, #0
	b boot_kernel