RISC-V: Support separate firmware and kernel payload

[michaeljclark]

Support for separate firmware and kernel payload is added
by updating BBL to read optional preloaded kernel address
attributes from device-tree using a similar mechanism to
that used to pass init ramdisk addresses to linux kernel.

chosen {
    riscv,kernel-start = <0x00000000 0x80200000>;
    riscv,kernel-end = <0x00000000 0x80590634>;
};

These attributes are added by QEMU and read by BBL when combining
-bios <firmware-image> and -kernel <kernel-image> options. e.g.

$ qemu-system-riscv64 -machine virt -bios bbl -kernel vmlinux

With this change, bbl can be compiled without --with-payload
and the dummy payload alignment is altered to make the memory
footprint of the firmware-only bbl smaller. The dummy payload
message is updated to indicate the alternative load method.

This load method could also be supported by a first stage boot
loader that reads seperate firmware and kernel from SPI flash.
The main advantage of this new mechanism is that it eases kernel
development by avoiding the riscv-pk packaging step after kernel
builds, makes building per repository artefacts for CI simpler,
and mimics bootloaders on other platforms that can load a kernel
image file directly. Ultimately BBL should use an SPI driver to
load the kernel image however this mechanism supports use cases
such such as QEMU's -bios, -kernel and -initrd options following
examples from other platforms that pass kernel entry to firmware
via device-tree.

Cc: @palmer-dabbelt
Cc: @alistair23

[michaeljclark]

Here's the QEMU PR:

• RISC-V: Support separate firmware and kernel payload riscvarchive/riscv-qemu#145

Here's a QEMU branch to test this PR:

• https://github.com/riscv/riscv-qemu/tree/wip-firmware-kernel

[michaeljclark]

There is a bug in #ifdef to relax the alignment if --with-payload is not selected. I didn't realize DUMMY_PAYLOAD_ENABLED is also set when --with-payload is used. I can disable the feature to relax the alignment, but it will results in the kernel kernel being loaded at 0x8040_0000 instead of 0x8020_0000, as QEMU calculated the kernel load address based on the firmware size, and when the firmware has megapage alignment in its internal payload, the end address is > 0x8020_0000

$ more bbl/payload.S 
#include "config.h"
#include "encoding.h"

  .section ".payload","a",@progbits

#ifdef DUMMY_PAYLOAD_ENABLED
  /* align payload minimally */
  .align 3
#else
  /* align payload to the next megapage */
  .align RISCV_PGSHIFT + RISCV_PGLEVEL_BITS
#endif

  .globl _payload_start, _payload_end
_payload_start:
  .incbin BBL_PAYLOAD
_payload_end:

Extract from config.h when --with-payload=../../riscv-linux/vmlinux is specified.

/* Define if subproject MCPPBS_SPROJ_NORM is enabled */
#define DUMMY_PAYLOAD_ENABLED /**/

It seems DUMMY_PAYLOAD_ENABLED is always defined. I might need to do some configure.ac hacking to get the "relaxed-alignment-when-firmware-only-mode" feature working.

[michaeljclark]

It seems the DUMMY_PAYLOAD_ENABLED macro was defined due to the precense of the submodule. I've updated bbl/bbl.ac to define RELAXED_ALIGNMENT=1 when dummy_payload is enabled and RELAXED_ALIGNMENT=0 when an external payload is being used:

AC_ARG_WITH([payload], AS_HELP_STRING([--with-payload], [Set ELF payload for bbl]),
  [
   AC_SUBST([BBL_PAYLOAD], $with_payload, [Kernel payload for bbl])
   AC_DEFINE(RELAXED_ALIGNMENT,[0],[Use relaxed payload alignment])
  ], [
   AC_SUBST([BBL_PAYLOAD], [dummy_payload], [Kernel payload for bbl])
   AC_DEFINE(RELAXED_ALIGNMENT,[1],[Use relaxed payload alignment])
  ])

I've tested alignment with the dummy payload and external kernel (RELAXED_ALIGNMENT=1) using -bios bbl -kernel vmlinux. The internal payload alignment is relaxed and QEMU loads the kernel at 0x8020_0000. I've also tested with internal payload (RELAXED_ALIGNMENT=0) using -kernel bbl_vmlinux and everything is now fine.

This is the output if the dummy payload is run on its own:

$ ./riscv64-softmmu/qemu-system-riscv64 -nographic -machine virt -kernel qemu-images/bbl.bios 
bbl loader
This is bbl's dummy_payload.  To boot a real kernel, reconfigure bbl
with the flag --with-payload=PATH, then rebuild bbl. Alternatively,
bbl can be used in firmware-only mode by adding device-tree nodes
for an external payload and use QEMU's -bios and -kernel options.

    chosen {
        riscv,kernel-start = <payload_start>;
        riscv,kernel-end = <payload_end>;
    };

Power off

Squashed the changes back into the original commit.

[palmer-dabbelt]

I like this. My one question is: is there a standard way of providing the kernel to the firmware so we can avoid defining RISC-V specific chosen nodes (`riscv,kernel-start')?

If not then I vote we merge it.

[michaeljclark]

There are a couple of other boards that do this but they use qemu,boot-kernel with 4 cells, 2 for start and 2 for size. Instead, I chose to name the nodes riscv, so that this mechanism could potentially be used outside of QEMU. i.e. if there was a first stage loader that could load 2 blobs instead of 1. I also followed the convention of linux's initrd nodes, which specifiy start and end vs start and size. i.e.

   chosen {
        riscv,kernel-start = <payload_start>;
        riscv,kernel-end = <payload_end>;
        linux,initrd-start = <initrd_start>;
        linux,initrd-end = <initrd_end>;
    };

There is symmetry with the Linux mechanism, with the difference that the riscv,kernel nodes are to be consumed by the bootloader and the linux,initrd nodes are to be consumed by the kernel.

The choice to not name them with a qemu, prefix was so thay they could potentially be used by hardware (however for hardware it might be more likely that we add SPI/MMC loader and possible filesystem drivers to bbl). lowRISC have a BSD licensed ff.c (fat filesystem) along with SPI/SD code in their fork of bbl:

• https://github.com/lowRISC/lowrisc-kc705/tree/ab3e216c9369704a96cf23fdd21a55d656045395/driver

@terpstra doesn't like the idea of using FAT, but the reality is that it is pretty much universally supported. i.e. Mac and Windows users can just insert a SDCard adapter and drop a BOOT.BIN and VMLINUX.BIN onto a simple universally supported filesystem for firmware loading. This is what most other vendors do. The current hardware mechanism is tricky i.e. requires user to have Linux. i.e. power-user, and is somewhat not aligned with pointy-clicky IDE types. It's going to happen anyway, when UEFI gets ported to RISC-V.

In any case, as you see from the comments in the associated QEMU change, we'll need to modify the boot protocol to handle kernels that decompress themselves. Putting the FDT right after the kernel doesn't support kernels that have header code that decompresses the kernel (like linux does on platforms that support vmlinuz). The initrd is placed halfway into memory and I think linux-kernel has code to reserve or free this memory.

We still have quite a bit of work to do on early boot protocol stuff, like potentially changing the kernel to use 4KiB pages to map itself so that we can add kernel memory protection i.e. map .rodata +R
text +RX and data +RW

[palmer-dabbelt]

I agree: calling the nodes qemu,* is no better than calling them riscv,*. I suppose one resulting architecture could be BBL continuing to exist, but existing as a light-weight firmware that lives between a proper bootloader (which can get the kernel image from anywhere) and the kernel.

FYI: Wes is on vacation for a bit, so if you want to merge some FAT32 patches now's your chance :)

[michaeljclark]

Agree. Calling them riscv,* means we could actually use them outside of QEMU and IMHO is better.

BTW We haven't added this mechanism to sifive_u because we want it to match hardware. At this point we are only adding it to virt. If it is supported by loaders available on U series hardware we could add it to sifive_u, but given we're pointing bbl at a kernel, it implies FSBL support.

That said, it might make development easier if we did add it to sifive_u.

@alistair23 is working on SPI emulation for SiFive SPI so we can plug an emulated SD card into sifive_u and this will then allow boot loader development in QEMU.

[michaeljclark]

This is the patch to add it to sifive_u
sifive_u.patch.txt

[alistair23]

Overall this looks good to me. I think it moves in the right direction. There are some nit picks in the actual commit to better match QEMU style. Do you want me to comment here or wait until you send it to the mailing list?

I have had to put the SPI work on the back burner for the moment unfortunately. Linux can probe it can't detect the rootFS. Is someone working on SPI boot loader support at the moment? Like I mentioned the other day a non-Linux test case would help, but I still can't find one.

[michaeljclark]

Okay i've merged the associated commit to QEMU for the virt machine.

@alastair23 this is the riscv-pk PR. We can fix up QEMU nits when we send it for patch review. The PR just gets the commit into into the qemu-2.13-for-upstream branch which we need to send to qemu-devel as the number of commits is growing...

[michaeljclark]

@alastair23 the QEMU commit passes scripts/checkpatch.pl. I'll send out the QEMU patch queue to qemu-devel today and we can rebase/revise based on feedback... it's now in the riscv-qemu tree.

[michaeljclark]

@alastair23 nice thing in QEMU is that we now get kernel symbols with -d in_asm as it loads an ELF kernel, so it might help with SPI debugging. i.e. adding it to sifive_u in your local branch might help with debugging. I can email you the patch...

[alistair23]

Ah, that might help. Although the kernel doesn't hang, it loops around and I think I know where. Are you going to patch all of the machines?

[michaeljclark]

We could add it to sifive_u and spike_v1.10. Those are the only other two device-tree machines. sifive_e and spike_v_1.9.1 just take a single payload as they don't use device-tree.

[michaeljclark]

@alistair23 you might find the lowRISC bbl fork useful for SPI testing. They have a tiny SD card driver i.e. you could modify bbl to exercise the SiFive SPI emulation in QEMU. Might be a simpler test harness than a full blown kernel. See here https://github.com/lowRISC/lowrisc-kc705/tree/ab3e216c9369704a96cf23fdd21a55d656045395/driver

[alistair23]

Thanks for the link, unfortunately that driver is for the Xilinx SPI device, not the SiFive one.