Boot in the Linux way

[tatetian]

Background and Motivation

The first three pieces of software that gets running after one presses the power button of a machine are

• Firmware (e.g., BIOS, UEFI);
• Bootloader (e.g., GRUB, Limine);
• OS kernel (e.g., Linux, OpenBSD, Windows).

Ideally, a bootloader should be able to load different types of OS kernels. This is called multi-booting. And the reversed direction is also desired: an OS kernel should also be able to get booted by different bootloaders. These two goals mean that the bootloaders and the kernels must agree upon an interface or protocol to interact with one another.

Currently, Jinux chooses to use the Limine boot protocol. And since the only bootloader that implements the protocol is Limine itself, Jinux is not compatible with any other bootloader.

The problem is that we do need to support in the near future a second bootloader, which is tdshim. Tdshim is a light-weight virtual firmware for Intel TDX VM that enables starting up confidential containers with lower latencies. It also acts as a bootloader. Tdshim only supports loading Linux kernels as well as ELF files.

As every open-source bootloader, including td-shim, supports booting Linux, I think it is a smart move for us to adopt the Linux boot protocol. This way, every bootloader that works with Linux can do with Jinux.

TODOs

• Adopt the Linux boot protocol. For regular VMs, we can choose GRUB, which also supports the Linux boot procotol. For TDX VMs, we use tdshim.
• Support the self-decompression format of vmlinuz/zImage to reduce the kernel image size. Tdshim is limited in the sizes of kernel images that can be loaded. So compression is a must.
• Parse kernel command-line arguments passed from bootloaders. Support of CLI arguments gives us the flexibility to alter the behavior of Jinux without re-compiling.

Learning resources

• Hands-on Booting: Learn the Boot Process of Linux, Windows, and Unix. This books provide a good overview of the boot process of machine, including how the firmware, the bootloader, and the kernel interacts with each other.
• Linux Insides. This tutorial describes how Linux kernel boots.

[junyang-zh]

I am somewhat interested in this topic. May assign it to me days later after I studied this subject (as well as multi-arch things). Now just mark it. :)

[tatetian]

I am somewhat interested in this topic. May assign it to me days later after I studied this subject (as well as multi-arch things). Now just mark it. :)

Great. Then the task is yours :) This task won't be easy: it requires a lot of low-level, in-depth knowledge about the bootloader and kernel. Take your time to learn relevant materials. I am almost certain that you will need to study the source code of Linux.

[junyang-zh]

Related to #281

[junyang-zh]

The first stage of implementing Linux boot #379 is completed by now. And we run into a problem that upstream GRUB efi cannot boot bzImage in QEMU directly by the linux command. The debian version of GRUB has a series of patch making GRUB boot the bzImage using the legacy efi-handover protocol. Trying to fix it now.

[junyang-zh]

There are multiple candidate boot methods used by linux, listing them now to help decide whether we should support:

• Loadfile2: seems most advanced, used by upstream GRUB 2.12-rc1;
• EFI handover: deprecated, but used by ubuntu as the only way;
• legacy 32 bit: supported well, but cannot use some UEFI functions after kernel loaded;
• PVH: used by QEMU.

Not clear which way does TDX-GRUB would recommend.

[junyang-zh]

#454 Implements the legacy 32 bit protocol, the Loadfile2 protocol may be another imminent goal.

[junyang-zh]

Syncing the current status of linux boot support.

Previously, Jinux is developed with Limine. Now it supports booting with Multiboot, Multiboot2 and Linux (legacy) 32-bit Boot Protocol. The support for EFI handover is on its way since TDVF needs it.

TLDR, now the compiled Jinux binary supports booting by GRUB with the following grub.cfg:

menuentry "jinux" {
	linux /boot/jinux init=/usr/bin/busybox -- sh -l
	initrd /boot/initramfs.cpio.gz
	boot
}

I am too lazy to translate the following document to English. Just excuse me with that.

Linux 启动的历史沿革

Linux x86 Boot Protocol 的历史大致有三个阶段：

1. 史前时代（< 1.3）：zImage/Image 格式内核，顾名思义，文件是内存的镜像，内核还只支持 32 位，内核被加载到 0x10000；
2. 标准时代（1.3 - 3.6）：Linux x86 Boot Protocol 被实质上标准化了，规定了一个 setup header，有了 bzImage；
3. 乱搞时代（> 3.6）：由于体系结构发展，UEFI/secureboot 广泛使用，虚拟化和轻量虚拟化应用，协议一团乱糟。

乱搞时代（现在），Linux Kernel 还支持这些启动方式：

• Loadfile2：来自 EFI specification 的启动方式，应该是未来的主流；
• Linux 64-bit boot protocol：这个被几个不主流的固件/Bootloader 使用，主要用来直接启动 vmlinux；
• Linux 32-bit boot protocol：这个是之前最主流的启动方式，对启动后 UEFI call 的支持不太完善；
• EFI handover 64：从 EFI STUB 进内核，仅被 Debian GRUB（基于 GRUB 2.06）使用，被 Linux 6.2 标记为弃用；
• EFI handover 32：类似上面那个，但是根本没人用，已经弃置了；
• XEN PVH：QEMU 的 -kernel 选项启动 vmlinux 用的是这个方式；
• 其他我可能不知道。

由于历史包袱，加上自解压/KASLR 等技术，x86 启动部分的代码还是异常庞杂的。

构建原理

偷两张幻灯片介绍下 Linux 的 bzImage (big zImage) 结构和构建：

目前我设计的 Jinux 支持 bzImage 格式，是这样构建的：

• jinux-frame 内新增一个二进制 Rust crate，jinux-x86-boot-setup，这个二进制使用 i386 格式编译，负责加载 jinux ELF；
• jinux-frame 的构建脚本在构建框的时候顺便把 jinux-x86-boot-setup 编译成 ELF；
• jinux-runner 把 jinux-x86-boot-setup 拍平转换成映像，和 jinux ELF 连接，并填写头中的字段，生成 jinuz。

Memory map

我实现的 Jinux 启动的 memory map 现在是这样，但是显然需要改善。

              ~                        ~
              |  Long-mode kernel      |	<- Jinux entrypoint 8001000, temporarily
     8000000  +------------------------+    <- Loaded by Jinux boot setup, should implement relocation
              |  Protected-mode heap   |
     7000000  +------------------------+
              |  Protected-mode kernel |
              |  as the Jinux setup &  |
              |  loader                |    <- Protected kernel entrypoint 100000
      100000  +------------------------+    <- Loaded by file offset 200 * (SETUP_SECTS + 1)
              |  I/O memory hole       |
      0A0000  +------------------------+
              |  Reserved for BIOS     |
              ~                        ~
       89000  |  Command line          |
     X+10000  +------------------------+
              |  Stack/heap            |    For the kernel real-mode code        -
     X+08000  +------------------------+                                          |
              |  16-bit kernel setup   |    The kernel real-mode code             } pretty useless now
              |  Kernel boot sector    |    The kernel legacy boot sector         | 
     X        +------------------------+                                         -
              |  Boot loader           |    <- Boot sector entry point 0000:7C00
      001000  +------------------------+
              |  Reserved for MBR/BIOS |
      000800  +------------------------+
              |  Typically used by MBR |
      000600  +------------------------+
              |  BIOS use only         |
      000000  +------------------------+