==== WORK IN PROGRESS ====

Working SMP/multicore support, there is still some work to be done.

End Goal

Perf Kernel, is a new domain-specific x64 kernel that focuses solely on performance and provides researchers with an easy way to move from user space to kernel space. This means there are no Spectre or Meltdown patches or any kind of user/kernel separation. Only one process and at most two threads per core are allowed. A classic interrupt based scheduler that reduces performance is not required. A cooperative scheduler is provided by the kernel, i.e. 'async/wait' can be used. The allocator is cache-optimised and the MMU uses large pages to reduce memory access latency. The kernel itself is built with sse and avx optimizations. Performance registers are enabled and accessible via KVM. A Perf-like tool produces performance statistics for the kernel code.

Setup & Debug Build

Clone the repo with submodules:

$ git clone --recursive <url>

Install the nix package manager.
The installation script requires that you have sudo access to root.

$ curl -L https://nixos.org/nix/install | sh

You may ask yourself why I use this weird package manager. The answer is simple: A completely reproducable and pinned development environment that works across every Linux distribution the same. Also through nix installed packages are contained and have no side effects on your system.

To download all required pinned dependencies just execute:

$ cd <project_root>
$ nix-shell shell.nix

Then install some cargo dependencies:

$ cd <project_root>
$ cargo install --path tools/glue_gun
$ rustup component add llvm-tools-preview rust-src

Now compile & run the kernel in qemu with:

$ cd <project_root>/kernel
$ cargo run

Integrated code editor

This projects ships with a customized & pinned vscodium (vscode without telemetry) with all the necessary addons. Included features are:

• in editor kernel debugging with source code breakpoints and variables window
• rust analyzer completion support of kernel code
• clippy linting
• rust optimized dark theme

Open kernel source

# Open kernel source code
$ code kernel/kernel.code-workspace

# Open bootloader source code
$ code crates/bootloader/bootloader.code-workspace

Keyboard shortcuts

• F4 builds and runs the kernel in qemu awaiting a debugger
• F5 attaches debugger to running kernel
• F6 builds and runs the kernel normally

View assembly with radare2

$ cd <project_root>/kernel
$ r2 target/x86_64-os/debug/isofiles/boot/kernel.elf # View bootloader asm

$ cd <project_root>/kernel
$ r2 target/x86_64-os/debug/perf_kernel # View kernel asm

Look into crates/bootloader/linker.ld to find the offset where the kernel gets mapped to.

Debug with gdb

To run the kernel in debugger await mode execute:

$ cargo run -- -d

Debugging the bootloader with gdb

$ cd <project_root>/perf_kernel
$ gdb -ex "target remote: 1234" -ex "symbol-file target/x86_64-os/debug/isofiles/boot/kernel.elf"

Debugging the kernel with gdb

$ cd <project_root>/perf_kernel
$ gdb -ex "target remote: 1234" -ex "symbol-file target/x86_64-os/debug/perf_kernel"

Debugging a different cpu core

In gdb cpu cores get handled like threads. So to display all cpu cores execute: info threads To set a breakpoint on a different core execute: hb <address> thread <cpu_num>

Important

If you use qemu with kvm you have to use hardware breakpoints. Those are set with hb <address>

In qemu emulation mode just use the normal breakpoints set with b <address>

Debug with qemu monitor

Connect to qemu monitor with

$ nc 127.0.0.1 8124
(qemu) help

To switch to a different cpu core, execute:

(qemu) cpu <core_num>-1

Linker map

The linker generates a linker map where all ELF objects are listed with their respective addresses. You can find the file under <project_root>/perf_kernel/external/bootloader/target/linker.map.

Debugging MMU with vmsh

vmsh is a tool that spawns a thread in a qemu process to extract the kvm filedescriptor. This enables us to read VM guest memory from the host. The restart.sh does all of this automatically and then writes the MMU state as text into target/dump.analysis

Excerpt:

Virt Addr         Phys Addr       Size Perms Cache  NX
0x0            -> UNMAPPED        4Kb 
0x1000         -> 0x1000          4Kb  R       PCD NX 
...
0x3000         -> 0x3000          4Kb  R       PCD NX 
0x4000         -> 0x4000          4Kb  W              
0x5000         -> 0x5000          4Kb  R       PCD NX 
...
0xb7000        -> 0xb7000         4Kb  R       PCD NX 
0xb8000        -> 0xb8000         4Kb  W       PCD NX 
0xb9000        -> 0xb9000         4Kb  R       PCD NX 
...
0xff000        -> 0xff000         4Kb  R       PCD NX 
0x100000       -> 0x100000        4Kb  R              
...

ISO file

To create a new ISO file, cargo run needs to be executed. cargo build does not suffice, it only generates a new kernel executable file but not a new ISO file. The path to the ISO is target/x86_64-os/debug/bootimage-perf_kernel.iso or if build in release mode target/x86_64-os/release/bootimage-perf_kernel.iso. To create a bootable USB stick just flash the image onto the USB device with:

$ dd bs=5M if=target/x86_64-os/release/bootimage-perf_kernel.iso of=/dev/<YourUSB> status=progress

PXE boot

Previously I tried to use pixiecore to setup PXE however there are a couple of incompatibilities because it always uses it's own IPXE build integrated into the tool. But IPXE does not currently support Multibootv2 booting, that's why shell.nix builds a custom version of IPXE that can be found under $IPXE/undionly.kpxe

LLVM assembly

If you are interested in the LLVM assembly of your kernel then execute cargo asm this generates the LLVM asm in release mode under: target/x86_64-os/release/deps/perf_kernel-*.s

Build system

The build system is highly custom but well integrated into cargo. The glue_gun tool goes into more detail.

Important configuration files for the build system are:

• .cargo/config.toml
• Cargo.toml
• x86_64-os.json
• i686-uknown-linux-gnu.json
• linker.ld
• build.rs
• rust-toolchain

Run tests

To execute tests run:

$ cd <project_root>/perf_kernel
$ cargo test

Run specific test:

$ cargo test --test heap_allocator

Developed on a

• AMD Ryzen 5 3500U
• EPYC-v1
• AMD Family 17h Model 18h

Resources

• https://os.phil-opp.com/
• https://www.amd.com/system/files/TechDocs/24593.pdf
• https://github.com/gamozolabs/chocolate_milk/
• https://uefi.org/sites/default/files/resources/ACPI_6_3_final_Jan30.pdf
• Use 1Gib pages sparringly
• Don't touch MTRRs
• https://virtio-fs.gitlab.io/index.html#overview
• https://gitlab.redox-os.org/redox-os/tfs
• http://9p.cat-v.org/
• https://www.linux-kvm.org/page/Tuning_Kernel