Skip to content

Files

Latest commit

 

History

History
228 lines (163 loc) · 6.83 KB

rootfs-and-kernel-setup.md

File metadata and controls

228 lines (163 loc) · 6.83 KB

Creating Custom rootfs and kernel Images

Creating a Linux kernel Image

Manual compilation

Currently, Firecracker supports uncompressed ELF kernel images on x86_64 while on aarch64 it supports PE formatted images.

Here's a quick step-by-step guide to building your own kernel that Firecracker can boot:

  1. Get the Linux source code:

    git clone https://github.com/torvalds/linux.git linux.git
cd linux.git

  2. Check out the Linux version you want to build (e.g. we'll be using v4.20 here):

    git checkout v4.20

  3. You will need to configure your Linux build. You can start from our recommended guest kernel configurations by copying the relevant one to .config (under the Linux sources dir). You can make interactive config adjustments using:

    make menuconfig

    Note: there are many ways of building a kernel config file, other than menuconfig. You are free to use whichever one you choose.

  4. Build the kernel image:

    arch=$(uname -m)
if [ "$arch" = "x86_64" ]; then
     make vmlinux
elif [ "$arch" = "aarch64" ]; then
     make Image
fi

  5. Upon a successful build, you can find the kernel image under ./vmlinux (for x86) or ./arch/arm64/boot/Image (for aarch64).

For a list of currently supported kernel versions, check out the kernel support policy.

Use the provided recipe

The kernel images used in our CI to test Firecracker's features are obtained by running the script resources/rebuild.sh.

Users can build those locally by running:

./tools/devtool build_ci_artifacts kernels

This will build all versions that we currently use in our CI. kernels subcommand allows passing a specific kernel version to build. For example:

./tools/devtool build_ci_artifacts kernels 6.1

will build only the 6.1 kernel.

Currently supported kernel versions are: 5.10, 5.10-no-acpi (same as 5.10 but without ACPI support) and 6.1.

After the command finishes, the kernels along with the corresponding KConfig used will be stored under resources/$(uname -m).

Creating a Linux rootfs Image

A rootfs image is just a file system image, that hosts at least an init system. For instance, our getting started guide uses an ext4 filesystem image. Note that, whichever file system you choose to use, support for it will have to be compiled into the kernel, so it can be mounted at boot time.

In order to obtain an ext4 image that you can use with Firecracker, you have the following options:

Manual build

  1. Prepare a properly-sized file. We'll use 50MiB here, but this depends on how much data you'll want to fit inside:

    dd if=/dev/zero of=rootfs.ext4 bs=1M count=50

  2. Create an empty file system on the file you created:

    mkfs.ext4 rootfs.ext4

You now have an empty EXT4 image in rootfs.ext4, so let's prepare to populate it. First, you'll need to mount this new file system, so you can easily access its contents:

mkdir /tmp/my-rootfs
sudo mount rootfs.ext4 /tmp/my-rootfs

The minimal init system would be just an ELF binary, placed at /sbin/init. The final step in the Linux boot process executes /sbin/init and expects it to never exit. More complex init systems build on top of this, providing service configuration files, startup / shutdown scripts for various services, and many other features.

For the sake of simplicity, let's set up an Alpine-based rootfs, with OpenRC as an init system. To that end, we'll use the official Docker image for Alpine Linux:

  1. First, let's start the Alpine container, bind-mounting the EXT4 image created earlier, to /my-rootfs:

    docker run -it --rm -v /tmp/my-rootfs:/my-rootfs alpine

  2. Then, inside the container, install the OpenRC init system, and some basic tools:

    apk add openrc
apk add util-linux

  3. And set up userspace init (still inside the container shell):

    # Set up a login terminal on the serial console (ttyS0):
ln -s agetty /etc/init.d/agetty.ttyS0
echo ttyS0 > /etc/securetty
rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot:
rc-update add devfs boot
rc-update add procfs boot
rc-update add sysfs boot

# Then, copy the newly configured system to the rootfs image:
for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /my-rootfs; done

# The above command may trigger the following message:
# tar: Removing leading "/" from member names
# However, this is just a warning, so you should be able to
# proceed with the setup process.

for dir in dev proc run sys var; do mkdir /my-rootfs/${dir}; done

# All done, exit docker shell.
exit

  4. Finally, unmount your rootfs image:

    sudo umount /tmp/my-rootfs

Use the provided recipe

The disk images used in our CI to test Firecracker's features are obtained by using the recipe (in a Ubuntu 22.04 host):

./tools/devtool build_ci_artifacts rootfs

The images resulting using this method are minimized Ubuntu 22.04. Feel free to adjust the script(s) to suit your use case.

You should now have a rootfs image (ubuntu-22.04.ext4), that you can boot with Firecracker.

Creating FreeBSD rootfs and kernel Images

Here's a quick step-by-step guide to building a FreeBSD rootfs and kernel that Firecracker can boot:

  1. Boot a FreeBSD system. In EC2, the FreeBSD 13 Marketplace image is a good option; you can also use weekly snapshot AMIs published by the FreeBSD project. (Firecracker support is in FreeBSD 14 and later, so you'll need FreeBSD 13 or later to build it.)

    The build will require about 50 GB of disk space, so size the disk appropriately.

  2. Log in to the FreeBSD system and become root. If using EC2, you'll want to ssh in as ec2-user with your chosen SSH key and then su to become root.

  3. Install git and check out the FreeBSD src tree:

    pkg install -y git
git clone https://git.freebsd.org/src.git /usr/src

    Firecracker support is available since FreeBSD 14.0 (released November 2023).

  4. Build FreeBSD:

    make -C /usr/src buildworld buildkernel KERNCONF=FIRECRACKER
make -C /usr/src/release firecracker DESTDIR=`pwd`

You should now have a rootfs freebsd-rootfs.bin and a kernel freebsd-kern.bin in the current directory (or elsewhere if you change the DESTDIR value) that you can boot with Firecracker. Note that the FreeBSD rootfs generated in this manner is somewhat minimized compared to "stock" FreeBSD; it omits utilities which are only relevant on physical systems (e.g., utilities related to floppy disks, USB devices, and some network interfaces) and also debug files and the system compiler.