loader: add support for initrd

[marcov]

Reason for This PR

The 2020 Roadmap welcomes contributions that adds initrd support. A previous PR was opened to add it (#670), but cannot be reused as is, because it also added support for loading a compressed kernel.

Hence #670 was reworked and rebased to only include initrd support.
Fixes #208 .

Description of Changes

• Add initrd_path property to the boot-source API.
• Pass ramdisk_image (memory offset) and ramdisk_size to the guest bootparams table.
• Implement the load_initrd loader function, to load the initrd file content into the guest memory at a (for the moment) fixed address at then end of the available low memory (for x86_64).

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license.

PR Checklist

[Author TODO: Meet these criteria. Where there are two options, keep one.]
[Reviewer TODO: Verify that these criteria are met. Request changes if not]

• All commits in this PR are signed (git commit -s).
• Either this PR is linked to an issue, or, the reason for this PR is
  clearly provided.
• The description of changes is clear and encompassing.
• Either no docs need to be updated as part of this PR, or, the required
  doc changes are included in this PR. Docs in scope are all *.md files
  located either in the repository root, or in the docs/ directory.
• Either no code has been touched, or, code-level documentation for touched
  code is included in this PR.
• Either no API changes are included in this PR, or, the API changes are
  reflected in firecracker/swagger.yaml.
• Either the changes in this PR have no user impact, or, the changes in
  this PR have user impact and have been added to the CHANGELOG.md file.

[marcov]

Disclaimer: I am complete rust n00b!

Missing things:

• aarch64 support
• tests (bzImage / initrd support (v2) #670 had dedicated commits for tests that can be ported too).

[marcov]

For x86_64 I am now loading the initrd at the end of the available low memory, instead of using a fixed address.

[marcov]

Hi, x86_64 is now ready for review. I just need a new fixture added to S3 for initrd testing.

Do you expect aarch64 be supported too, or can it be postponed in a later PR?

[rgooch]

I would suggest merging and postponing aarch64, so that x86_64 people can test sooner (more easily).

[marcov]

Spent some more hours and added aarch64 support. But I don't have any aarch64 hardware with KVM hypervisor mode to be able to test it.

[marcov]

Update: aarch64 is functional.
Also, almost all tests are passing, just waiting for a initrd update on S3 to align the ramdisk images with the tests.

[andreeaflorescu]

@marcov do you need help in addressing the outstanding review comments?

[marcov]

Thank you for the ping @andreeaflorescu, pushed some updates now.

[andreeaflorescu]

@marcov I created a prototype on my branch on top of your commits: andreeaflorescu@994b23d.

The initrd_load_addr function doesn't need to live in arch. It is more of a utility function for loading the initrd image into guest memory. I moved it to the kernel crate along with load_initrd function. To break the dependency between the arch crate and the kernel crate, we can send the page size as a parameter to load_initrd. This is still not an ideal state because now arch depends on kernel which is not something that we want. This dependency can further be broke by refactoring the fdt implementation in arch, but this is out of scope of this PR.

To give you more context, it is important to have the loading of initrd in the kernel crate because we are working towards consuming the linux-loader crate from rust-vmm. Once we make sure that loading the initrd works as expected, we should also send a PR with the code here: https://github.com/rust-vmm/linux-loader. If you don't want to also send a PR in rust-vmm, one of the Firecracker maintainers can take care of that. Loading an initrd image is quite a common feature that we would like to have in linux-loader as well. Having that functionality in the vmm crate makes it impossible to share the common code with the other VMMs written in Rust (for example with Cloud Hypervisor). Besides this, there is an ongoing effort to minimize the code in vmm.

[dianpopa]

@marcov I created a prototype on my branch on top of your commits: andreeaflorescu@994b23d.

The initrd_load_addr function doesn't need to live in arch. It is more of a utility function for loading the initrd image into guest memory. I moved it to the kernel crate along with load_initrd function. To break the dependency between the arch crate and the kernel crate, we can send the page size as a parameter to load_initrd. This is still not an ideal state because now arch depends on kernel which is not something that we want. This dependency can further be broke by refactoring the fdt implementation in arch, but this is out of scope of this PR.

To give you more context, it is important to have the loading of initrd in the kernel crate because we are working towards consuming the linux-loader crate from rust-vmm. Once we make sure that loading the initrd works as expected, we should also send a PR with the code here: https://github.com/rust-vmm/linux-loader. If you don't want to also send a PR in rust-vmm, one of the Firecracker maintainers can take care of that. Loading an initrd image is quite a common feature that we would like to have in linux-loader as well. Having that functionality in the vmm crate makes it impossible to share the common code with the other VMMs written in Rust (for example with Cloud Hypervisor). Besides this, there is an ongoing effort to minimize the code in vmm.

The initrd_load_addr does need to live in arch -> or at least this is how we achieve consistency with the way the kernel deals with initrd. Please take a look at how things work for x86_64 (starting here https://elixir.free-electrons.com/linux/v4.14.158/source/arch/x86/kernel/setup.c#L304) -> the first interaction of the kernel with the initrd config data is in the setup_arch function which contains the architecture-specific boot-time initializations which is kind of the equivalent of our configure_system function. You can see there that the two essential things that the kernel needs are the address where the initrd is loaded and the size of it.

• For the first one (i.e the address) you need to be aware of the memory layout for a specific platform (which we are defining in arch). The one function that does the job for that is initrd_load_address from arch. If we were to move this in the kernel crate (as it is the case with your model):

    let lowmem_size: usize = guest_mem.region_size(0).map_err(|_| Error::InitrdAddress)?;

we would make the kernel crate dependent on memory layout representation for a specific platform. Does not that break the separation of concerns we strive for?

• As far as the size, a fseek is sufficient. This functionality is achieved in the current implementation by the load_initrd function from the vmm.
  After address and size is obtained, the image is written to memory. If we want to avoid passing around the guest memory (which is what we should be striving for in our decoupling goals), this should also stay in vmm.

[marcov]

Taking load_kernel as example, I can keep part of load_initrd in vmm to retrieve the initrd address and size, and have a load_initrd in kernel to write data into the guest memory. The kernel function would only be a one liner though.

[andreeaflorescu]

@marcov that was my initial thought as well and noticed the same problem as you. Let's keep it as is for now and we will find a better place for it in a future iteration.

I am still a bit concerned regarding the rather large boot time on aarch64. I was thinking about doing some comparison with Qemu boot times (if they have something similar we can compare against). I can check that tomorrow.

[marcov]

@andreeaflorescu using a non-gzipped initrd will shave off around 100ms, but even doing that we are still far aways from the max time of 160ms.
Checking the boot log timestamp, most of the time is spent by the kernel unpacking the initramfs, and that time depends on the size of the initrd file. This is with a 25MB initrd:

[    0.110152] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.112462] NET: Registered protocol family 1
[    0.114731] Unpacking initramfs...
[    0.353062] Freeing initrd memory: 24712K

[dianpopa]

@andreeaflorescu using a non-gzipped initrd will shave off around 100ms, but even doing that we are still far aways from the max time of 160ms.
Checking the boot log timestamp, most of the time is spent by the kernel unpacking the initramfs, and that time depends on the size of the initrd file. This is with a 25MB initrd:

[    0.110152] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    0.112462] NET: Registered protocol family 1
[    0.114731] Unpacking initramfs...
[    0.353062] Freeing initrd memory: 24712K

On the a1.metal aarch64 machine (9 MB initrd), things look like this:

[    0.114070] Unpacking initramfs...
[    0.468239] Freeing initrd memory: 9420K
[    0.470117] audit: initializing netlink subsys (disabled)
[    0.472665] Initialise system trusted keyrings

[marcov]

@dianpopa try to gzip -d the initrd.img, you should gain around 100ms.

[andreeaflorescu]

If it is expected for the boot time to be so large with the provided initrd image, we can merge the PR as is. I didn't really have time to investigate this, and I don't want to block the PR. Feel free to update the time for aarch64 to a value that you're confident with.

[dianpopa]

@dianpopa try to gzip -d the initrd.img, you should gain around 100ms.

I tried with the initrd.img that the CI leaves behind. @marcov, are you implying that it would be a good idea to reupload in S3 the decompressed version?

[marcov]

@dianpopa: right! An ASCII cpio archive should be faster.

I'm going to do a final squash and set a reasonable boot time for aarch64, checking for platform.machine()

[marcov]

Happy to see this merged!
Thanks for the help @dianpopa @andreeaflorescu , and happy holidays :-)

[dianpopa]

Thanks for the effort and responsiveness @marcov ! Happy holidays!