Add GICv2 support to support Raspberry Pi 4

[luxas]

As per this the discussion in this thread #1186, it seems like RPi 3/4 has a GICv2, not v3/4 that Firecracker at the moment requires for arm64.

I would be very thrilled to have Raspberry Pi support for Firecracker, I believe it would open up the doors to a large Pi hacker community, and make Firecracker more accessible. My personal story TL;DR; is that my RPis was the only linux computer I had (when I was 15), and I wanted to use k8s. So I ported k8s over to ARM, and many nice things happened. Now k8s on ARM is a frequently-used way to learn/prototype/teach Kubernetes in various contexts.

This could be a major news item, maybe even for the RPi Foundation. Firecracker on RPi would most likely not be used in production, but could

a) boost Firecracker's reputation / usage
b) serve as the go-to way to teach / learn students how virtualization works

Bringing Firecracker to Raspberry Pi would be similar to when I made k8s accessible on the Pi -- it sparked a global k8s community learning how to use the thing "by doing" with the help of Pis. It even found its way into the University world (a master's thesis): http://bit.ly/kubecloud

I don't personally know how hard it would be to implement GICv2 support to Firecracker, but would be more than happy to try stuff out. I know @DieterReuter and @vielmetti would be, too.

Combined with ARM64 support in Ignite (weaveworks/ignite#172), running virtual machines on RPis would become as easy as doing ignite run centos:7 😄

What do you all think?
Thanks!

cc @andreeaflorescu @dianpopa @raduweiss @sudeep-holla

[vielmetti]

This looks like it's a useful discussion of GICv2 vs GICv3:

http://events17.linuxfoundation.org/sites/events/files/slides/ARM_Interrupt_Virtualization_Przywara.pdf

written by @apritzel

[apritzel]

Since the core of the GIC emulation is in the kernel, userland does not have to do much. It might help to look into this and this kvmtool patch (plus others near those), which enabled GICv3 in addition to GICv2 emulation. It's probably much easier to "retro-support" GICv2 when you use a vGICv3 already.
This commit then tries to auto detect the proper GIC type, so no user interaction needed.
Good luck!

[andreeaflorescu]

@luxas I am not 100% sure that the device you were using for running Firecracker had a GICv2 (or maybe the host kernel didn't have the GICv2 compiled in?).

I am saying this because as we discussed on Slack the Firecracker error comes from checking the IRQCHIP capability (KVM_CAP_IRQCHIP). This capability checks for any interrupt controller and not specifically GICv3. So from my understanding this check shouldn't fail if the device indeed has a GICv2.

If the Rasberry Pi does have a GICv2, as @apritzel pointed out it's relatively easy to add support for it in Firecracker. We can initialize a GICv3 by default and if that fails, we can fall back to a GICv2.

Just to be 100% sure, can you check that with a kernel build with GICv2 that Firecracker only crashes when you start to issue an InstanceStart command? I do not have access to an arm device that has a GICv2 :(, I would've checked it myself.

CC: @firecracker-microvm/compute-capsule

[luxas]

I am not 100% sure that the device you were using for running Firecracker had a GICv2 (or maybe the host kernel didn't have the GICv2 compiled in?).

Right, what was mentioned in the issue previously was RPi 3. That might or might not have an (at least somewhat) compatible GICv2. However, the RPi 4 has a GICv2, so we could check how far we can go with that at the moment, too.

We can initialize a GICv3 by default and if that fails, we can fall back to a GICv2.

👍

cc @DieterReuter for more details

[DieterReuter]

Just to clarify. My tests were on a Raspberry Pi 3B, and this device has no GIC hardware supoort at all. But the newer Pi 4B has GICv2 built-in. Currently there is no official 64bit kernel available from the Raspberry Pi Foundation, but there are some great tutorials how to build a 64bit kernel for the Pi 4B. There are still some limitations for this 64bit kernel, like it's limited to use max of 1 GByte of memory, but it should be good enough to build a 64bit kernel for the Pi 4B which also supports KVM and GICv2.

[dhrgit]

Right. Just to clarify, then: do we have an rpi4, running a 64-bit Linux, with kvm enabled, that could potentially be used to test Firecracker with vGICv2?

[bchalios]

Hi guys, we have been trying to run firecracker and ran into the same issues as well, on a rpi4 and a Hikey970 board.

I am looking into the GIC-specific code now, but if anybody wants, we 'd be more than happy to provide access to any of these two boards, for dev/testing.

[DieterReuter]

@bchalios can you probably share a 64bit kernel build? I could easily build a SD card image for the Pi 4B. Then more guys are able to work/test on it independently. I had no time to investigate to run that 64bit kernel build by myself, but will do as soon as I find time for, maybe next week.

[bchalios]

@DieterReuter We've put together instructions on how to build an ubuntu cloud-init server image here: https://blog.cloudkernels.net/posts/rpi4-64bit-image/

In there there's a link to an actual image we built as well, so you could use that.

(Instructions for building are mostly based on: https://andrei.gherzan.ro/linux/raspbian-rpi-64/ && https://andrei.gherzan.ro/linux/raspbian-rpi4-64/ )

[dhrgit]

I got an RPi4 and fiddled with it for a bit - got the 64-bit kernel and distro from @bchalios up and running. Unfortunately, as @andreeaflorescu mentioned, the error has more to do with KVM than Firecracker.

Apparently, KVM cannot create a vGIC (v2), and, as such, the KVM_CAP_IRQCHIP capability is missing. The CPUs are booted up in EL2 (hypervisor) mode, but /proc/interrupts shows no vGIC maintenance interrupt. Unfortunately, I'm not sufficiently familiar with the GIC-400 device, nor with its handling by KVM, in order to debug this within a reasonable time.

I also tried to compile a newer version of the RPi kernel, but that failed to boot - another issue that proved to be pretty difficult to debug on the RPi.

[serban300]

Implemented in #1235