KVM inside Firecracker VM

[kylecarbs]

Is there any documentation on forwarding /dev/kvm inside a Firecracker VM?

[iggy]

I am not a developer, but I think nested virtualization is outside the scope of what firecracker is trying to build. There are other projects with a broader scope that might be better to look at if you want nested virt. (i.e. cloud-hypervisor, qemu, etc)

[kylecarbs]

Got this working by enabling virtualization when compiling the kernel.

[raduweiss]

@kylecarbs hold on a second! :) While this may work, it's super-important to call out that we have done no testing on what kind of performance you would get with layer 2 hypervisor (probably very poor), and that any 2 VMs running inside of Firecracker in this way are not to be considered isolated from each-other.

In general this is really not something you should use in any production system.

Also, as @iggy pointed out, it's not something that matches our charter right now, so we're not very likely to invest in it (we want to stay focused on building the best possible serverless compute VMM).

[kylecarbs]

I appreciate the response @raduweiss. Will keep that in mind for sure. Mostly just messing around right now anyways :)

[jeromegn]

@kylecarbs I'm playing around with this and was wondering how you got it working? For testing / development purposes.

I added these to my VM kernel config:

CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m

and enabled nested KVM on my host's kvm_intel module.

Inside the guest, /dev/kvm is not created. My guest's /proc/cpuinfo does not have vmx. I'm unsure why.

[jeromegn]

I'd love to make this work somehow. It doesn't have to be perfect.

I assume this is mostly a question of enabling the right CPU features for the virtual CPU?

[kylecarbs]

@jeromegn sorry for the slow response! I'll check my repo.

[kylecarbs]

@jeromegn here is the kernel config we used.

[kylecarbs]

@jeromegn let me know if that works, I'm happy to hop on a call and chat too. I'm a fan of Fly and I know a bit too much about nested virtualization.

[jeromegn]

@kylecarbs thanks a lot 😄, I never tried it, but I do like what coder is doing!

I do have a quick question: which config do you think makes it work? I'm comparing what we got with yours, it's unclear. I'm going to try enabling as many _KVM_ configs I can!

[kylecarbs]

We have a repo where I made a CRI for Kubernetes with Cloud Hypervisor (that used Firecracker in the first iteration). If you'd like I can add you (and you can essentially just steal, I don't care haha). It never became OSS because we never needed to complete it, but it's probably helpful for the work y'all do!

DM me on Discord so we can chat a bit more in real-time: Kyle#6348

[alexellis]

@kylecarbs I couldn't find any claims that cloud-hypervisor supports and / or certifies isolation of nested VMs. Can you point us at what docs or issues you found stating this it's suitable and that VMs within VMs remain securely isolated?

[kylecarbs]

@alexellis AFAIK nested virtualization is a feature of KVM, not of any specific hypervisor. I'm not sure they have the ability to restrict it even if they wanted to. The security is also ensured by KVM (assuming cloud-hypervisor doesn't intentionally expose a loophole).

I haven't used Firecracker or Cloud-hypervisor for a while, so I'm not sure my recommendation would be sound anymore. At the time of posting in March of 2020, cloud-hypervisor had more virtio compatibility, which was helpful for passing through block devices at high-throughput speeds.

https://www.linux-kvm.org/page/Nested_Guests

[alexellis]

If I understand correctly, the issues are going to be the same as they are when @raduweiss weighed in earlier with:

While this may work, it's super-important to call out that we have done no testing on what kind of performance you would get with layer 2 hypervisor (probably very poor)

and that any 2 VMs running inside of Firecracker in this way are not to be considered isolated from each-other.

In general this is really not something you should use in any production system.

The later point would be the most concerning for running untrusted code.

[kylecarbs]

@alexellis I'm not sure what Firecracker could be doing that would impact layer 2 performance. AFAIK there are no oddities with the abstraction that would break expected Linux behavior.

So I understand, you're trying to run isolated VMs inside another VM? Because if you're just running on the host with VMs that enable nested virtualization, this would still be secure, but double nesting would indeed not be.

[alexellis]

Bare-metal Intel host -> Firecracker MicroVM (1) -> KVM VM with qemu/Nix/Firecracker (2)

[kylecarbs]

Ahh, gotcha. Why do multiple layer 2 VMs need to be isolated if they are wrapped in 1?

[klutchell]

@alexellis did you also discover that nested KVM doesn't appear to be supported for arm64 until at least kernel 6.3 if not later?

[alexellis]

Hi @klutchell yes, nested virtualisation isn't available on the current generation of Arm chips, I'm unaware of whether there needs to be additional support in the Kernel.

See point 3, here

Alex