QEMU MicroVM: suspiciously bad performance

[dimakuv]

Thanks for the interesting paper! I read it from here: https://arxiv.org/ftp/arxiv/papers/2110/2110.11462.pdf

• First, I couldn't find in the paper where you report the exact versions/commits you used in the evaluation? At least, for QEMU, I couldn't find the exact version used.

• Second, you have this finding: " Boot time depends heavily on the used machine model, as QEMU with the µVM machine model is (unexpectedly) the slowest out of all." (µVM = microvm machine model)

Looking at your script, I see this qemu-with-microvm command line:

container_benchmarks/startup/osv/run.sh

Line 63 in 683fbf6

measure-endtoend result-uvm "qemu-system-x86_64 -m 256M -smp 1 --nographic -kernel /path/to/kernel.elf -append --nopci -append --console=serial -M microvm,x-option-roms=off,pit=off,pic=off,rtc=off -nodefaults -no-user-config -no-reboot -global virtio-mmio.force-legacy=off -device virtio-blk-device,id=blk0,drive=hd0,scsi=off -drive file=/path/to/usr.img,if=none,id=hd0,cache=none,aio=native -netdev user,id=un0,net=192.168.122.0/24,host=192.168.122.1 -device virtio-net-device,netdev=un0 -enable-kvm -cpu host,+x2apic -serial stdio"

ACPI is enabled by default in microvm machine model, and when it's enabled QEMU uses SeaBIOS. When ACPI is disabled, QEMU uses qBoot (supposedly much faster than SeaBIOS). See e.g. https://www.kraxel.org/blog/2020/10/qemu-microvm-acpi/, section "ACPI support for microvm" for details on enable/disable ACPI.

qBoot can supposedly boot a VM in 40ms: https://lwn.net/Articles/645455/ ("Enter qboot, a minimal x86 firmware that runs on QEMU and, together with a slimmed-down QEMU configuration, boots a virtual machine in 40 milliseconds...")

Did you try the qemu -no-acpi experiment? It seems that you imply that QEMU always uses qBoot in your experiments, which doesn't seem to be true.

[dimakuv]

Just read through the master thesis, also couldn't find any mentions of the versions of software you tested.

Took another look through the repository, and found some installation scripts for CloudHV and Firecracker -- at least I can see the commits/versions you used in the experiments. But couldn't find nothing on the QEMU version still.

Am I missing something? Could you please add the exact QEMU version you used at least to the README in this repo?

[vrfn]

Hi @dimakuv,

Thank you for the feedback, I'm pretty happy to see someone taking the time to dig into and verify our work.

Regarding your two points:

• Good point, I will add the version/commit used to the README of this repository. Unfortunately I don't think I can update the thesis pdf.
• The boot times for the hypervisors (including QEMU) in Figure 14, that the finding you quoted refers to, were not obtained using the code that you found. The results in Figure 14 were obtained using the benchmarking setup by the NSDI '20 paper 'Firecracker: Lightweight Virtualization for Serverless Applications', which can be found here. The relevant µVM boot script can be found here, which indeed uses qBoot.

[dimakuv]

@vrfn Thanks for the reply.

I still have questions. The paper on Firecracker (https://www.usenix.org/system/files/nsdi20-paper-agache.pdf) doesn't mention QEMU's microVM and doesn't have any performance numbers on it. I observe that there is indeed a utility to test QEMU's microVM in the Firecracker-data GitHub repository, but it was not used in the paper, and I also see no discussions of this data anywhere in the repo.

I see this commit which adds some QEMU's microVM data: firecracker-microvm/nsdi2020-data@b35147f. I also see that the Firecracker authors updated the plot with boot time of all VMMs: https://github.com/firecracker-microvm/nsdi2020-data/blob/master/data/m5d.metal/boot-serial-all.pdf.

The plot shows that QEMU microVM is terribly, unexplainably bad. I guess the Firecracker authors didn't want to investigate this and just left it out of the final paper. But this very bad result makes very little sense to me -- I assume that Firecracker authors used a buggy QEMU version or there is something wrong in the arguments they used. Hard to say without more analysis.

Now, looking at your Figure 15 (which uses your own QEMU command line), QEMU's MicroVM actually looks reasonable -- it is slightly worse than Firecracker and significantly better than QEMU. These are the results of QEMU's MicroVM that I would expect. (I still have a suspicion you're using SeaBIOS in your QEMU invocation, and switching to qBoot may have improved the OSv boot time even more.)

Ok, I understand this setup a bit more. In Figure 14 (where QEMU MicroVM performance is unexplainably bad), you took the scripts from https://github.com/firecracker-microvm/nsdi2020-data. These scripts contain also QEMU MicroVM tests, which didn't make it into the final Firecracker paper (suspiciously). So indeed, you replicated the same unexplainably-bad results of QEMU MicroVM that were first achieved by Firecracker authors.

At this point I guess I need to ask Firecracker authors what happened :) Thanks for responding!