Slower networking of OSv on firecracker vs QEMU/KVM #1034
Before I dive into details of my performance test results I would to take this occasion to announce on this forum that firecracker is officially and fully supported by OSv unikernel as of latest 0.53.0 release (nickname "Firebird", for details please read here). It can boot in as low as 5ms per bootchart and 10ms per firecracker guest boot time measurement. Maybe it is worth mentioning on https://firecracker-microvm.github.io/ (section "What operating systems are supported by Firecracker?") that besides Linux, OSv can boot on firecracker as well ;-) ? From what I am aware of, OSv is the only unikernel and possibly the only other OS besides Linux that can claim this as of this point in time.
As far as performance comparison between OSv running on firecracker vs QEMU/KVM goes, first I must say that at least in one aspect firecracker beats QEMU - file I/O. I have not done any other elaborate file I/IO tests but for example mounting ZFS filesystem is at least 5 times faster on firecracker - on average 60ms on firecracker vs 260ms on QEMU.
My tests were focused of number of REST API requests handled per seconds by a typical microservice app implemented in Rust, using hyper, Golang and Java using vertx.io. Each app in essence implements simple todo REST api returning a json payload of 100-200 characters long.
The test setup looked like this:
The host and client machine were connected directly to 1 GBit ethernet switch and host exposed guest IP using a bridged TAP nic.
Here is a list of pure req/sec results:
For more detailed results please see the files where I captured full output from wrk - https://github.com/wkozaczuk/unikernels-v-containers/tree/master/test_results/remote/OSv_firecracker and https://github.com/wkozaczuk/unikernels-v-containers/tree/master/test_results/remote/OSv_qemu.
Would you have any insight of what might be the reason of relatively slower performance of firecracker? I think I have disabled the rate limiting which is what this script does - https://github.com/cloudius-systems/osv/blob/master/scripts/firecracker.py#L23-L97. It could be also that virtio-mmio implementation on OSv side is not very well optimized - with QEMU OSv uses virtio-pci.
Any help will be greatly appreciated.
The text was updated successfully, but these errors were encountered:
If you do not want to have a rate limiter, you can just write:
because the rate_limiter is an optional field. I am not sure what are the effects of setting every field of the rate_limiter to 0.
We will get back to you after we get a chance to investigate this.
@wkozaczuk , first of all I'll say that the folks in the Firecracker maintainer team have seen (and several of us are really excited by) OSv running with Firecracker. Your start-up times are awesome! Frankly, I think we're a bit behind on recognizing Firecracker integrations with other projects, and we will be working on making this better.
We also appreciate the in-depth issue descriptions that have helped us make Firecracker better.
The website (and maybe a docs page in the repo) is one place to showcase this, but I'd also like to write about out our current integrations in something like a blog post. If that's all right with you, we'll get in touch once we have a more clear idea, probably in a couple of weeks.
Regarding IO, I'm not surprised by the results. Rate limiting aside, we simply didn't spend much time on IO optimization (especially disk), since it wasn't a priority for our current users/customers. While IO is definitely something we want to improve on, prioritizing it will depend on user/customer demand (unless someone contributes it
@raduweiss I am very much open in collaborating on blog post.
I do not think that we have any specific use case in mind. I was myself curious to compare how OSv fares on firecracker vs QEMU. I wonder if slowness on networking side is cause by more frequent exists to the host comparing with QEMU. I would be nice to do similar comparison with Linux.