OSv on new KVM-based AWS virtual machines

[nyh]

Amazon recently switched their new instances to using KVM instead of Xen - see for example
https://www.theregister.co.uk/2017/11/07/aws_writes_new_kvm_based_hypervisor_to_make_its_cloud_go_faster/

We want OSv to be able to run on these new instances. @avikivity says that these instances will not support virtio-net or virtio-blk, and OSv will need NVMe and ENA drivers to support the disk and network, respectively, on these VMs :-(

[rodlogic]

What would it take to get these drivers in place?

[gburd]

FreeBSD has been ported, here are their notes. http://www.daemonology.net/blog/2017-11-17-FreeBSD-EC2-C5-instances.html

…

-greg

On Fri, Dec 1, 2017, 3:28 PM rodlogic ***@***.***> wrote: What would it take to get these drivers in place? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#924 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAbENAqjy59jikWyHE5n3LsUxuSkbmpks5s8GFqgaJpZM4QgKwo> .

[avikivity]

1. Write an NVMe driver for OSv; this should be easy, since the protocol is simple and well documented, and can be tested on QEMU's NVMe implementation
2. Write (or port) the ena driver. This is harder since you have to test it on AWS.

[nyh]

If someone is curious what Amazon did in these new instance types, and why, Anthony Liguori has a very good explanation (38-minute video) here:

https://www.youtube.com/watch?time_continue=2&v=LabltEXk0VQ

He explains why they have thesse NVMe and ENA devices with a hardware backend (created by a startup they bought, Annapurna Labs) instead of software in Xen, and that they already have done this incrementally for several years as an additional option, but now they took one final step - dropping the old Xen device support (and Xen itself). They also replaced Xen with KVM, but did not use QEMU, and thus none of QEMU's virtio code is available. By not supporting the older Xen paravirtual protocols and using hardware accelerators, more CPU cores (and more CPU time per core) are available for the users. There is no real reason why they cannot provide slower virtio emulation, but also no real reason for them to do it...

[geraldo-netto]

Dear Friends,

I don't know if it's still relevant but maybe we could consider the NVMe/ENA driver from FreeBSD:
https://github.com/amzn/amzn-drivers/tree/master/kernel/fbsd/ena
https://github.com/freebsd/freebsd/tree/master/sys/dev/nvme

Kind Regards,
Geraldo Netto