Home
-
Ongoing projects in the FreeBSD-UPB organization
- bhyve on ARM
- Save/restore for multiple same-type devices for bhyve
- JSON File format for the save/restore functionality in bhyve
- CAPSICUM integration for the save/restore functionality in bhyve
- Checkpoint functionality for bhyve using ZFS snapshots
- Checkpoint functionality for bhyve using libvdsk
- Warm migration for bhyve
- Live migration for bhyve
- AMD support for the save/restore functionality
- USB-passthrough for bhyve
- FreeBSD as a compute node in OpenStack
- Snapshot support for NVMe devices in bhyve
- Interesting links
This project implements the entire bhyve hypervisor for devices with ARMv8 processors.
It is able to start FreeBSD and Linux guests (follow the Run bhyve on EspressoBin[3] wiki page and
the review's description[4]). Some of the key features are VirtIO (block, device, network and console),
bootrom (start a guest using a custom bootloader, instead of bhyveload) and the UART serial between
the host and the guest.
As a side note, this implementation works for the ARM devices with the third version of the Generic Interrupt Controller (GICv3).
The utils repository[1] is a collection of files and scripts, used for compiling the host and the guest. At [3] you can find a demo on how the scripts are used.
Some useful links:
The feature provides a way to snapshot a guest that uses multiples devices of the same type(e.g. two or more disks, two network interfaces, etc.). Initially only the first device was considered during the suspend and restore processes. The solution also proposes a generic way to store devices of different types.
Links regarding the project:
The project implements a way to save and restore a bhyve guest using JSON file format in order to provide a better way to present state information. In order to save buffer in the file while keeping it in text format, base64 encoding is used. While the current solution uses 3 files in order save and restore the state of the guest, the proposed solution uses only 2, one for the RAM memory and one for the state of each device.
A portion from a generated file after suspend can be seen below:
"device_params": [
{
"vcpus": [
{
"vcpus@0": [
{
"x2apic_state$uint32": 0,
"exitintinfo$uint64": "0",
"exc_vector$int32": 0,
"exc_errcode_valid$int32": 0,
"exc_errcode$uint32": 0,
"guest_xcr0$uint64": "7",
"exitinfo$b64": "FQAAAAAAAACWXQiB/////wCwHDsBAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPYSsBAAAAAAAAAAAAwAAAAMAAAAAAAAAiYGAAwAAMcBdw2YuDx+EDwZIAAgBAAQAAAAAAAIAAAAtAAAALQAAAIADAAAAAAAAAAAAAAAAAAABAIkBAAAAAA==",
"nextrip$uint64": "ffffffff81085d96",
"tsc_offset$uint64": "15063e03a9f0"
}
],
Links regarding the project:
The project applies the necessary changes in order to use bhyve save and restore processes while capsicum is enabled. The provided changes are mainly focused on using libcasper in order to use sysctl after capsicum is enabled; another change is targeting the usage of openat system call instead of open system call. In order to use openat an initial file descriptor that contains all the state files needs to be provided so the project implements a way to use the path of the first disk as the containing directory for all the state files.
Links regarding the project:
In order to implement the checkpoint mechanism that bhyve is missing at the moment, the project provides a way to do it by using functionalities from libzfs.
The project provides a way to checkpoint a bhyve guest by using zfs checkpoint and zfs clones; all the snapshots are organized in a tree internally such that both directions of restore are possible(the guest can be restored to a checkpoint that was created before or after the one that is currently in use).
Links regarding the project:
- git repository
- step by step tutorial on how to install and use the solution
- demo video on how to use the solution
The project proposes a checkpoint mechanism for bhyve using the qcow2 disk format provided the by livdsk library.
The implementation provides two types of checkpoint:
The internal checkpoint uses the qcow2 disk format capability to store the snapshot information using new clusters inside the disk. On restore, in order to avoid copying all the clusters only the L1 table is copied.
In order to create an internal checkpoint the following command needs to be used:
bhyvectl --checkpoint <checkpoint_file_name> --vm=<guest_name>
The above command works only if the guest virtual machine is "alive" (powered on) because the checkpoint command is sent using an UNIX socket.
In order to restore an internal checkpoint the following command can be used:
bhyve -c 1 -m 512M -H -A -P -s 0:0,hostbridge -s 1:0,lpc -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 2:0,virtio-net,tap0 -s 3:0,ahci-hd,<qcow2_disk_name> -l com1,stdio -r <checkpoint_file_name> <guest_name> # (*)
Some of the parameters that can be seen above can be changed because they represent the guest configuration.
The external checkpoint use the qcow2 base file and overlay file concepts. After a checkpoint creation finished successfully a new qcow2 file will be created based on the provided checkpoint file name and it will have as base the disk before the checkpoint creation was requested.
In order to create an external checkpoint the following command can be used:
bhyvectl --ext-checkpoint <checkpoint_file_name> --vm=<guest_name>
The guest must be powered on for the command to work.
If you want to restore a guest just used the resulted file in order to boot the virtual machine or you can use a command similar with (*) while specifying the resulted file if you want to have the state of the guest as well.
Ahci disks should be used for starting the guest because the solution was tested using these type of disks.
Useful links:
- libvdsk qcow2 internal/external checkpoint implementation branch When using this branch for build use the following command inside the libvdsk directory:
make clean && make -DWITH_BHYVE_SNAPSHOT && make install
Library for working with various virtual disk formats. To be used by bhyve to support other popular formats such as QCOW2, VMDK or VDI instead of only raw files.
<TODO - add description and links> Virtual machine migration represents the process of moving a virtual machine from one host to another. There are three types of migration:
- cold migration - shutdown the virtual machine on one host, start the virtual machine on the other host.
- all the services will be stopped.
- warm migration - suspend the virtual machine on one host, transfer state to the other host, resume its state there.
- the guest's state is frozen, when restoring, all the processes will start from the same point they were frozen.
- it is based on the snapshot save/restore functionality.
- live migration - move the virtual machine from one host to another while the virtual machine is still running.
- the guest is running while the virtual machine is moved.
- there is an usually smaller downtime (compared to warm migration) when the guest state is frozen.
<TODO - add description and links>
<TODO - add description and links>
The project aims to implement the USB passthrough functionality for bhyve hypervisor. This feature allows directly assignment of a USB device to a virtual machine for exclusive use and it represents the more practical alternative of the PCI passthrough mechanism. PCI passthrough gives the entire host controller to a virtual machine, making it unavailable for other guests or even for the host system.
At the moment, the USB device is visible in the virtual machine, but the communication with the guest is not yet possible.
Links regarding the project:
<TODO - add description and links>
The virtual machine snapshot feature for bhyve - the FreeBSD hypervisor - should enable the users to quickly create a quick copy of a VM. This project allows NVMe (non-volatile memory express) compatible devices to take part in the snapshot process. NVMe is connected to PCI bus and it provides an alternative to the SCSI and the ATA standard for connecting and transmitting data between a host system and a target storage device.
Quick links:
You may find interesting the following links:
- Save and Restore for a virtual machine in bhyve procedure
- Migrate a virtual machine using bhyve
- Running a FreeBSD guest under bhyve on ARM systems
- Using bhyve checkpoint based on libzfs
NOTE: The projects are under development and may not be completed.