BMT - Bhyve Management Tool
This is a super lightweight yet very functional tool to manage Bhyve VMs on FreeBSD. It needs:
- A ZFS volume to place VMs into (UFS is not supported)
- GNU screen (
pkg install screen)
- Grub2 Bhyve loader (
pkg install grub2-bhyve), if you'll be running Linux VMs
- BHyve UEFI Firmware, if you'll be running Windows VMs (
pkg install -y bhyve-firmware)
It supports most UNIX OSes and Windows, handles auto-booting VMs at system start and shutting them down at system shutdown/reboot.
By default BMT will create a ZFS volume
zroot/vms to house the VM configs and virtual disks, and mount the base in
If you wish to have this somewhere else (ie; if you wish to place it on a different zroot) create a file called
# Base ZFS Root BASE_ZPATH="zssd/vms"
zssd is your preferred ZPOOL, and
vms is a standard dataset.
The first time setting up BMT on a system (assumes bmt is installed into
ln -s /usr/local/bmt/bmt.rc.sh /usr/local/etc/rc.d/bmt && sysrc bmt_enable="YES" && bmt rcstart && bmt setup
It is best to reboot after this just to be sure everything is applied.
You can run just
bmt to get a list of options. This section needs to be expanded more but the following should get you started:
List All VMs
bmt list will show all VMs, their states, used memory, CPU and if they're set to come up on system boot.
bmt netmap command will attempt to map out VM's TAP devices an label them in an easy to understand way against each bridge.
You can specify user-friendly labels for the bridges by adding the following format lines to
BRIDGE_bridge0_NAME="Private LAN" BRIDGE_bridge1_NAME="Public WAN"
To create a new VM with a 16 GiB virtual disk:
bmt create newvmname -V 16G
bmt edit <vname> launches opens the appropriate vm.conf file in your preferred editor.
To stop a VM:
bmt stop vmname
To start a VM:
bmt start vmname
(Where "vmname" is the name you gave it).
Attach To Console
To attach to the text console:
bmt attach vmname
Execute a console command
bmt exec <vmname> '<command>'
bmt exec freebsd 'root'
bmt exec freebsd 'pass'
bmt exec freebsd 'uname'
A copy creates a new free standing "thick" VM with no links to the source's volumes.
To copy a vm:
bmt copy vmname new-vmname
-d to the end to destroy snapshots on the new VM to have an entirely clean copy with no snapshots. Otherwise any snapshots on the source VM will be copied (but not linked in any way to the source VM).
A clone uses ZFS snapshots to make a new "thin" VM linked to the source.
To clone a vm:
bmt clone vmname new-vmname
To destroy a vm, it must be off:
bmt destroy vmname
To see the status of a vm:
bmt status vmname
Block until vm stops
This command will block until a vm powers off:
bmt wait_for_poweroff vmname
The vm send/receive functionality works very similar to zfs.
This example sends the vm to a compressed file:
bmt send vmname | xz > vmname.bmt.xz
This example receives a vm from a compressed file:
xzcat vmname.bmt.xz | bmt receive vmname
This example sends a vm between hosts:
bmt send vmname | ssh user@host bmt receive vmname
You can get and set parts of the vm configuration with these commands:
bmt get vmname VM_CPUS
bmt set vmname VM_CPUS=2
By default with
AUTO_NETWORKING="YES" set for a VM, a TAP device will automatically be created and assigned to bridge0 (for the example below).
NOTE: It will not automatically create bridge0, you still need to set that up in
This auto-provisioning can be overridden via these config blocks:
# -- Networking # Up to 12 nics are possible following the same naming convention ) # AUTO_NETWORKING="NO" VM_N1_BRIDGE_NUM="0" VM_N1_TAP_NUM="21" VM_N2_BRIDGE_NUM="" VM_N2_TAP_NUM=""
NOTE: If you have PF enabled make sure to 'skip' any VM taps and bridges
set skip on bridge0 set skip on tap21
Otherwise nothing will work. Obviously you can apply more granular control over this but this is a common issue when VM networking doesn't work.
If you need to, you can use the new E1000 network driver instead of the virtio-net driver by setting:
This has proven necessary under Debian 8.10 as there is a bug somewhere that causes packets larger than 230 bytes to be truncated using virtio-net.
PCI passthru requires the variable
VM_PCI_PT to be set to the PCI address of the PCI device that you would like to passthru to the VM. For example:
root@donchor--> pciconf -l -v re1@pci0:5:0:0: class=0x020000 card=0x80011297 chip=0x816810ec rev=0x06 hdr=0x00 vendor = 'Realtek Semiconductor Co., Ltd.' device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller' class = network subclass = ethernet
If I wanted to pass this network card through to a VM, I would use the PCI address 5/0/0 (derived from the string
re1@pci0:5:0:0 [last three digits, replace colons with slashes]).
Now that we have the PCI address, things get easier. On the VM in question's configuration, add a line like this:
Finally, you need to keep an aggregate list of devices that bhyve is using in
/boot/loader.conf. In our example above, we use
5/0/0, and another VM is using
6/0/0. My aggregate entry in
/boot/loader.conf would look like this:
At boot time, bhyve will attach to these devices early before the kernel has a chance to decide on a driver for them. In the
pciconf example above the string
re1@pci0:5:0:0 tells us that the kernel has already attached a driver (
re) for the pci device. A reboot will be necessary so that bhvye can get between the kernel and driver, and shunt it into the VM once it starts.
Thats it. After a reboot, your VM should have access to the PCI device!