Skip to content

Run bhyve on EspressoBin

Jump to bottom
Andrei Martin edited this page Dec 29, 2020 · 8 revisions

Setup

The setup consists in two repositories: the FreeBSD source code and the arm64 utils, which contains the building scripts, config files and auxiliary files (e.g. Device Trees, kernel configs). The arm64 bhyve can be found on the projects/bhyvearm64 branch.

root@freebsd:~# mkdir demo
root@freebsd:~# cd demo
root@freebsd:~/demo# git clone https://github.com/FreeBSD-UPB/freebsd
root@freebsd:~/demo# cd freebsd/
root@freebsd:~/demo/freebsd# git checkout projects/bhyvearm64-v29Dec2020
root@freebsd:~/demo/freebsd# cd ..
root@freebsd:~/demo# git clone https://github.com/FreeBSD-UPB/bhyvearm64-utils
root@freebsd:~/demo# cd bhyvearm64-utils/
root@freebsd:~/demo/bhyvearm64-utils# git checkout projects/bhyvearm64-v29Dec2020

To build the host, the config/arm64/host.json has to be updated:

  • kernconf should be GENERIC
  • update the paths to root/demo/<...>
  • start building the world and the host kernel

To build the guest, the config/arm64/guest_kernel.json has to be updated:

  • kernconf should be ESPRESSOBIN_GENERIC
  • update the paths to root/demo/<...>
  • start building the guest kernel

Cross-compiling the FreeBSD guest and host kernels

Host

To cross-compile the FreeBSD world and host you have to load the filemon module and run the build.sh script with the host.json config file and buildworld/buildkernel build parameter.

root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/arm64/conf/GENERIC ../freebsd/sys/arm64/conf/
root@freebsd:~/demo/bhyvearm64-utils# kldload filemon
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build buildworld
<!--- TODO root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/arm64/arm64/busdma_bounce.c ~/demo/freebsd/sys/arm64/arm64/--->
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build buildkernel
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build installworld
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/host.json --build installkernel

Guest

In order to build the guest some files are needed, the kernelconfig and the device tree. The first step is to update the MFS_IMAGE variable in the ESPRESSOBIN_GUEST kernel config, then to copy ESPRESSOBIN_GUEST and armada-3720-espressobin-guest.dts in the FreeBSD repository.

MFS_IMAGE=/root/demo/bhyvearm64-utils/files_guest/ramdisk-guest.img
root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/arm64/conf/ESPRESSOBIN_GUEST ~/demo/freebsd/sys/arm64/conf/
root@freebsd:~/demo/bhyvearm64-utils# cp freebsd/sys/dts/arm64/armada-3720-espressobin-guest.dts ~/demo/freebsd/sys/dts/arm64
root@freebsd:~/demo/bhyvearm64-utils# ./build.py --config config/arm64/guest_kernel.json --build buildkernel

Note: The guest kernel will be build with small filesystem built directly into the binary. The generation of this filesystem is done by the build script.

U-Boot and storage

Firstly, we have to configure U-Boot on ESPRESSObin. Here are some tutorial on how to do that:

Secondly, we have to store the host's filesystem on a microSD card:

  1. Make two partitions: FAT32 and UFS
  2. Copy the built filesystem (arm64_rootfs) on both partitions
  3. Copy the virtio.img and virtio_run.sh files from the utils repository on the UFS partition
  4. Copy the guest kernel binary, in the same directory as previous files(3).

On U-Boot, set some variables with the paths to the device tree and the loader file and a command which boots the host.

setenv fdt_name 'boot/dtb/marvell/armada-3720-espressobin.dtb'
setenv image_name 'boot/loader.efi'
setenv bootcmd 'mmc dev 0; fatload mmc 0:1 $kernel_addr $image_name;fatload mmc 0:1 $fdt_addr $fdt_name; bootefi $kernel_addr $fdt_addr'
Marvell>> run bootcmd
OK set currdev=disk0p2
OK boot
mountroot> ufs:/dev/mmcsd0s2

The FreeBSD host started. You can run the virtio_run.sh script and start a guest.

Clone this wiki locally