FreeBSD Support (host)

[heywoodlh]

Description

FreeBSD lacks support for containerization generally but it does support QEMU. I was able to get Lima running on FreeBSD 13.0 but it doesn't work out of the box. I suspect that it wouldn't be difficult to fine-tune to get it up and running. And I feel that Lima could bring Docker support to a platform that currently lacks it which would be awesome.

I wrote-up a bit more detail on how I got Lima running on FreeBSD 13.0 here: https://the-empire.systems/freebsd-docker-2022

It's extremely slow but it works! I'm sure that I could have added something to QEMU's settings to make it perform better but I'm ignorant to the features that would help in this situation.

Installing Lima on FreeBSD 13.0:

1. Compile+install Lima from source:

pkg install gmake go git
git clone https://github.com/lima-vm/lima /opt/lima
cd /opt/lima

For some reason, compiling Lima fails with: pkg/sshutil/sshutil_others.go:14:19: undefined: err

Adding the following to the detectAESAcceleration() function in pkg/sshutil/sshutil_others.go fixes it:

var err error

Once err is defined, Lima compiles just fine with gmake:

gmake && gmake install

Not sure why that fails just on FreeBSD but nothing else.

2. Compile+install QEMU 7.0 from source:

pkg install ninja pkgconf glib pixman

git clone --depth=1 git://git.qemu-project.org/qemu.git /opt/qemu
mkdir -p /opt/qemu/build
cd /opt/qemu/build

../configure
gmake ## add -j to speed up the compilation
gmake install

Not sure how necessary this actually is but the version of QEMU packaged with the FreeBSD repos is 6.2.0 so upgrading to 7.0 seemed like a good idea.

3. Set the CPU to emulate:

FreeBSD doesn't have -cpu host (unlike Linux and MacOS) available so I had to manually specify the processor on my machine (I was using an Intel VPS with Vultr):

export QEMU_SYSTEM_X86_64="qemu-system-x86_64 -cpu Cascadelake-Server"

Problems needed to be solved before it's usable on FreeBSD:

1. Slowness (I am not to sure where to start there but I suspect that -cpu host would really help with this)
2. Minor: Fix that undefined: err in pkg/sshutil/sshutil_others.go for FreeBSD (could be as simple as defining that variable if the OS is FreeBSD)
3. Implement logic to not use -cpu host if OS is FreeBSD

It would be so nice to be able to interact with containers while on FreeBSD. Please let me know if you have any questions or if this is not something desired to support.

[AkihiroSuda]

Thanks, would it be possible to patch QEMU to use bhyve as the accelerator just like KVM/HVF/NVMM ?

[heywoodlh]

I'm not sure. But I did a bit more research and it looks like per this forum post you can build the QEMU port with the KQEMU kernel module to provide acceleration. So I'm gonna give that a try and see if that helps with the speed issue.

[AkihiroSuda]

KQEMU is dead

[heywoodlh]

Ah, I did not realize that. Looking through the documentation it doesn't look like there is any alternative accelerator to KQEMU specifically that would work for FreeBSD.

I see why you leapt to utilizing Bhyve for acceleration. Unfortunately, at the moment I don't know enough about QEMU to patch it to work with Bhyve and searching around online hasn't yielded any results for me on how to do that.

I'll try to do some more digging (perhaps reach out to the #qemu IRC channel) and see what I can come up with.

[afbjorklund]

Presumably you could write a FreeBSD-specific "driver", similar to macOS Virtualization.framework ?

Like #889

---

This looks helpful:
https://vmrc.bsd.lv

Examples for Xen and Bhyve

[afbjorklund]

@heywoodlh : you should be able to use "max", for the emulator:

$ qemu-system-x86_64 -cpu help

x86 base base CPU model type with no features enabled
x86 host KVM processor with all supported host features
x86 max Enables all features supported by the accelerator in the current host

Not sure it would help with performance, but it would not hurt to have the CPU extensions.

Probably need a sibling function to IsNativeArch, like IsAcceleratedOS or somesuch...

[heywoodlh]

@afbjorklund yeah, I've tried it with -cpu max and you're right in that it doesn't seem to help with performance with no accelerator. But it's a good point that it would serve as a generic CPU target to emulate.

[afbjorklund]

FreeBSD lacks support for containerization generally but it does support QEMU.

Always ironic since BSD doesn't support containers, but it does support jails. But that's another story*...

* https://github.com/containerd/nerdctl/blob/master/docs/freebsd.md

And https://wiki.freebsd.org/LinuxJails

I guess bhyve is your best VM bet, now that kqemu doesn't seem to be supported by modern versions ?

[afbjorklund]

@heywoodlh Note also that: "Vultr cloud servers do not support nested virtualization."

So I guess you would have to go for the Bare Metal option, if you want to start Linux VMs ?

Just looking at https://www.vultr.com/

🐻 🤘

[heywoodlh]

Shoot, thanks for checking that -- for some reason I thought Vultr had enabled nested virtualization! I'll test on a bare metal instance and see how performance is there.

[heywoodlh]

Spoke too soon: looks like you can't deploy FreeBSD on Baremetal on Vultr. So I'll test on some actual hardware. Will follow up with how it goes!

[afbjorklund]

I made a PR to avoid -cpu host on FreeBSD, note that Lima by default uses a more conservative type (than "max")

        cpuType := map[Arch]string{
                AARCH64: "cortex-a72",
                // Since https://github.com/lima-vm/lima/pull/494, we use qemu64 cpu for better emulation of x86_64.
                X8664:   "qemu64",
                RISCV64: "rv64", // FIXME: what is the right choice for riscv64?
        }

I think the Cortex-A72 is equivalent to a Raspberry Pi. Previously it was using Haswell-v4, for the x86_64 architecture.

• Only use cpuType=host for OS with accel available #894

[afbjorklund]

Not sure why that fails just on FreeBSD but nothing else.

It was broken on NetBSD too, CI only tests Mac and Linux.

This code was: //go:build !darwin && !linux

• Remove nonexisting err from AES detection warning #895

[afbjorklund]

Lima could bring Docker support to a platform that currently lacks it

If I remember correctly, it had support for docker-machine and virtualbox

https://cgit.freebsd.org/ports/tree/sysutils/docker-machine

https://cgit.freebsd.org/ports/tree/emulators/virtualbox-ose

[afbjorklund]

@heywoodlh

I was now able to compile and run qemu and lima on FreeBSD 13.1 :

FreeBSD generic 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC arm64

Worked OK - even on arm64, but timed out (10m) while creating the VM...

Was running it on the Raspberry*, so took minutes instead of seconds.

* my old BSD box was broken

[   51.416337] Run /init as init process
[   61.767736] Alpine Init 3.6.0-r0
...
[   87.530402] Loading boot drivers: ok.
[   89.246475] Mounting boot media...
...
[  460.428534] Installing packages to root filesystem...
[  656.511692] Installing packages to root filesystem: ok.

Welcome to Alpine Linux 3.15
Kernel 5.15.38-1-virt on an aarch64 (/dev/ttyAMA0)

lima-alpine login: 

Some observations:

• Installing "python" and "meson" was missing from the list of package dependencies in your blog post

• Compiling qemu goes faster if narrowing down the target list to x86_64-softmmu,aarch64-softmmu etc

• No 9p: ERROR: Feature virtfs cannot be enabled: virtio-9p (virtfs) requires Linux or macOS

• The processor features (like AES) can be found in /var/run/dmesg.boot after BSD machine is booted

---

I don't think it will be very useful, without hardware acceleration available ?

Even if it "just" runs 10x slower rather than 100x like here, it is still not usable.
Containers are slow enough (non-native) on a VM with hardware acceleration.

So the best would be to have another VM driver, for virtualbox or for bhyve.
(Neither is available for arm, but nobody runs FreeBSD outside x86 anyway)

• https://wiki.freebsd.org/VirtualBox

• https://wiki.freebsd.org/bhyve/UEFI

Using jails would be nicer from a technical perspective, to avoid the extra layer.

Then again, WSL1 (syscalls) was replaced by WSL2 (virtual machine) 🤷

• https://wiki.freebsd.org/Linuxulator

• https://docs.microsoft.com/en-us/windows/wsl/compare-versions

[tuxillo]

What happened to this eventually? Is FreeBSD a supported platform now?

@heywoodlh do you know if there are any plans to add this to FreeBSD ports? Can't see anything in freshports.

In any case I made it run in DragonFly BSD as well (with some tweaks to the vendor software etc):

~/s/lima$ uname -a
DragonFly devbox.localhost 6.5-DEVELOPMENT DragonFly v6.5.0.11.gefd26-DEVELOPMENT #0: Tue Jan 17 10:40:18 CET 2023     root@devbox.localhost:/usr/obj/home/user/s/dragonfly/sys/X86_64_GENERIC  x86_64
~/s/lima$ lima nerdctl run -it --rm hello-world
command-line line 0: Unsupported option "gssapiauthentication"

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

I'm quite impressed on how well lima-vm works in DragonFly BSD, I was expecting a lot of pain but it wasn't the case :-D

[heywoodlh]

Outside of doing the initial work to get Lima working on FreeBSD, I haven't really touched this since. I don't really use FreeBSD, I was mostly pursuing this as an experiment.

I eventually got Lima working on a FreeBSD running on normal hardware and it seemed to work pretty okay for me. Additionally, I have received feedback from readers of my blog that it worked for them as well.

I know I'm not going to put any more effort into this, such as building a FreeBSD port for this. But, if somebody did that, this could potentially make it easier to use Linux containers in Lima/QEMU on FreeBSD.

Anyway, if you or anyone does more work on getting Lima running (better) on BSD, I would absolutely be interested and would hope you would update this issue. 😀

[afbjorklund]

Same here, I sympathize with the "cause" - but don't really use FreeBSD anywhere anymore.

Once upon a time, it was used an Open Source escape route when Apple closed down Darwin...

[AkihiroSuda]

DragonFly

Nice, does that work with hardware acceleration?

[afbjorklund]

I think it supports "nvmm" ?

https://www.dragonflybsd.org/docs/docs/howtos/nvmm/

[tuxillo]

DragonFly

Nice, does that work with hardware acceleration?

Yep, I had to do some vendor/lima patching that I'll upstream soon.

❯ sudo nvmmctl list
Machine ID VCPUs RAM  Owner PID Creation Time           
---------- ----- ---- --------- ------------------------
0          4     4.0G 672994    Sat Mar 25 00:51:56 2023

And the actual qemu process:

user   672994  0,0  8,7  4.50G  2.84G  9  I10    12:51a. m.  39:17,77 /usr/local/bin/qemu-system-x86_64 -m 4096 -cpu max -machine q35,accel=nvmm -smp 4,sockets=1,cores=4,threads=1 -boot order=c,splash-time=0,menu=on -drive file=/home/user/.lima/default/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/home/user/.lima/default/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:45:eb:63 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/home/user/.lima/default/serial.sock,server=on,wait=off,logfile=/home/user/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/home/user/.lima/default/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-default -pidfile /home/user/.lima/default/qemu.pid

I must say it's pretty snappy!

[AkihiroSuda]

Yep, I had to do some vendor/lima patching that I'll upstream soon.

Do you plan to submit a PR?

[tuxillo]

Yep, I had to do some vendor/lima patching that I'll upstream soon.

Do you plan to submit a PR?

Ooops, I forgot about this one, sorry! If I recall correctly I had to patch vendors before being able to actually use it.

[afbjorklund]

Building with GOOS=dragonfly says:

# runtime/cgo
gcc_dragonfly_amd64.c:6:10: fatal error: sys/signalvar.h: No such file or directory
    6 | #include <sys/signalvar.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/containerd/continuity/fs
../go/pkg/mod/github.com/containerd/continuity@v0.4.1/fs/copy.go:118:12: undefined: copyFileInfo
../go/pkg/mod/github.com/containerd/continuity@v0.4.1/fs/copy.go:122:12: undefined: copyXAttrs
../go/pkg/mod/github.com/containerd/continuity@v0.4.1/fs/copy.go:172:13: undefined: copyFileInfo
../go/pkg/mod/github.com/containerd/continuity@v0.4.1/fs/copy.go:176:13: undefined: copyXAttrs
../go/pkg/mod/github.com/containerd/continuity@v0.4.1/fs/copy.go:202:9: undefined: copyFileContent

Probably just missing a "go:build"
https://github.com/containerd/continuity/blob/v0.4.1/fs/copy_unix.go

EDIT: the first error is "normal", you need a cross-compiler and headers I think ?

# runtime/cgo
gcc_freebsd_amd64.c:7:10: fatal error: sys/signalvar.h: No such file or directory
    7 | #include <sys/signalvar.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.

So there are some extra steps in order to cross-compile from Linux, unlike Darwin.

https://wiki.freebsd.org/BuildingOnNonFreeBSD

[AkihiroSuda]

signalvar.h

CGO_ENABLED=0 may work?

[afbjorklund]

Possibly, but then we need to isolate host agent into a separate binary first ?

[tuxillo]

I don't get that include error, not sure which Go version you're using. I've fixed locally some vendor stuff but I'm stuck here now:

❯ gmake
rm -rf _output vendor
mkdir -p _output/bin
cp -a ./cmd/lima _output/bin/lima
# The hostagent must be compiled with CGO_ENABLED=1 so that net.LookupIP() in the DNS server
# calls the native resolver library and not the simplistic version in the Go library.
CGO_ENABLED=1 go build -ldflags="-s -w -X github.com/lima-vm/lima/pkg/version.Version=v0.17.0-32-g8e5eb5f" -tags "" -o _output/bin/limactl ./cmd/limactl
# github.com/lima-vm/lima/pkg/networks/usernet
pkg/networks/usernet/gvproxy.go:89:23: undefined: transport.Listen
gmake: *** [Makefile:115: _output/bin/limactl] Error 1

Maybe need to add a listen_dragonfly.go or something.

[afbjorklund]

I don't get that include error, not sure which Go version you're using.

The error was on Linux, but using CGO_ENABLED=0 avoids it.

(go1.20 something)

What is "vendor stuff" ?

[tuxillo]

What is "vendor stuff" ?

I mean some of the dependencies that the build process download and place in GOPATH. I had to modify them manually (and do some cache cleaning after making the changes). I would need to upstream those changes before Lima can be built directly, unless I'm missing something.

[afbjorklund]

Right, like ../go/pkg/mod/github.com/insomniacslk/dhcp@v0.0.0-20220504074936-1ca156eafb9f/interfaces/bindtodevice_bsd.go which is missing a dragonfly

[tuxillo]

Yes, exactly 😄

In any case I was able to build it locally and it's able to start up some containers. I don't know how much functionality is missing tho.

I'll see if I can fix the dependencies before submitting the PR.

[tuxillo]

In any case, DragonFly BSD supports NVMM (which can be used with the QEMU driver) but FreeBSD has BHYVE which would need a driver. I'm kind of hijacking the FreeBSD issue, wouldn't it make sense to create a different one for DragonFly?

[afbjorklund]

Maybe need to add a listen_dragonfly.go or something.

Or if possible a listen_bsd.go for all of them, unless it varies ?

Can you open an upstream issue for it (gvproxy), as well ?

[tuxillo]

Maybe need to add a listen_dragonfly.go or something.

Or if possible a listen_bsd.go for all of them, unless it varies ?

Can you open an upstream issue for it (gvproxy), as well ?

Yeah, I'm planning on going through all the failing deps. I'll handle the gvproxy too.

[tuxillo]

I think that's all the upstream PRs needed, being the gvisor vsock one required for the BSDs as well. Now we wait I guess 😃

[cfergeau]

An alternative is to use libvirt which has a bhyve driver https://libvirt.org/drvbhyve.html

[AkihiroSuda]

libvirt seems too complicated

[tuxillo]

Quick update on DragonFly BSD, still getting an error:

# github.com/insomniacslk/dhcp/dhcpv4
../../go/pkg/mod/github.com/insomniacslk/dhcp@v0.0.0-20220504074936-1ca156eafb9f/dhcpv4/bindtointerface.go:9:20: undefined: interfaces.BindToInterface
gmake: *** [Makefile:169: _output/bin/limactl] Error 1

[tuxillo]

Just tested latest master commit in DragonFly BSD and it builds. I should probably send a PR already.