FreeBSD Support (host)

[heywoodlh]

Description

FreeBSD lacks support for containerization generally but it does support QEMU. I was able to get Lima running on FreeBSD 13.0 but it doesn't work out of the box. I suspect that it wouldn't be difficult to fine-tune to get it up and running. And I feel that Lima could bring Docker support to a platform that currently lacks it which would be awesome.

I wrote-up a bit more detail on how I got Lima running on FreeBSD 13.0 here: https://the-empire.systems/freebsd-docker-2022

It's extremely slow but it works! I'm sure that I could have added something to QEMU's settings to make it perform better but I'm ignorant to the features that would help in this situation.

Installing Lima on FreeBSD 13.0:

1. Compile+install Lima from source:

pkg install gmake go git
git clone https://github.com/lima-vm/lima /opt/lima
cd /opt/lima

For some reason, compiling Lima fails with: pkg/sshutil/sshutil_others.go:14:19: undefined: err

Adding the following to the detectAESAcceleration() function in pkg/sshutil/sshutil_others.go fixes it:

var err error

Once err is defined, Lima compiles just fine with gmake:

gmake && gmake install

Not sure why that fails just on FreeBSD but nothing else.

2. Compile+install QEMU 7.0 from source:

pkg install ninja pkgconf glib pixman

git clone --depth=1 git://git.qemu-project.org/qemu.git /opt/qemu
mkdir -p /opt/qemu/build
cd /opt/qemu/build

../configure
gmake ## add -j to speed up the compilation
gmake install

Not sure how necessary this actually is but the version of QEMU packaged with the FreeBSD repos is 6.2.0 so upgrading to 7.0 seemed like a good idea.

3. Set the CPU to emulate:

FreeBSD doesn't have -cpu host (unlike Linux and MacOS) available so I had to manually specify the processor on my machine (I was using an Intel VPS with Vultr):

export QEMU_SYSTEM_X86_64="qemu-system-x86_64 -cpu Cascadelake-Server"

Problems needed to be solved before it's usable on FreeBSD:

1. Slowness (I am not to sure where to start there but I suspect that -cpu host would really help with this)
2. Minor: Fix that undefined: err in pkg/sshutil/sshutil_others.go for FreeBSD (could be as simple as defining that variable if the OS is FreeBSD)
3. Implement logic to not use -cpu host if OS is FreeBSD

It would be so nice to be able to interact with containers while on FreeBSD. Please let me know if you have any questions or if this is not something desired to support.

[AkihiroSuda]

Thanks, would it be possible to patch QEMU to use bhyve as the accelerator just like KVM/HVF/NVMM ?

[heywoodlh]

I'm not sure. But I did a bit more research and it looks like per this forum post you can build the QEMU port with the KQEMU kernel module to provide acceleration. So I'm gonna give that a try and see if that helps with the speed issue.

[AkihiroSuda]

KQEMU is dead

[heywoodlh]

Ah, I did not realize that. Looking through the documentation it doesn't look like there is any alternative accelerator to KQEMU specifically that would work for FreeBSD.

I see why you leapt to utilizing Bhyve for acceleration. Unfortunately, at the moment I don't know enough about QEMU to patch it to work with Bhyve and searching around online hasn't yielded any results for me on how to do that.

I'll try to do some more digging (perhaps reach out to the #qemu IRC channel) and see what I can come up with.

[afbjorklund]

Presumably you could write a FreeBSD-specific "driver", similar to macOS Virtualization.framework ?

Like #889

---

This looks helpful:
https://vmrc.bsd.lv

Examples for Xen and Bhyve

[afbjorklund]

@heywoodlh : you should be able to use "max", for the emulator:

$ qemu-system-x86_64 -cpu help

x86 base base CPU model type with no features enabled
x86 host KVM processor with all supported host features
x86 max Enables all features supported by the accelerator in the current host

Not sure it would help with performance, but it would not hurt to have the CPU extensions.

Probably need a sibling function to IsNativeArch, like IsAcceleratedOS or somesuch...

[heywoodlh]

@afbjorklund yeah, I've tried it with -cpu max and you're right in that it doesn't seem to help with performance with no accelerator. But it's a good point that it would serve as a generic CPU target to emulate.

[afbjorklund]

FreeBSD lacks support for containerization generally but it does support QEMU.

Always ironic since BSD doesn't support containers, but it does support jails. But that's another story*...

* https://github.com/containerd/nerdctl/blob/master/docs/freebsd.md

And https://wiki.freebsd.org/LinuxJails

I guess bhyve is your best VM bet, now that kqemu doesn't seem to be supported by modern versions ?

[afbjorklund]

@heywoodlh Note also that: "Vultr cloud servers do not support nested virtualization."

So I guess you would have to go for the Bare Metal option, if you want to start Linux VMs ?

Just looking at https://www.vultr.com/

🐻 🤘

[heywoodlh]

Shoot, thanks for checking that -- for some reason I thought Vultr had enabled nested virtualization! I'll test on a bare metal instance and see how performance is there.

[heywoodlh]

Spoke too soon: looks like you can't deploy FreeBSD on Baremetal on Vultr. So I'll test on some actual hardware. Will follow up with how it goes!

[afbjorklund]

Might be easier to just use two cloud VMs, one FreeBSD one Linux.

That is what you will end up with, anyway ? If not linux jails