Remove the Qemu in Docker functionality

[justincormack]

This was introduced before qemu had full acceleration support
on OSX. Now the Qemu platform supports most platforms it does
not make som much sense, and allows us to simplify the code.

Signed-off-by: Justin Cormack justin@specialbusservice.com

[oliviabarrick]

One nice thing about Qemu in Docker is that it removes any risk of version differences and also means that users do not need to separately install Qemu (for example, on Mac OS, that's an extra brew install).

Not sure if this is is important for you, but thought I'd mention it as I've found it useful to package Qemu in Docker myself.

[justincormack]

@justinbarrick since Qemu started supporting Hypervisor.Framework on OSX its actually much faster to run it natively than in a container as you get full hardware acceleration, so I kind of changed my mind about this. But if it is useful we can leave it (my thought it was more confusing as it might run differently without being obvious why not).

[justincormack]

Alternatively we could make it opt in only, so you need to specify --containerized

[rn]

We used to use the containerized version on CI (until we installed qemu in the CI machines and also enabled nested virt). I've seen CI failures which were not reproducible with accelerated qemu (both on KVM and HVF) and saw mixed results with TCG. So having a known baseline version might be useful. OTOH, we had issues/regressions with running qemu containerised in the past and we don't really test it. Removing this option would clean up the code a bit too.

So I'm in two minds about removing it, but wouldn't object merging this unless someone has a strong opinion otherwise.

[oliviabarrick]

Accelerated versus non-accelerated is a good point on Mac OS as that could cause unexpected behavior. Do you know if it is possible to nest HVF virtualization and run it in Docker for Mac still?

My ideal situation would be containerized Qemu that still supports acceleration. This is how we deploy our images - Qemu inside Docker in Marathon. Inside that image, before running Qemu, we setup the disk, TAP interface, metadata disk, and run the image in a loop. But we’re not easily able to test the image in that environment on a Mac because our options are run directly in HyperKit or an unaccellerated Qemu container.

My container doesn’t actually invoke Linuxkit to run Qemu at runtime (we call Qemu from a bash script), but for users that deploy their images to Qemu and dev on a Mac, the closer it can be to a standard environment that people would use Qemu in the better (i.e. a Linux-like host).

[rn]

you can't run "nested" on macOS. The qemu in the (linux) container almost certainly does not have the acceleration compiled in for HVF (and wn't be able to use it even if one could map this in somehow).

If we run qemu containerised on Linux we mount in /dev/kvm if present so you'd get acceleration there even if run in a container

[oliviabarrick]

Bummer. That does remove much of the utility there for me. Maybe someday!

[justincormack]

@justinbarrick VMWare Fusion is the only reliable way of getting nested virt on OSX. You should be able to set it up for testing; we do have LinuxKit support for it too.

Wondering if I should just split qemu in docker to a different target instead of removing it?

[rn]

Wondering if I should just split qemu in docker to a different target instead of removing it?

That might be a cleaner solution, depending on how much code (apart from cmdline parsing and a few other bits) is shared between them. I do find the current code a little convoluted.

[dave-tucker]

👋 @justincormack this can be closed now since #3443 was merged.

[rn]

@dave-tucker thanks for picking this up.