lima+wayland

[jfmc]

Thanks for this amazing project!

I wonder if it would make sense devise a roadmap to support high-performance graphics in the style of https://github.com/microsoft/wslg ?
"WSLg is short for Windows Subsystem for Linux GUI and the purpose of the project is to enable support for running Linux GUI applications (X11 and Wayland) on Windows in a fully integrated desktop experience."

There were some discussions about Wayland on macOS (Miguel de Icaza https://twitter.com/migueldeicaza/status/1254929025988005889). XQuartz worked great before retina display was introduced but AFAIK support for full resolution was never completed (https://gitlab.freedesktop.org/xorg/xserver/-/issues/187).

[AkihiroSuda]

Does QEMU support it? If so, yes, Lima will be able to support it.

[fredldotme]

I don't think QEMU alone is going to help here, it needs a compositor + a host-side renderer to achieve something like WSLg.
EDIT: According to the Virgl webpage it sounds like accelerated virtual GPUs are only possible with the GTK3 frontend of QEMU.

I'm not sure of the performance impact of using vsock (just read about it) but it would be interesting to have a custom Wayland compositor in the guest, where the pixel buffer of a surface is forwarded over vsock to a host-side client which displays the pre-rendered pixel buffer. I don't think DMA-BUF is possible using current means, so copying between the guest and the host seems necessary for a start.

Writing a custom guest-side Wayland compositor using something like Mir (https://mir-server.io/, there are others, but that's the one I'm most familiar with) sounds like one approach. It's supported by a company, has a pluggable platform mechanism (which could handle the pixel buffer forwarding), it's in most distro repos and would handle the Wayland client protocol.

EDIT:
I see Microsoft is using Weston + RDP on WSLg. This could also solve the problem, I'm weighting options as I'd be willing to implement something similarly using Mir. Maybe RDP is not going to cut it if one day it becomes possible to share a guest VMs GPU resources with the host system.

[fredldotme]

Is something like this of interest to the Lima developers @AkihiroSuda? I'm not sure of Lima/QEMU's limitations currently, so if there is interest I would like to use this issue to plan ahead, collecting opinions etc.

[AkihiroSuda]

I guess we can use https://github.com/knazarov/homebrew-qemu-virgl (https://gist.github.com/akihikodaki/87df4149e7ca87f18dc56807ec5a1bc5) to support this, but anyway we need to have the app nap issue to be fixed first https://gitlab.com/qemu-project/qemu/-/issues/334

[fredldotme]

Agreed. That's the QEMU fork I'm currently focusing on, yeah.
Also, the comment on how to fix App Nap was done by me. But maybe we can even get away with making an off-screen window in QEMU to allow handling OpenGL acceleration, avoiding the need to spawn an actual QEMU Cocoa window and App Nap screwing around in the first place.

What it needs though is a macOS-side app that presents the pixels pushed by the guest-side compositor to the host. I'm not familiar with GUI programming in Go, so I would probably do it in C++ using something like SDL or Qt if that's okay.

[akihikodaki]

Hi there, I'm the author of the fork.

As you may noticed know, there is a major problem in graphics virtualization: zero-copy buffer sharing. DMA-BUF is a solution for this but specific to Linux. The problem appears all over the place.

The issue has two different parts:

1. providing a "headless" graphics device to the guest
2. integrating the guest windows to the host window system

1 is not that hard. QEMU already has egl-headless, which provides headless OpenGL for the SPICE and VNC server. It should work with ANGLE, an OpenGL ES compatibility layer which I use with my fork. However, egl-headless depends on GBM to provide zero-copy buffer sharing. The feature is optional, so you can just add #ifdefs to remove the dependency. egl-headless may require some additional code to configure ANGLE properly. As it would be a simple command line program, App Nap should not matter.

2 is the difficult part. While X was designed as a remote protocol, Wayland is designed for zero-copy buffer sharing. As such, Wayland relies on shared memory and Direct Rendering Mechanism provided by Linux kernel, which are obviously incompatible with remote communication.
A possible solution here is to create a Wayland compositor which talks a protocol for remote communication. Waypipe is a transparent proxy program specific to Wayland. WSLg used RDP with their proprietary VAIL extension to allow zero-copy buffer sharing for RAIL (remote applications integrated locally). VAIL relies on an interface of Windows and Hyper-V to share frame buffers among them.
You have to develop a Wayland compositor for macOS if you use Waypipe. If you use RDP-RAIL, you may be able to use Microsoft Remote Desktop for mac and bunch of open source software modified for WSLg. (Microsoft Remote Desktop is of course proprietary. A free alternative, FreeRDP does not support RAIL on macOS, but I think implementing RAIL on FreeRDP on macOS is much easier than implementing a full Wayland compositor.) Either way you choose, you will not have zero-copy buffer sharing. Ultimately the problem should be solved by implementing a VAIL-like protocol, which requires to hack all of the guest-side proxy, QEMU, and the host-side compositor. Today Macs have Retina displays so zero-copy buffer sharing is very important in my opinion.

[fredldotme]

Hey @akihikodaki, thanks for jumping into the discussion.

Yeah I'm aware that writing a whole Wayland compositor from scratch might be a little much, so I'm thinking of using something off-the-shelf, like Mir. I've talked to the Mir developers about a two-step PoC process:

1. Write a simple Mir shell that glReadPixel's every surface on damage and sends the pre-rendered buffer over the wire to the host compositor. It's enough for an initial PoC, though not very performant since pixels get pushed and pulled from the vGPU unnecessarily.

2. Write a Mir "platform" that receives raw, unrendered buffers (not sure when that is ready in Mir yet) and send that over the wire. It's not zero copy, but still better than approach 1.

Right now, since the RAIL/VAIL stuff seems a little complicated for my taste, I might as well implement a custom protocol for now and abstract that away as soon as other alternatives prove to become better.

[fredldotme]

The project for my proposal is now located at: https://github.com/fredldotme/stereopticon

This repo includes sources for both the guest as well as the host side, and it's using Mir (guest compositor) and SDL (host window painter).

Both processes are able to spawn but are not exchanging data yet, that will come soon(tm).
It's now using a custom protocol (structs located in src/common/guestpixelbuffer.h) instead of RDP, good enough for most applications.
What's implemented are the ability to spawn windows, redraw them and destroy them again as the guest compositor sees fit.

I welcome contributions, but keep in mind that I haven't decided on the license yet, other than on the guest side (GPLv2 OR GPLv3)
I'm thinking the shared common code and host side might be better off with Zlib though.

[fredldotme]

I just tried to implement vsock capabilities into the compositor but it turns out that QEMU on macOS can't be built with that support enabled.
@akihikodaki do you by any chance know what's necessary to get this kickstarted?

[akihikodaki]

@fredldotme Apparently vsock relies on vhost, which is a Linux kernel feature. vhost-user is a userspace alternative, but it may also rely on some Linux-specific. (At least, it was not compatible enough with macOS to bring vhost-user-gpu. It provides a process isolation of graphics stack and is necessary for security and reliability.)

If vhost-user does not work, there are two alternatives: ivshmem and TCP/IP.

ivshmem stands for Inter-VM shared memory device. It can share memory and deliver interrupts among guests and the host. ivshmem-server is a server and QEMU and your application will connect to it with a library named ivshmem-client. Both of ivshmem-server and ivshmem-client are included in QEMU. It can be exposed as a PCI device to the guest but the kernel apparently does not have a driver so you will instead use uio_pci_generic. It should be trivial enough to use the device without a special driver anyway. See docs/specs/ivshmem-spec.txt to know how to use the device. Still, programming with shared memory is difficult by nature while it should provide better performance.

TCP/IP is the easiest for sure. I don't think there are much overhead when comparing with vsock. The greatest gap should exist between ivshmem and socket-based communications (including both of vsock and TCP/IP).

[fredldotme]

Thanks for the valuable info @akihikodaki! Highly appreciated!
So based on that, I'm actually thinking of implementing both.

There's something to be said of running a RPi4 connected to an iPad via USB-C and having TCP be used in that scenario, as I'm already doing regularly via SSH, as well as having a locally-optimized variant for Guest-VM-communication via shared memory, assuming it's really not doing copies somewhere.

Doing a semaphore between a guest and the host might be interesting for sure. If it turns out to be too cumbersome I will just scan out prepared buffers based on the host's vsync.

[AkihiroSuda]

macOS seems already supporting vsock, though not supported by QEMU
https://github.com/apple/darwin-xnu/blob/xnu-7195.81.3/bsd/man/man4/vsock.4

[akihikodaki]

macOS seems already supporting vsock, though not supported by QEMU
https://github.com/apple/darwin-xnu/blob/xnu-7195.81.3/bsd/man/man4/vsock.4

The interface is for the guest, not for the host. The host implementation is up to the virtualization application, and QEMU in particular uses vhost, which is specific to Linux kernel.

[fredldotme]

I'm struggling to find ways how to word this:

My approach currently implies queue semantics. Add commands to the host-queue, read that out until empty, and proceed to do what the guest compositor wants the host to do during read.
I haven't done that sophisticated shared memory development yet, though I'm willing to try.
Having variable-size messages inside a shared memory "queue" will be hard, let alone draining it fully, without leaving traces.
Does anyone have a suggestion here on how to proceed?

EDIT: I'll go with a simple ring buffer and fake pointers (calculating offsets of messages on the fly) for a start, I'll let you know how it goes.