DRM leasing

[ddevault]

I'm working on implementing DRM leasing (and with it, VR support) and wanted to start the discussion here on how it should shake out. For those unaware, DRM leasing allows us to "lease" some DRM resources (connector, crtc, encoder) to another process, allowing us to share the DRM master. This is used by VR applications to have exclusive and low-latency control over the HMD (head mounted display). I think a comprehensive solution for VR on Wayland (and wlroots) involves the following broad steps, in no particular order:

• Expose the "non-desktop" flag to downstream compositors, so they can decide how to use non-desktop outputs (HMDs generally have this flag set, and are generally unsuitable for inclusion in the desktop layout). A typical compositor will not use this output at all, and will make it available for leasing. A VR compositor might seek out this output specifically to run its VR interface on. etc.
• Clean up and get merged the DRM leasing Wayland protocol extension. I have a cleaned up draft here for your consideration.
• Implement wlr_drm_lease_manager_v1. We could do this at a low-level or a high-level. The low-level API would involve expanding the public DRM backend API so that its users can enumerate our DRM resources and offer fine-grained leases of connectors, crtcs, and encoders separately. Until someone comes up with a use-case that demands this, I'd prefer a high-level approach: the downstream compositor just tells wlr_drm_lease_manager_v1 which wlr_outputs it would like to offer for leasing. This will still require some changes to the DRM backend to allow the lease manager to ask the backend to stop using certain resources, but it could just use the existing DRM backend headers to dig into DRM internals and since it all lives inside of wlroots (rather than downstream compositors), it simplifies the maintenance burden significantly.
  • In the high-level model, when a client takes you up on the lease offer, the backend will emit an output_remove event for that wlr_output, so you stop using it. It'll reappear if you revoke the lease, and you can still access it (though it'll be a programming error to mutate it) by browsing the list of leases.

At that point, we can write our own Wayland clients which take advantage of DRM leasing to provide a VR experience. I'll write some basic demo as part of this workstream. However, to be useful, we'll probably want it to work with OpenVR (which is a misnomer, because it's closed source) - this is what Steam uses. It only supports X11, so my plan is to extend Xwayland with support for the DRM leasing protocol. DRM leasing via X11 was added to xrandr 1.6.

In the long term, I want to expand wlroots to support VR compositors better. This will involve at least:

• Wayland protocols for rendering 3D content (hard)
• Wayland protocols for VR-specific HIDs - plus libinput/eudev support? Today these use libusb and bypass the input subsystem entirely

[ascent12]

Clean up and get merged the DRM leasing Wayland protocol extension. I have a cleaned up draft here for your consideration.

It may be worth making the protocol multi-GPU aware. DRM object IDs are guaranteed to be unique across different devices, but I don't think we can create leases involving multiple devices (although I can't say I've tried). It could also help a client choose which device it wants to render with. I don't know if I client actually has a way to determine what device it has a lease for otherwise.

I'm don't think zwp_drm_lease_connector_v1 is flexible enough. "name" alone isn't enough information to identify most things properly. Relying on compositor-specific information from other wayland protocol objects doesn't seem like the best thing. It needs to be something more rigid.
Ideally, we could just give them the DRM object ID and they can query whatever they want themselves, but I don't think that'll work, because the client can't necessarily open /dev/dri/cardX.

Looking at the DRM API, it doesn't seem that there is a way to actually modify existing leases. We have to destroy and create them again.

I'd prefer a high-level approach: the downstream compositor just tells wlr_drm_lease_manager_v1 which wlr_outputs it would like to offer for leasing

Agreed. The DRM object matching logic is already stupidly complicated enough, and I'm sure that users don't really want to deal with that or have much they can add.

However, to be useful, we'll probably want it to work with OpenVR (which is a misnomer, because it's closed source) - this is what Steam uses

I'm just going to bring up
https://www.khronos.org/openxr
https://gitlab.freedesktop.org/monado/monado

XR is not something I know a lot about or how it works, as well as these aren't complete technologies yet, but surely this is relevant.

In the long term, I want to expand wlroots to support VR compositors better. This will involve at least:

Another potential (non-VR) use-case is running multiple Wayland/X11 servers on the same GPU, independent of each other. Kind of like how logind multiseat works.

[ddevault]

It may be worth making the protocol multi-GPU aware

Hm, good point. I'll ponder this a bit.

Ideally, we could just give them the DRM object ID and they can query whatever they want themselves, but I don't think that'll work, because the client can't necessarily open /dev/dri/cardX

Yeah, I don't think that this is ideal, either, but I also don't know of a good way to expand on it elegantly. I think this is good enough for v1 because the usual case is not going to be affected - most compositors will only offer VR headsets for lease, and there'll probably only be one of them. And if not - the name is probably Good Enough to disambiguate them until we figure out something better.

Looking at the DRM API, it doesn't seem that there is a way to actually modify existing leases. We have to destroy and create them again.

Hm, drmModeChangeLease was proposed, but I guess it didn't make it. It looks like it almost did, I found a kernel with the ioctl implemented. I'll dig and see why this didn't land and update the protocol accordingly. Bleh.

OpenXR

See also: OpenHMD. I'd like to support these as well, but they'll almost certainly rely on DRM leasing as well so I think the work here translates quite directly to that. This stuff hasn't materialized in an entirely practical way yet, so I'm thinking of Steam only in terms of something useful to test with. Of course, I would like to dive into OpenHMD and get it working on Wayland as well.

Another potential (non-VR) use-case is running multiple Wayland/X11 servers on the same GPU, independent of each other. Kind of like how logind multiseat works.

Aye. I plan on adding support for this to the Wayland backend, where Wayland bootstraps a DRM lease and hands it over to the DRM backend, for this very purpose. It should be a pretty cut-and-dry change.

[emersion]

I'd prefer a high-level approach

Works for me.

when a client takes you up on the lease offer, the backend will emit an output_remove event for that wlr_output, so you stop using it.

We don't have output_remove, we only have wlr_output.destroy. I suppose that could work, too.

Today these use libusb and bypass the input subsystem entirely

They'll probably want to keep doing so, to minimize latency. However some VR Wayland clients would need such a protocol to benefit from compositor multiplexing.

Multi-GPU

Indeed, that needs to be considered. But see below, maybe not in this end.

I don't know if I client actually has a way to determine what device it has a lease for otherwise.

drmGetDevice should work.

I'm don't think zwp_drm_lease_connector_v1 is flexible enough. "name" alone isn't enough information to identify most things properly. Relying on compositor-specific information from other wayland protocol objects doesn't seem like the best thing. It needs to be something more rigid.
Ideally, we could just give them the DRM object ID and they can query whatever they want themselves, but I don't think that'll work, because the client can't necessarily open /dev/dri/cardX.

I wonder if the client should choose at all.

I'm starting to think that maybe the client should indicate why it needs a lease. For instance a VR client could say "I want non-desktop outputs", instead of saying "I want this particular output". In case multiple DRM nodes or non-desktop outputs are available compositors could provide configuration options to choose what to lease.

---

Also, another note: surely this protocol should be privileged, all clients shouldn't be able to randomly create leases. As such the protocol might be rejected from wayland-protocols (or at least the current wayland-protocols).

[ddevault]

They'll probably want to keep doing so, to minimize latency

This makes more sense for e.g. head tracking than e.g. controller input, imo

I'm starting to think that maybe the client should indicate why it needs a lease. For instance a VR client could say "I want non-desktop outputs", instead of saying "I want this particular output". In case multiple DRM nodes or non-desktop outputs are available compositors could provide configuration options to choose what to lease.

With some rejiggering (a lot, rather), we could allow the client to become a DRM master with no leased resources, scan connectors themselves for the outputs they want, then ask for those in particular.

[emersion]

With some rejiggering (a lot, rather), we could allow the client to become a DRM master with no leased resources, scan connectors themselves for the outputs they want, then ask for those in particular.

I'm unfamiliar with leases. What happens when a client gets a lease? Does the leased FD expose all connectors? Or does the client only have a restricted view of the DRM node?

[ddevault]

They can see everything, but only mutate the leased things. If I understand it correctly.

[ddevault]

Update on drmModeChangeLease: I reached out to Keith to ask about why it didn't make the cut, if it wasn't a fundamental issue I'll adopt the effort and try to get it merged. I updated the protocol regardless, to make it more vague and support both the case where drmModeChangeLease exists and the case where it doesn't in a reasonably backwards-compatible way.

Update v2: it looks like it wasn't added just because it wasn't necessary for VR.

[ddevault]

Regarding the question of multiple GPUs, a reasonably clean answer (for the time being) occurred to me as I was working on the protocol implementation: a different drm_lease_manager_v1 global for each DRM master offering leases.

[ascent12]

@ddevault asked for a description of the DRM object matching, so I'll post it here. It should probably be transferred over to a comment in the source code itself.

For a basic intro to the DRM object types, see https://github.com/ascent12/drm_doc. However, I never did get around to talking about planes or the atomic API.
Since planes are relevant here, I'll quickly describe them. A plane is attached to a CRTC, and there are 3 kinds.

• Primary: The main surface, has the lowest z-order, there is always exactly 1 plane per CRTC
• Cursor: Intended for hardware cursors, has the highest z-order, in practice there is exactly 1 or 0 of these per CRTC
• Overlay: The most general, z-order is undefined respective to each other (unless zpos property exists), basically any amount per CRTC is possible (including 0), may work with multiple CRTCs. In practice, they are tied to a single CRTC, at least on the hardware I've used. But to write properly general code, this assumption shouldn't be made. wlroots does not support these yet, but there are some remnants of these in the code.

The drmModePlane.possible_crtcs value says which CRTCs this plane works with. It's a bitmask for the entries in the drmModeRes.crtcs array. So possible_crtcs & (1 << 2) means that it works with res->crtcs[2].

Requirements:

We want to configure each pipeline like

                      ↙ [cursor]?
[connector]+ ← [crtc] ← [primary]
                      ↖ [overlay]*

• +, *, and ? have the same meaning as regex.
• Technically there an an encoder between the connector and crtc, but due to the way encoders work in the DRM API, you may as well pretend they don't exist here.
• We don't actually support multiple connectors for CRTC yet, but I eventually want this.

---

• We want to successfully match and configure as many pipelines like this as possible.
• When we configure a connector with a new CRTC, this is a modeset, and causes the monitor to blank for the user, which is slow and looks bad. Ideally, when mapping all of these pipelines, we should pick one that is as close as possible to the current mapping to prevent unnecessary modesets.
• When a user hotplugs a new monitor and we can't map them all with CRTCs, we should prefer to fail the new monitor and leave existing mappings in place. I think this is just what most people would prefer.

Current implementation

I can definitely think of a few bits of unnecessary bits of complexity with our current solution. It would probably make your life significantly easier to address these before moving on to changing anything else.

There is half-assed and incomplete overlay plane handling in the code. The way I've done it is shit, and will definitely be changed when we get proper support. We should remove any code currently mentioning overlays.

We run the match_obj algorithm on primary planes and cursors, but this is massively overkill and not taking advantage of any of the guarantees the kernel gives us. We should remove any of this dynamically matching shit, and just stash the planes with their CRTCs using a naive matching algorithm at start-up. This basically means nuking the entire realloc_planes function.

The DRM Atomic API allows us to very cheaply test if a mapping should work (TEST_ONLY), which honestly could just let us take a more brute force approach. Unfortunately the legacy API doesn't have this, but I'm not going to let any legacy issues hold any atomic code back. I'd be fine with giving the legacy code a suboptimal solution if it's reduce a bunch of complexity. This is quite a rework, and may not be something you'd want to do right now.

match_obj

The current core of the algorithm found in backend/drm/util.c.
This is a recursive backtracking algorithm trying to solve the above constraints.
This function is too complicated for its own good, which is why I eventually want to take the atomic brute-force approach.

• objs is an array of possible_crtcs that we are trying to satisfy. E.g. you can think of this as an array of connectors.
• res (terrible name; short for resource, not result) is the current mapping and is an index into objs.
  res[0] == 1 means that we've mapped crtc 0 to obj[1]. There is also the special values UNMATCHED, which means there is no mapping, and SKIP, which means that this resource/crtc should not be taken into consideration.
• out is the result, and is the same formal as res.

I don't know if it's really worth going into too much detail about the code itself. If you're somewhat familiar with backtracking algorithms, and keep the above requirements in mind, you should be able to work it out. There are a few small comments peppered in there that explain a few of the steps, but there definitely could be more.