In-house compositor design discussion

[ddevault]

Note: the approach described in this comment is obsolete

We're going to switch to an in-house compositor instead of using wlc. This issue should serve as a place to brainstorm about the design goals of this compositor. Some of the rationale for using our own compositor is described here: #1027 (comment)

Firstly, some refactoring will need to happen to what we already have:

• Much of sway/* will move into i3wm/* as the i3 implementation. This directory will implement i3 commands, window management, IPC, etc as a compositor plugin
• The wayland client support code will move to client/ and be refactored quite a bit
• Current implementations of protocol extensions will be implemented as compositor plugins. desktop-shell and current sway-specific protocols will be removed in favor of surface-layers.

New subdirectories will be established:

• server/ will contain server utility code, similar to the role our shared client code serves
• targets/ will contain render "target" implementations, such as DRM, Wayland, X11, fbdev, etc
• renderers/ will contain a gles2 renderer, which will render each output into a buffer, and cooperate with targets to offload work onto them when appropriate (i.e. fullscreen-shell)
• sway/ will leverage server/, renderers/, and targets/ to provide the actual sway compositor

Each of these logical areas will provide their own APIs, which can be leveraged by the plugins. These new directories will constitute the core of sway, and features like i3 behavior can be implemented on top of their APIs. Note when I say plugins, I'm presently only talking about it from a design perspective - dynamic linking of external plugins is out of scope for the time being.

Some goals to set from the outset to address current problems or lack of features in wlc:

• Output rotation and modesetting (note: I wish to deprecate setting resolutions that aren't available as modes)
• Address performance issues inhernet in wlc's (immediate mode) design
• Achieve flexibility necessary to cleanly implement things like surface-layers, input-injection, and other features that require a deeper connection with our compositor
• TTY handling
• Rendering views on multiple outputs at once (and dragging views between outputs)

Some discussion points for design:

• Eliminate swayc_t from the non-i3 parts of the codebase, and consider renaming it to i3container_t or something similar?
• Use wl_list and such instead of list_t?
  • wl_list is a linked list, which will be less efficient
  • wl_list has a lot of functions implemented as macros, which is strongly discouraged in sway code
  • We will have to use wl_list et al at least to some degree to expose lists over the wayland protocol, though.
• Isolating i3 functionality to the i3wm directory is not only for organizational reasons, but to possibly in the future provide implementations of other WMs (bspwm, awesome, xmonad, openbox, etc), be it internal or external implementations. Should we design with such external plugins in mind? Should we move sway-i3wm into such a plugin?
• Use multiple processes? We could leave a privledged process running to do things that we currently use caps for, which would improve security. We could also use this process to do TTY handling to isolate TTY handling from any problems that may arise in the compositor (which would make crashes easier to recover from).

Some useful software to reference while writing our compositor: wlc, weston, swc

[Timidger]

Isolating i3 functionality to the i3wm directory is not only for organizational reasons, but to possibly in the future provide implementations of other WMs (bspwm, awesome, xmonad, openbox, etc), be it internal or external implementations. Should we design with such external plugins in mind? Should we move sway-i3wm into such a plugin?

If that's the direction you'd go in, would this attempt to be a replacement for wlc? I agree with the issues you raised about wlc -- We are running into the same issues on Way Cooler. Is this compositor implementation only going to be internal to Sway or do you foresee a situation closer to wlc where it's more general so any WM can be implemented?

Personally, I think that future of Wayland window managers hinges on good, general compositor libraries being the backbone for most non-"mainstream" window managers (e.g, everybody that isn't Gnome or KDE). Otherwise, it's too painful to the end user in terms of what features are actually implemented for a given window manager (e.g. good xwayland support, remote desktop, other niceties like redshift, etc.).

I'm going to look into libweston as well, since that wasn't an option when I started Way Cooler. Is this something that this project is going to turn into, or is this first and foremost going to be for Sway?

[ddevault]

I think designing sway in a way that permits it to be a replacement for wlc is a good design choice, though that's not necessarily the goal. I do foresee a time when more window managers could be built on top of sway if that design works out. However, since replacing wlc is a consequence of this design rather than the goal, I would be willing to sacrifice design details that support wlc-like use-cases in favor of design decisions that benefit the direct goals of the project. To that end I foresee other window managers being implemented as sway plugins rather than as standalone compositors that leverage sway like sway leverages wlc, for example.

However, within this model and if plugins work out, I see no reason why pretty much every WM of today's X ecosystem couldn't be implemented as a sway plugin. Similar to how some people thought weston was going to be.

[ddevault]

To clarify, I think isolating the i3 implementation is a good design and I think that building a (plugin API)-like thing is a good design, but I'm not exactly aiming for supporting that for generalized use cases at the moment.

[Timidger]

Ok, thanks for the response. I hope that this project achieves its goals.

[FSMaxB]

I really like your last point. I think multiple processes and especially keeping privileged processes small and auditable is a good way to go forward.

[robotanarchy]

I bet you have put some deep thought on this, but let me ask the obvious question: Why you want to restructure the whole repository, and rename everything?
It seems to me, an easier upgrade path would be the following:

• Do not rename the actual compositor from sway to "i3wm" (it is a completely different implementation after all!) or change the repository structure
• Create a new repository, that will have the replacement for wlc. Call it swaylc (sway library for compositors) or whatever :p
• Begin replacing parts of wlc in sway with the new library, until everything is replaced.

The thing is, that sway is already used as day-to-day wm by some of us, including me, and I'd rather have a smooth transition than having stopped development and waiting for an unknown time period until wlc is replaced. Also the renaming will probably confuse people.

Plus I find it awkward to have one repository, that will now have the code of a wlc-replacement, "i3wm for wayland", and possibly "openbox for wayland", tons of other tiling/simplistic wms, and to call all that together "sway" from now on.

Aside from that, I think it is a good thing to replace wlc, if we can have a better/faster/stronger sway in the long run. And privilege separation is something I definitely would like to see.

[ddevault]

I bet you have put some deep thought on this, but let me ask the obvious question: Why you want to restructure the whole repository, and rename everything?

I don't, really. I want to maintain the status quo. Right now the actual compositor of sway is being kept seperate, in wlc. If we write our own, I want to maintain that separation. We currently have some components that are compositor-like (such as protocol extensions, rendering window borders, etc). The bulk of the code is more analagous to a WM than to a compositor, and what little compository code we have is intermixed with it. If we're going to write an in-house compositor, I would prefer to move the handful of things that we currently have that belong there into it.

Do not rename the actual compositor from sway to "i3wm" (it is a completely different implementation after all!) or change the repository structure

The compositor renders windows to the screen. Strictly speaking wlc is the compositor. I'm not renaming sway to i3wm, I would like to clearly separate i3wm behavior into a seperate logical unit from the compositor, which is just a good design decision. It makes sense to call that module i3wm because that's what it does, implement i3wm. It is an implementation of i3wm. That is, any behaviors in i3wm that you can't reproduce in sway are considered bugs in sway (always have been).

The thing is, that sway is already used as day-to-day wm by some of us, including me, and I'd rather have a smooth transition than having stopped development and waiting for an unknown time period until wlc is replaced.

I use sway every day, too, I'm using it right now :) trust me, the transition will be careful, slow, and smooth. I don't intend to ship any release that causes major breakage. I don't want to halt development, but I also don't want to ship half-assed versions of the code. Though I want to avoid breaking changes, iif we have to make them, the time to do so is while doing 0.x releases. We'll be thankful for that attitude 10 years from now. I also don't intend to halt all development - only some features will be postponed. Most of the work that's already being done in Sway can continue, only the work that might have been being done in wlc will be stalled.

Also the renaming will probably confuse people.

No renaming is happening externally, this project is still called "Sway". How end-users use it isn't changing either. We're just talking about the internal organization of the codebase.

Plus I find it awkward to have one repository, that will now have the code of a wlc-replacement, "i3wm for wayland", and possibly "openbox for wayland", tons of other tiling/simplistic wms, and to call all that together "sway" from now on.

If we get to the point that we are implementing more WMs here (which is a dubious prospect in the first place), they won't all live in one repository and we'll find a way for it to make sense.

[robotanarchy]

Thanks for the clarification, now it makes sense to me 👍

[ddevault]

Something else to mention: wlc abstracts some of wayland away from the user, as if it were designed to also be able to support some other display protocol. We should not do this with our thing, it will save us some headache and make it easier to do some things.

[ddevault]

After some of the design discussion with way-cooler (even though way-cooler has decided to go another direction), I'm considering doing this out-of-tree and making it a generic library like wlc/swc/libweston, but with these design goals in mind. Thoughts?

[Timidger]

I think having it out of tree would make the most sense -- it would allow others to easily bind with it better, and it will be easier to separate the concerns of the two layers if they live in separate projects. This also means issues with Sway won't get mixed in with issues about the compositor library.

Also--Way Cooler hasn't decided to go in another direction. Fireplace has decided to work on their own compositor framework Smithay. I'd still like to work with you on a non-Rust compositor library, as I think it will be easier to support in the future (there are more people who know C, there will more than likely be more compositors using it, I expect most of the contributors to wlc to contribute to this project, etc.). If it turns out to be better for my needs I might switch to Smithay in the future but for now I want to try to work on a more general solution.

[ddevault]

Oh, I didn't realize Fireplace was a separate project.

[Timidger]

Yes it's relatively new, despite the high version number. It's similar to orbment in that it's modular, but with Rust instead of C. Since users are expected to edit Rust, they benefit more from having a more ergonomic compositor library.

[robotanarchy]

Something I just thought about:
Please make the in-house compositor code (be it a separate library or not) fully POSIX (and therefore musl libc) compatible. This means, do not use any glibc specific features, such as qsort_r. wlc did this right from the start afaik, and it was later patched in sway. I think the best way would be, to specify a POSIX_SOURCE define when compiling sway.

[ddevault]

Of course.

[ddevault]

I hate GNUisms, they were added by mistake and anywhere you find them is a bug.

[ddevault]

Here are my current thoughts. I want to better support @Timidger and others with similar use-cases, so instead of doing this in-tree we'll be making a new library that will fulfill a similar role as wlc, but with a different design. We'll call this library "wlroots" (or "wlr"). The design I have in mind is this:

Several independent components (compositor, wayland server, xwayland, xdg-shell, etc) that the user can compose freely. No functionality will be mandatory, and the user can implement any functionality themselves (no necessary information like libinput pointers will be hidden). The user will have ownership over lots of things that wlc currently has ownership over, but stewardship of these resources will be assisted by wlroots. Notably, the user will own the wayland registry and will probably own the event loop.

Assuming we like this design, I'm ready to get started in the next few weeks.

[ghost]

@SirCmpwn Thank you. I'd be interested in joining this project, or shadowing to some extent if possible.

[ddevault]

Keep an eye on this thread and contribute when the code starts happening.

[ghost]

Thanks. Will do.

[vially]

Will this support EGL streams/proprietary nVidia drivers? You said that's unlikely but does that mean it's unlikely to have it in the beginning or ever?

[ddevault]

Ever.

[ddevault]

I have started working on wlroots. The repository is available here: https://github.com/SirCmpwn/wlroots

I've started putting together the wayland backend (for running the compositor nested in another wayland session), since this should be the simplest backend to make.

My current plan is:

1. Flesh out the wayland backend
2. Establish the wayland server
3. Build the renderer

We'll go from there.

[ejiektpobehuk]

Since libweston is a reference implementation for Wayland what is the reason to write custom solution?

offtopic:
My team at university is about to write our own wayland based WM for mobile device (nothing fancy just one term project) and we have the same questions.
We've planed to use weston as a base due to its raspberry support but since vc4 it doesn't really mater.

[ddevault]

libweston inherits all of the problems of weston, of which there are many. It also is run with a different philosophy than my approach and the changes I would want to make would be difficult to upstream.

[ddevault]

wlroots is well underway - closing this.