Zellij is expanding into a desktop environment - let's build it together

[imsnif]

Zellij was never meant to only live in the terminal, and the last year has shown its progression: first into the browser with the web-client, and now into the desktop with the built-in terminal. We're now building it into something greater: a terminal-first desktop environment. Allowing those who love the terminal to invert the GUI-imposed order. Live in the terminal, run GUI apps such as browsers and photo editors inside your TUI as separate panes and tabs.

The basic functionality of Zellij will not change: it will always be a terminal multiplexer and allow current users to keep their workflows. Additionally, they'll be able to integrate their GUI needs into them if they so wish.

This thread is the start of this work. Nothing is implemented yet, and I'd like to invite those who wish to participate to get in on the ground floor. So, what's next?

Guiding principles

• Opt-in. Terminal-only users should see no difference: no new required dependencies, no behavior changes.
• GUI apps are panes. They should be laid out, resized, stacked, floated, moved between tabs and bound to keys like any other pane, not as a separate window manager.
• Linux/Wayland first. It's the path of least resistance. Other platforms are very much welcome, but will probably require more work.
• Learn before we build. The first phase is research and throwaway prototypes. Architecture decisions come after we've seen what works.

Directions of research

1. Zellij as a Wayland compositor

The most likely foundation: Zellij acts as a (nested) Wayland compositor, GUI apps connect to it, and their surfaces get composited into pane rectangles. Smithay is the Rust candidate.

• What's the minimal compositor that can host a real app (a browser, Inkscape) in a pane?
• How does it fit into Zellij's server/client architecture? Does the server own the compositor, and clients only render?
• X11 apps via XWayland: what does that cost us (see xwayland-satellite)?

2. Rendering, one surface, three targets

Zellij runs in three places now, and each one needs a way to show a GUI surface:

• Built-in terminal: we control the whole window, so surfaces can be drawn natively, ideally zero-copy (dmabuf/GPU).
• Other terminal emulators: kitty graphics protocol/sixel, and a character-cell fallback. What frame rates are realistic, and how do we only send damaged regions?
• Web client: streaming surfaces to the browser (video encoding with WebCodecs/WebRTC, or tiled image updates). What's the latency, bandwidth and CPU cost?

3. Window management semantics

How do GUI concepts map onto Zellij's?

• Toplevel window <-> pane. Multiple windows from one app ↔ tab, or stacked panes?
• Popups, menus, tooltips and dialogs: built in ones, or something else?
• Sizing: apps think in pixels, panes think in cells. How do we handle resizing, scale and HiDPI?
• Layouts and session resurrection: can a KDL layout include firefox the same way it includes htop?

4. Input and integration

• Keybindings: how do Zellij's modes coexist with apps that want every shortcut?
• Focus, mouse, scrolling, IME and text input
• Clipboard bridging between terminal panes and GUI apps; drag & drop

5. Detach, reattach and remote sessions

Zellij sessions survive detaching. Could GUI apps too? Detach from your desktop, reattach from another machine (or a browser via the web client), and your browser tabs are still there. This might be one of the most cool features, and it overlaps heavily with direction 2.

6. Plugin API

What should plugins be able to do with GUI panes? Launch, arrange, observe, screenshot, inject input?

7. Beyond Linux

What would this look like on macOS or Windows, if anything?

Prototypes wanted

This is where you can have the most impact right now. Each of these is a self-contained spike that answers a real question:

1. Hello, pane: a minimal Wayland compositor that shows a single GUI app inside a Zellij pane, through any rendering path.
2. Terminal graphics benchmark: render a live surface into a pane through the kitty graphics protocol (and/or sixel). Measure frame rate, bandwidth and CPU.
3. Web client streaming: get a GUI surface into the web client and measure latency.
4. Input model: a write-up or mockup of how keybindings, focus and mouse should work between Zellij and GUI apps. No code required.
5. Detach/reattach: keep a GUI app alive across a Zellij detach and reattach.

Ground rules for prototypes:

• Throwaway is fine, even preferred. Hack in a fork or branch. Don't polish.
• Report back here with a short video or GIF, what worked, what broke, and what you learned. Share source code. Failures are valuable results.
• Big findings get their own discussion in this category.

How to take part

• Reply with what you want to dig into (a research direction or one of the prototypes above), so people can team up instead of duplicating work. I'll keep this post updated with who's working on what.
• Bring expertise: Wayland/compositors, graphics/GPU, video streaming, Rust, input/accessibility, UX.
• For general chat, find us on Discord.

Let's do this!

[divens]

@imsnif Very interesting. I'll definitely have a go at prototyping some of it on Windows.

[moosingin3space]

@imsnif I am extremely intrigued by this idea. Zellij has excellent UX already, and the sheer customization capability of Zellij combined with the ability to act as a desktop environment sounds simply amazing, especially for the coding agent era. Wanted to share a tool that might be helpful for the Linux prototype, my tool waypuppet, which I've used to help me build GUI applications for Linux.

A few reasons I find it useful:

1. It's a good feedback tool for a coding agent, similarly to how the Playwright CLI makes it easy to test web applications.
2. It demonstrated a way to run a Wayland compositor while there's already one running. I was able to develop a few applications using Waypuppet as a compositor that did not interfere with my Gnome compositor. On my Bluefin Linux machines, running Gnome, it uses the GPU, and in a GitHub Actions runner, it uses llvmpipe.

I have a few ideas:

• Web client streaming: should probably use a similar technique to MoQ, as WebTransport is now well supported.
• Detach/reattach: this will probably require the Wayland server socket to be held by the zellij server -- and the app can be effectively halted by simply not producing frame callbacks.

Unfortunately, I'm not as knowledgeable on many of the non-networking/non-systems concepts and will need to read up before I can meaningfully contribute. Happy to help once I've had a chance to read more, I would absolutely use this as a daily driver, especially on my laptop.