Proposal: backend-neutral external compositor — zero-copy composition of externally rendered wgpu textures (CAD/scientific viewports)

[canesin]

Problem

GPUI applications that embed a domain renderer — a CAD viewport, a scientific/medical volume viewer, a map engine — currently have no zero-copy path to composite externally rendered GPU content into a GPUI window on the wgpu backends:

• surface()/Window::paint_surface accepts a CVPixelBuffer and is macOS-only.
• Every other image path (ImageSource::Custom, img(), canvas() + paint_image) goes through the sprite atlas as CPU BGRA bytes, which means a GPU→CPU readback per frame for GPU-rendered content. For an interactive 3D viewport (e.g. 512³ volume raycasting at 60 fps) that readback and the extra frame of latency are the difference between viable and not.
• Creating a second wgpu::Surface on the same window is not an option (swapchain exclusivity on Vulkan WSI), and GpuContext/WgpuContext are not exposed publicly, so an embedder can't even render on the same device.

Material PR: #60573

Proposal

A backend-neutral external compositor API in gpui, with an implementation in gpui_wgpu:

• ExternalCompositorRegistry (per window): registers slots described by an ExternalSlotDescriptor (format, alpha mode, size in texels, sample count, graphics-context generation). Handles are generational (ABA-safe). The compositor object itself is stored as an opaque Box<dyn Any> — gpui core carries no callback trait and no wgpu types; each backend defines its own trait and double-boxes it.
• ExternalCompositorElement + Window::paint_external_compositor: inserts a neutral ExternalCompositorPrimitive into the Scene, ordered like any other primitive (same shape as the macOS Surface path). An optional .background(color) paints behind the composited texture (visible while not ready, or on backends without support).
• gpui_wgpu: a WgpuExternalCompositor trait — compose(&mut self, slot, ctx: &mut WgpuCompositorBackendCtx<'_>) -> ExternalComposeOutput with real lifetimes, zero unsafe — is called once per slot per frame at a controlled point (a dedicated encoder submitted before the main pass), and the returned Arc<wgpu::TextureView> (created on the shared device) is drawn by a small RGBA pipeline that follows the existing storage-instance idiom. GPUI keeps ownership of the one queue.submit per frame, so ordering is correct by construction.
• Device-lost is handled via a shared, monotonically increasing context generation (SharedGpuContext), bumped on real device recreation and propagated to every registered compositor across all windows sharing the context; stale slots are skipped and apps re-register.
• macOS/Metal and Windows/DirectX renderers get exhaustive match arms that fall back to the element background (no behavior change otherwise); test/headless keep working with the registry absent.

Status / proof

Working implementation on a fork branch: 0xCarbon/zed#feat/external-compositor (+2186/−33 across 19 files, no new unsafe, cargo test -p gpui --lib 216 green, checks clean on gpui/gpui_wgpu/gpui_linux/gpui_macos/gpui_windows). Includes crates/gpui/examples/external_compositor.rs — a live-updating procedural texture composited zero-copy — validated on Wayland + RADV (RX 7900 XTX), and honors EXTERNAL_COMPOSITOR_EXIT_AFTER for headless smoke runs.

We (0xCarbon) are building CAD/geoscience workstations on GPUI and will maintain this feature on our fork regardless; we'd much rather land it upstream where the whole embedder ecosystem can use it. Draft PR with the full diff to follow — happy to reshape the API to fit your plans for gpui (naming, feature-gating the wgpu surface, splitting the PR, or aligning with any internal design you already have for external/video surfaces on the wgpu backends).

[MSIsunny]

I like this feature, it's exactly what I need!

How do you plan to implement the same functionality on Mac and Windows?

[MSIsunny]

I liked this proposal and tried to build on top of it. With AI assistance, I added experimental macOS and Windows implementations for the external wgpu compositor path in my fork.

Roughly what this adds:

• Keeps the backend-neutral ExternalCompositorRegistry / ExternalCompositorElement model from the original proposal.
• On macOS, creates a wgpu Metal context from GPUI’s existing Metal device, validates that both sides use the same MTLDevice, then samples the underlying Metal texture directly in the Metal renderer. Retina/device-pixel sizing is also handled.
• On Windows, uses a DX12-backed wgpu device/queue, creates a D3D11On12 device from the same DX12 device/queue, wraps the returned ID3D12Resource with CreateWrappedResource, creates a shader resource view, and samples it in the existing DirectX renderer. This path is DX12-only.
• Includes basic handling for context generation, stale slots, resize, device/context recreation, resource lifetime retention, and cached wrapped textures on Windows.

I have tested the demo locally on both macOS and Windows, and it is usable in my current experiments.

This is still experimental code, and there are probably rough edges around broader GPU/driver coverage, more complex multi-slot cases, and long-running stress tests. But if anyone is interested in this direction or has a similar need, feel free to try the branch or use it as a reference:

https://github.com/MSIsunny/zed/tree/feat/external-compositor

check:

cargo run -p gpui --example external_compositor

[canesin]

Thanks to taking it to spin @MSIsunny - that is moreless the direction I was thinking of, choosing wgpu makes it portable and the match arms were filled to allow the composition, if you want to PR against our fork with your macOS/Win improvements I could give it a code review so we keep the PR branch on #60573 as a central point of the SOTA for this proposal.

[MSIsunny]

Thanks! I implemented this mainly because I personally need a way to render 3D content inside GPUI, so this proposal was very close to what I was looking for.

One thing I should be transparent about: most of my implementation was written with Codex assistance, and I do not yet have a deep understanding of GPUI’s lower-level rendering architecture. So I cannot confidently claim that the current code quality or engineering stability is good enough. It works in my local macOS/Windows tests, but I would still treat it as experimental.

That said, I think this experiment at least proves one useful point: using wgpu as the cross-platform external texture compositor path seems feasible. wgpu-hal exposes enough backend-specific details for us to bridge into Metal on macOS and DX12/D3D11On12 on Windows, while still letting the external renderer use a portable wgpu API.

So I agree this is a direction worth exploring. My current branch may not be mature yet, but it could be useful as a reference or starting point.