Compose-integrated rendering surface for video and OpenGL

[yaroslavkulinich]

Could Nucleus support a GPU-backed rendering surface for high-performance video or OpenGL content?

"NativeView" provides native-view embedding and a dedicated Compose overlay slot, but the rendered content remains outside the normal Compose composition. What is still missing is an efficient rendering path that allows regular Compose content from the surrounding layout-controls, HUDs, popups, dialogs, and other composables - to overlap the rendered area normally, without moving everything into a special overlay slot or inheriting native-view/"SwingPanel"-style interop restrictions.

Conceptually, this would be similar to a "SurfaceView", but integrated with Compose composition and z-order, while avoiding the overhead of copying every frame through a Compose bitmap.

Is something like this technically feasible within Nucleus, and could it be considered for the roadmap?

[kdroidFilter]

Thanks for the well-framed proposal — you're describing what Flutter calls an external texture (the Texture widget) and Android solves with SurfaceView/SurfaceTexture. Good news: not only is it technically viable in Nucleus — arguably more so than anywhere else in the Compose Desktop ecosystem — it's something we already had in mind: ComposeMediaPlayer, which I also maintain, is planned to adopt exactly this rendering path as its backend.

Why it's feasible here. Nucleus's Tao window backend owns the entire render pipeline — we create the Skia DirectContext ourselves (Metal on macOS, ANGLE/EGL on Windows, GL on Linux) and wrap the swapchain texture via BackendRenderTarget. Importing an app-provided GPU texture into the composition is the same mechanism pointed at a different texture. Stock Compose Desktop can't offer this without Skiko cooperation; we don't have that constraint.

The import paths are the well-trodden ones Chromium and Flutter desktop use:

• macOS — producer renders into an IOSurface (for video: CVPixelBuffer → CVMetalTextureCache, zero-copy), wrapped as an MTLTexture on Skia's device.
• Windows — we render through ANGLE, which supports EGL_ANGLE_d3d_texture_client_buffer: a shared D3D11 texture (e.g. straight from a Media Foundation decoder) becomes a GL texture Skia can sample.
• Linux — dmabuf → EGLImage → GL texture in a shared context (VA-API for hardware video decode).

Because the pixels land inside the Skia scene graph, you get everything NativeView structurally can't give you: true z-ordering (Compose content above and below), scrolling inside lazy lists, clipping, graphicsLayer transforms/alpha, shaders — with no bitmap copies and no overlay-window machinery.

Scope we'd commit to. An ExternalTexture composable plus a per-platform producer handle (IOSurface / shared D3D11 texture / EGLImage), with double/triple buffering and fence-based synchronization. Deliberately not a video player — decoding stays in your hands, same split Flutter chose with its texture registry, and it's proven. For the video use case specifically, ComposeMediaPlayer would sit on top as the first consumer: it keeps owning playback/decoding and migrates its rendering onto this primitive, which also serves as the proof that the layering works for third-party producers.

One boundary worth being explicit about: this is for passive GPU pixels (video, OpenGL, 3D engines, map renderers). All input goes to Compose — the producer receives nothing unless you forward events yourself. So interactive native widgets (WebViews especially) remain NativeView territory; the two features are complementary, not competing.

Known caveats: Tao backend only (no AWT path), and YUV/NV12 video frames need a colorspace conversion step (producer-side or via Skia YUV sampling) — both solvable, just calling them out.

Could you share more about your concrete use case — video playback (which ComposeMediaPlayer would cover once migrated), a GL engine viewport, something else? That would help us prioritize which platform/producer path to prototype first (macOS is where we have the most infrastructure in place today).

[yaroslavkulinich]

Thanks for the detailed reply — this is exactly the primitive we've been missing. To answer your question, we have two concrete use cases, and I think a single ExternalTexture would serve both.

1. 3D map

The map is the central element of our system. We render it with WorldWind Java, a Swing/AWT-based 3D globe framework, embedded into Compose through the Compose–Swing interop — and this is where our biggest UX limitations come from. WorldWind has two rendering surfaces: a heavyweight GLCanvas and a lightweight GLJPanel. We use GLCanvas because it's the only one that gives us acceptable rendering performance. We can overlay Compose content on top of it only through the ComposePanel/SwingPanel wrapping that JetBrains documents for this, which is genuinely ugly — we keep it to a minimum and only use it for transient, temporarily-shown panels. For the central surface of the app, that's a hard ceiling on our UX.

2. Video / stream display

We need to display multiple video streams at once, each in its own panel or window. The pipelines are non-trivial: we use GStreamer for fairly complex processing — recording with data injection, re-streaming, and so on. Today we render through GStreamer's default native playback window. It performs well, but it's a standalone OS window: it can't live as a panel inside our main window, and can't be overlaid with interactive Compose content. We also tried rendering frames through a Compose Bitmap to get a real composable surface, but with several videos on screen simultaneously the CPU/RAM cost is prohibitive, so we can't rely on it.

In both cases

What we're missing is the same thing: a high-performance, GPU-backed composable we can render arbitrary content into and freely overlay with Compose. With that, we can migrate WorldWind's map rendering onto it and give GStreamer a proper in-window video surface — integrated rendering, good performance, and none of the current tradeoffs or workarounds.

For context, our system runs on all three desktop platforms — Windows, Linux (Ubuntu), and macOS.

[kdroidFilter]

Hello,

Yes, I understand your need, and I don't see any problem in implementing it. The new tao backend is capable of achieving this in a much more elegant way, but unfortunately, this feature is not a priority for me. The application for which the nucleus framework was developed does not have such a need, so it is on the roadmap, but I cannot say when it will actually be developed.

However, I can offer you two options: either you can develop the feature yourself—and in that case, assuming it works well and respects the architecture of nucleus, I see no reason to refuse it, quite the contrary—or you can contact me privately to fund the development of the feature.

Best regards,

[kdroidFilter]

It's now available in Nucleus 2.3.0