Add NexusViewer.render() to export frames off-screen#7
Conversation
The viewer could only present to its window; there was no way to get rendered pixels back to Python for headless frame/video export (as mujoco.Renderer.render() or a Genesis camera.render() would give). kiss3d's Window already exposes snap_image(), so this just wires it through: - nexus_viewer3d::NexusViewer::snap_rgb() -> (w, h, RGB bytes) - Python NexusViewer.render() -> (H, W, 3) uint8 NumPy array Call render_frame() to draw, then render() to read the frame back.
|
Heads-up: render() is correctness-first — per-frame readback is slow (a kiss3d-level concern) Profiling this on an RTX 5080 (release build, 1200×900, single-cube scene), render() works but is throughput-limited. Per-frame averages: ┌───────────────────────────────────────┬──────────┐ So the render/physics are fast (~3 ms, consistent with the viewer's "GPU passes" stat); the cost is entirely the GPU→CPU framebuffer readback. Notably, release ≈ debug (2.68 vs 2.36 fps overall) — i.e. it's a synchronization/latency cost, not CPU compute. Root cause is in kiss3d's read_pixels (window/wgpu_canvas.rs), which snap_image() calls: every invocation
That's ideal for the occasional screenshot it was designed for, but expensive at video frame rates. This PR is still worth landing as-is — it makes headless frame export possible, which it wasn't before, and it's fine for offline/short-clip capture. Making it real-time is a separate kiss3d optimization (out of scope here): reuse a persistent staging buffer, pipeline the readback (map frame N‑1 while the GPU renders N, dropping wait_indefinitely), and move/skip the color conversion. Happy to open that as a follow-up on kiss3d if useful. |
Adds NexusViewer::new_with_size and exposes width/height (default 1200x900) on the Python constructor, so headless capture and path tracing can run at arbitrary resolutions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
render_frame always drew the egui main panel, which then appeared in snap_rgb captures. Expose a toggle (default on) so headless recording gets clean frames. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sebcrozet
left a comment
sebcrozet
left a comment
There was a problem hiding this comment.
Just a small nitpick but that looks good, thank you!
| /// Call once per frame after [`render_frame`][Self::render_frame] to export | ||
| /// frames off-screen (e.g. to encode a video) instead of only presenting to | ||
| /// the window. | ||
| fn render<'py>(&mut self, py: Python<'py>) -> PyResult<Bound<'py, PyArray3<u8>>> { |
There was a problem hiding this comment.
We can keep the name of what’s it’s binding.
| fn render<'py>(&mut self, py: Python<'py>) -> PyResult<Bound<'py, PyArray3<u8>>> { | |
| fn snap_rgb<'py>(&mut self, py: Python<'py>) -> PyResult<Bound<'py, PyArray3<u8>>> { |
…ewer, pipelined readback Makes the nexus3d Python bindings usable for off-screen video capture: - NexusViewer(width, height): configurable resolution; set_draw_ui(False) keeps the egui panel out of captured frames. - NexusViewer.render() -> (H, W, 3) uint8 numpy frame export, like mujoco.Renderer.render(). - raytrace_frame() + set_raytracer_samples_per_frame/max_bounces/denoise + raytracer_backend(): kiss3d's GPU path tracer from Python, with progressive sample accumulation. sync takes the CPU-readback path while the tracer is active (the zero-readback kernel only updates the rasterizer's instance buffers, which the tracer's BVH never reads). - NexusViewer(..., headless=True): no OS window, no swapchain — frames render into an off-screen texture, never throttled by display vsync, works without a display server. - set_vsync(enabled)/vsync(): uncap a windowed viewer instead. - render_async()/render_flush(): pipelined capture — returns the previous frame while the current frame's GPU->CPU copy runs in the background. A windowed capture loop was vsync-locked at ~30 fps (blocking readback + Fifo swapchain = ~2 vblanks per iteration); headless + pipelined readback reaches 92 gen-fps at 640x480 (GPU physics) on an RTX 5080. Requires the kiss3d headless-capture branch (dimforge/kiss3d#398). Supersedes dimforge#7 and dimforge#8 (included here).
2 participants
The viewer could only present to its window; there was no way to get rendered pixels back to Python for headless frame/video export (as mujoco.Renderer.render() or a Genesis camera.render() would give).
kiss3d's Window already exposes snap_image(), so this just wires it through:
Call render_frame() to draw, then render() to read the frame back.