Add ability to generate python raytracing renderering#8
Open
haixuanTao wants to merge 7 commits into
Open
Conversation
The viewer could only present to its window; there was no way to get rendered pixels back to Python for headless frame/video export (as mujoco.Renderer.render() or a Genesis camera.render() would give). kiss3d's Window already exposes snap_image(), so this just wires it through: - nexus_viewer3d::NexusViewer::snap_rgb() -> (w, h, RGB bytes) - Python NexusViewer.render() -> (H, W, 3) uint8 NumPy array Call render_frame() to draw, then render() to read the frame back.
Adds NexusViewer::new_with_size and exposes width/height (default 1200x900) on the Python constructor, so headless capture and path tracing can run at arbitrary resolutions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
render_frame always drew the egui main panel, which then appeared in snap_rgb captures. Expose a toggle (default on) so headless recording gets clean frames. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haixuanTao
force-pushed
the
feat/python-rt-render
branch
from
f22e180 to
34da17c
Compare
sebcrozet
reviewed
Jul 8, 2026
sebcrozet
left a comment
sebcrozet
left a comment
Member
There was a problem hiding this comment.
Thank you! That looks good but it seems like this includes some of the code from other PRs. So let’s merge the others first.
haixuanTao
added a commit
to haixuanTao/nexus
that referenced
this pull request
Jul 8, 2026
…ewer, pipelined readback Makes the nexus3d Python bindings usable for off-screen video capture: - NexusViewer(width, height): configurable resolution; set_draw_ui(False) keeps the egui panel out of captured frames. - NexusViewer.render() -> (H, W, 3) uint8 numpy frame export, like mujoco.Renderer.render(). - raytrace_frame() + set_raytracer_samples_per_frame/max_bounces/denoise + raytracer_backend(): kiss3d's GPU path tracer from Python, with progressive sample accumulation. sync takes the CPU-readback path while the tracer is active (the zero-readback kernel only updates the rasterizer's instance buffers, which the tracer's BVH never reads). - NexusViewer(..., headless=True): no OS window, no swapchain — frames render into an off-screen texture, never throttled by display vsync, works without a display server. - set_vsync(enabled)/vsync(): uncap a windowed viewer instead. - render_async()/render_flush(): pipelined capture — returns the previous frame while the current frame's GPU->CPU copy runs in the background. A windowed capture loop was vsync-locked at ~30 fps (blocking readback + Fifo swapchain = ~2 vblanks per iteration); headless + pipelined readback reaches 92 gen-fps at 640x480 (GPU physics) on an RTX 5080. Requires the kiss3d headless-capture branch (dimforge/kiss3d#398). Supersedes dimforge#7 and dimforge#8 (included here).
Adds NexusViewer.raytrace_frame() (kiss3d 0.45 GPU path tracer with cross-frame sample accumulation), set_raytracer_samples_per_frame/ max_bounces/denoise, and snap_rgb() returning the framebuffer as an (H, W, 3) uint8 numpy array. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The zero-readback sync kernel writes body poses straight into kiss3d's GPU instance buffers, but the ray tracer rebuilds its acceleration structure from the CPU-side instance data — so path-traced frames froze at the initial scene. Route sync through the readback path whenever a RayTracer exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NexusViewer.raytracer_backend() returns "hardware" (RT-core ray queries) or "software" (compute-shader BVH fallback) so users can verify which backend kiss3d selected on their GPU. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
haixuanTao
force-pushed
the
feat/python-rt-render
branch
from
6aa6051 to
1b45312
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add ability to generate raytraced framed