Proposal: NVIDIA Instant-NGP on DisplayXR — NeRF scenes in glasses-free 3D in its own window

[joehillthunder]

Summary

NVIDIA's Instant-NGP shows NeRF scenes in glasses-free 3D inside its own window on DisplayXR, using the texture target. Instant-NGP's own renderer draws two off-axis eyes, its own camera drives the 3D, and its UI stays on top and usable. It runs windowed beside other 3D apps, and F11 toggles full panel. The VR/AR headset path is unchanged.

How it works

• Eyes: two off-axis eyes are rendered as two m_views, the same multi-view path as Instant-NGP's VR mode. Each eye's pose and projection come from the runtime, with the projection pinned to the runtime's eye tangents. The eyes render at the 3D window's native pixels. While 3D runs, dynamic resolution and training pause; both resume on Exit 3D.
• Session: the app-neutral host core (dxrh_*), the same one as the 3D Slicer, MotionBuilder and Nuke adapters, runs the OpenXR session with XR_DXR_win32_window_binding v9 on Instant-NGP's own window.
• Draw-back: the woven texture is blitted into the window through WGL_NV_DX_interop2, one texel per pixel, before the UI draws.
• Depth: content depth comes from the NeRF's own occupancy grid inside the eye frustum. That includes low-density haze, which never reaches a depth buffer but still pops out in stereo. The near end is clamped on every frame on the nearest occupied cell in either eye, with a 3 % margin. The glass sits a quarter of the way into the content, and the eye separation is fitted to the comfort budget.
• Controls: Start 3D / Exit 3D, "3D active", Depth −/+/Reset and a slider in the Rendering panel; F11 for full panel. Mouse orbit, pan and zoom drive the 3D. No gamepad is needed.
• Coupling: one file touches Instant-NGP internals; the main sources only get small compile-time hooks.

Measurements

Installed runtime 2.57.1, Leia SR plug-in 2.19.1, RTX 5070 Ti, the NeRF fox scene.

Check	Result	Conditions
3D starts in Instant-NGP's own window, Leia SR plug-in from the installed runtime	Pass	no fallback, no dev runtime
Framing (subject fully in view at a middle size)	Pass	Start 3D zooms out 1.4×; Exit 3D restores the view
Depth at 100 %	Pass	front ≤ −0.13 % / behind ≤ +0.25 % of panel width at the default view, three 25° orbits and a fast orbit (budget −0.14 / +0.42)
F11 full panel and back	Pass	3840×2400 eyes, ~8.5 fps
Beside another DisplayXR app (Model Viewer demo)	Pass	each app weaves in its own window
Seated check at the tested display (Leia SR)	Pass	2880×1800 eyes, ~10 fps with head motion

Open points

• Speed: about 10 fps at native resolution with head motion. Rendering at the runtime's requested eye size is about 4× faster but softer.
• Comfort budget is a constant until display info carries it (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
• Licence: Instant-NGP is under NVIDIA's non-commercial source licence, so this integration is a modified copy for non-commercial use. The DisplayXR files are Apache-2.0. Upstreaming would need NVIDIA's interest; for now it stays a local integration.

Part of the cross-app index in #2442; same host core as #2352 (3D Slicer) and #2433 (Nuke).