Proposal: NVIDIA Isaac Sim on DisplayXR — glasses-free 3D next to VR / AR

[joehillthunder]

Summary

NVIDIA Isaac Sim 6.2 runs on DisplayXR with glasses-free 3D as an XR output beside its VR and AR modes. It uses the existing displayxr.omniverse Kit extension (#2331), unmodified Isaac Sim, and an Isaac "experience" app titled Isaac Sim XR - VR / AR / 3D Display. Robots and scenes show in 3D in a window docked over Isaac Sim's viewport. Play / Pause runs the simulation in 3D, and plain mouse and keyboard move the camera.

What it does

• 3D Display as an XR mode. isaacsim.exp.base.xr.3d_display.kit is Isaac Sim's own XR experience plus displayxr.omniverse. It is loaded from outside the Isaac Sim install; nothing in Isaac Sim is changed.
• Z-up stages. Isaac Sim stages are Z-up. In a Z-up stage, Omniverse XR applies the space origin and then turns the result +90° about X on the stage side, (x, y, z) → (x, −z, y), so the 3D view looked at the floor. The extension appends the inverse turn. This was measured with get_physical_to_virtual_world_transform: the nominal eye now lands on the app camera at every pitch and yaw probed. Y-up stages (USD Composer) are unchanged.
• Simple controls in 3D. The app camera stays the single source of truth. While the cursor is over the 3D area:

Input	Action
Left drag	Orbit
Right or middle drag	Pan
Wheel	Zoom
Arrow keys	Move forward / back / left / right
Page Up / Page Down	Move up / down
Space	Back to the start view

Plain W A S D Q E stay with the app's own tool shortcuts. No gamepad is needed. The 3D window over the viewport takes the mouse, so the extension reads it through an in-process low-level hook. The hook never swallows input, and it only acts over the 3D area.

• Simulation. Play / Pause in Isaac Sim run the scene in 3D. The demo (a UR10 arm) moves through its own joint drives.

Measurements

Runtime 2.57.1, Leia SR plug-in 2.19.1, RTX 5070 Ti, Isaac Sim 6.2.0 (Kit 110.4, Omniverse XR 109.1).

Check	Result	Conditions
3D starts, Leia SR plug-in from the installed runtime	Pass	DisplayXR-LeiaSR.dll from Program Files, no fallback
Framing (rule zero)	Pass	UR10 fully in view at a middle size, camera 20° down (sim-display atlas)
Depth at stereo strength 0.2	Pass	front −0.074 % / behind +0.368 % of view width (budget −0.14 / +0.42)
Depth at stereo strength 0.3 (seated)	Over budget	behind +0.65 % (p95); more depth preferred in a seated check, so the demo opts into 0.45
Mouse and keyboard	Pass	seated check, controls above
Play / Pause	Pass	seated check
Clean exit	Pass	window close, no errors in the Kit log

What a shared host library would need

• An up-axis-aware space origin, or a runtime-side "pin the nominal eye to this camera pose" call, so each Kit app doesn't rediscover the stage-side turn.
• An input path for docked 3D windows: either forward mouse and keyboard to the host window (DXR_HOSTED_INPUT-style), or a documented click-through mode, so adapters don't need hooks.
• Display info with the comfort budget (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388), so the stereo strength can be fitted to the budget instead of picked by hand.

Open items

• Fit stereo strength to the budget automatically (content-fitted depth) instead of a per-scene value.
• Texture mode (3D drawn in Isaac Sim's own viewport) once the host library lands; today it is the docked-window path.
• The extension changes are on a branch, pending review by the extension's maintainer.