Proposal: DisplayXR for NVIDIA Omniverse (Kit extension, Leia SR tested)

[joehillthunder]

Proposal: DisplayXR for NVIDIA Omniverse

I've built a Kit extension, displayxr.omniverse, that runs Omniverse Kit apps in glasses-free 3D on DisplayXR displays. It works with USD Composer, USD Explorer, Isaac Sim, and custom Kit apps. It's tested end to end on a Leia SR laptop. I'd like the maintainers to review it, and to consider adopting it as a DisplayXR integration, for example as DisplayXR/displayxr-omniverse, alongside the Unity and Unreal plugins.

Code: joehillthunder/displayxr-omniverse (private for now). @dfattal is invited as a collaborator. I can add anyone else on the maintainers team.

What it does

Omniverse already speaks OpenXR (omni.kit.xr.system.openxr, D3D12), so there's no renderer plug-in. The extension adds Window ▸ DisplayXR. One click starts 3D, and the panel controls which scene distance lands on the glass.

• Runtime selection. Points Omniverse at DisplayXR_win64.json through Omniverse's custom-runtime setting, so the system OpenXR runtime is left alone. Turns on the vr profile with headset-only behaviour off: floor snapping, height matching, controller models.
• Zero-parallax placement. Omniverse arrives as a hosted legacy app, so it gets the runtime's converged camera rig rather than a 1:1 panel window. The extension calls xrLocateViews on Omniverse's own session (ctypes, via the handles omni.kit.xr.system.openxr exposes) to measure the rig's convergence and FOV. Then it sets Omniverse's XR user scale: scale = convergence / focus. On this panel the rig measures 2.03 m and 54° × 35°, which matches the documented legacy defaults (0.5 D, about 36° vertical).
• Camera pinning with pitch. Omniverse's own camera teleport keeps the view level with the horizon, which is wrong for a desk display looking down at a model. The extension sets the XR space origin to translate(-k * nominal eye) * camera, which maps the nominal eye exactly onto the viewport camera (verified numerically). Tracked eyes then move around it.
• Viewport docking. The runtime-owned 3D window is made borderless, owned by the Kit main window, and kept on the viewport's on-screen rectangle; it follows moves and resizes. The weave follows the canvas (weave: target=2610x1137 … canvas=(0,0 2610x1137)). Full screen is one click; it posts F11 to the runtime window, so the runtime's own toggle stays the single source of truth.
• Comfort. Shows a comfortable depth range from a calibrated parallax budget: 0.14 % of view width in front of the glass, 0.42 % behind, tuned by eye on the 34.4 × 21.5 cm panel. A stereoStrength setting seeds DXR_LEGACY_CAMERA_RIG's ipdFactor. It's verified to halve the rendered IPD, and it reapplies on each start, because Omniverse creates a new instance each time.
• Clean exit. Esc in the 3D window, or stopping from Omniverse's own XR panel, restores every setting the extension changed.

Tested

Kit 110.4 USD Composer (from kit-app-template) with Omniverse XR 109.1, on DisplayXR 2.47.0 and then 2.53.0. Leia SR plug-in 2.16.3, then 2.18.0; SR runtime 1.38. RTX 5070 Ti, Leia SR laptop panel, 3840 × 2400.

• The session reaches FOCUSED; the Leia D3D11 weaver weaves through the d3d12 → d3d11 reroute. The app runs at about 46 Hz with XR on.
• Eye captures (schedule_capture_display_frame) confirm the placement:
  • the target sits at 4 px disparity of 1920 at the chosen distance;
  • a shorter distance puts it behind the glass (uncrossed);
  • "Frame selection" fills the view at near-zero parallax.
• Docking, full screen and back, the move/resize follow, Esc, and restart all work.
• 10 Kit-free unit tests cover the framing maths, status parsing, and the convergence solver, using real view samples from this panel.

Runtime-side findings (for discussion)

1. Multi-app workspace doesn't render clients: filed as Workspace on v2.53.0 + Shell 2.4.2: clients get slots and commit frames, but none are rendered (client_renders=0, client_skips≈1200/10s); only the Shell splash is woven #2328. With the Shell running, I'd hoped to show Omniverse next to other 3D apps.
2. /input/pinch_ext/pose is rejected even though XR_EXT_hand_interaction is enabled. Omniverse enables the extension ([XR] Enabled Extension: XR_EXT_hand_interaction) and suggests pinch_ext paths on the controller profiles. DisplayXR returns XR_ERROR_PATH_UNSUPPORTED for every profile (for example khr/simple_controller), so every suggested binding fails. That's harmless on a desk display, but per the extension spec those paths should be accepted.
3. Closing the hosted window returns XR_ERROR_RUNTIME_FAILURE to Omniverse's frame loop (runSyncSystem) instead of walking the session to STOPPING/EXITING. Omniverse recovers, but logs the error.
4. Legacy compromise resolution. Omniverse renders 1920 × 2400 per view (0.5 × 1.0), while SR recommends 1920 × 1200, so it renders twice the pixels the weaver needs.
5. No window binding for engines that own their session. Kit builds its own xrCreateSession/xrLocateViews, so it can't chain XR_DXR_win32_window_binding or XR_DXR_view_rig. The docking above is the workaround. A runtime-supported way to attach a hosted window to a parent HWND or target rect would make the extension much simpler and also cover Blender and Substance Modeler. That could be an environment variable or app profile, or Design: windowless target-rect binding (decouple position/metrics channel from HWND) #697's windowless target-rect binding. An alternative is an OpenXR API layer that chains the binding and a display rig into the app's calls; I'd be happy to prototype that if it's a direction you'd take.

Asks

• A review of the approach and code.
• Interest in hosting it under the DisplayXR org, and in a displayxr.org listing for Omniverse.
• Thoughts on findings 2–5, especially 5.

[dfattal]

Thanks, this is a well-documented proposal, and the verification against eye captures is exactly the evidence we look for.

Hosting: yes in principle. We'll host it as DisplayXR/displayxr-omniverse under an experimental label, with you as maintainer, once it has a license, vendor-neutral wording, and the demo's inbox execution removed or made opt-in. Details are in the code review.

Findings 2 and 3 look like real conformance bugs. Please file them as issues so they get tracked.

Finding 5 is the most interesting part. An attach mechanism for engines that own their session would cover Omniverse, Blender, Substance Modeler and others. It overlaps with #697 (windowless target-rect binding), so let's settle the direction there before building more workarounds. An API-layer prototype would be a welcome input to that decision.

Finding 4 (legacy compromise resolution) is noted; it ties into the same question.

A displayxr.org listing can follow once the repo is in the org and has a tagged release.

[joehillthunder]

Update: Omniverse now weaves in its own viewport, next to other 3D apps

@dfattal, a progress update on this proposal. The code is in joehillthunder/displayxr-omniverse PR #1 (private; you have access).

Done since the first post

• Windowed and concurrent weaving. On the hosted-window embed (displayxr-runtime#2332), the extension publishes Kit's viewport rect as DXR_HostedViewRect, and the 3D stays exactly on the viewport.
  • Tested in USD Composer: opens on the viewport, follows resize, F11 and back, and stop/start.
  • Tested weaving at the same time as Substance Modeler, 3D Slicer and ParaView, each in its own viewer, on one Leia SR panel.
  • It never starts full-panel.
  • On released runtimes it falls back to docking the window itself.
• Live depth. A stereo-strength slider in the panel drives DXR_HostedDepth (d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350). It stays in sync with the shared depth slider (DXR_HostedDepthRel), and a convergence change is absorbed so the chosen screen plane stays on the glass.
• Legible 2D next to the 3D. The switchable lens covers the whole panel while 3D is on, so the panel UI was redesigned for that: large near-white text on dark surfaces, no grey-on-grey, and "3D active" status wording.
• USD in from other apps. 3D Slicer's USD export and ParaView's live USD (reloaded within about 0.5 s of each write) are verified in Composer next to the native scene.

New findings for discussion

1. True-geometry rig vs Kit. Blender, Slicer and ParaView converge on a per-start 1:1 "panel" rig (DXR_LEGACY_CAMERA_RIG: convergence 0.6 m, vFOV from the view's physical height). Kit 110 / Omniverse XR 109 renders it black, because for a viewport docked above the nominal eye the runtime's window-relative frustum lies wholly below the eye axis (e.g. up −0.8°, down −10.4°). A runtime option to report a view-centred frustum, rotating the view pose to match, would let Kit use the same rig as the other apps. Should I file it?
2. Teardown order. Clearing DXR_HostedViewRect before the session has ended makes the embed refit mid-teardown ("scanout swapchain is not following the window"). The extension now clears it after the window is gone. That may be worth a line in the Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332 docs.
3. Earlier findings 2 and 3 are now filed as Omniverse: pinch_ext paths rejected (XR_EXT_hand_interaction compiled out), which fails the whole controller-profile suggestion #2348 (pinch_ext paths) and Closing the hosted window returns XR_ERROR_RUNTIME_FAILURE from xrWaitFrame; apps that bail on it never see STOPPING/EXITING #2349 (RUNTIME_FAILURE on window close).

Asks, unchanged: a review, and whether you'd host this as DisplayXR/displayxr-omniverse. I can open it as a PR there whenever the repo exists.

[joehillthunder]

Final update: DisplayXR for Omniverse (Kit extension displayxr.omniverse)

Where it stands. USD Composer (Kit 110.4, Omniverse XR 109.1, OpenXR on D3D12) runs 3D active on the tested display (Leia SR, Windows 11) through the Kit extension. The 3D is pinned to the desktop camera: the nominal eye sits on the viewport camera and the user picks a screen-plane distance, so the scene stays where the 2D view left it. Code: private repo joehillthunder/displayxr-omniverse (Apache-2.0 + NOTICE). Master is 9e19f5f. PR #1 (a49cabf) holds the overlay-test work. All runtime and Kit internals live in one module, coupling.py.

Overlay PRs = test scaffolding (§0). #2332, #2350 and #2353 were used to measure and nothing more:

• The viewport embed (DXR_HostedViewRect) is pixel-exact and follows resize, F11 and stop/start.
• Composer wove concurrently beside three other hosted apps.
• Live stereo strength (DXR_HostedDepth) stays in sync with a shared depth slider.
• Mouse pass-through (DXR_HOSTED_INPUT=app) is still pending a hands-on check. A posted-message test can't decide it, because Kit reads the real cursor.

Target class: own viewport (texture mode). Kit should draw the woven image into its own viewport, with the vendor plug-in doing the weaving. Kit's menus, gizmos and picking then stay native. The extension becomes a thin adapter. It keeps the camera pin, the screen-plane control and the depth UI. Everything else moves to the shared host library.

What the adapter needs from the shared host library

1. Texture hand-off from a D3D12 renderer. Kit renders each eye into its own RTX textures. The library takes the pair plus the viewport's on-screen rect, and returns or presents the woven result into that rect, with no runtime-owned window.
2. Display info instead of constants. The adapter still hard-codes these, so each one is a named gap until display info provides it:
   • nominal viewing distance (0.6 m);
   • the legacy rig's 2.03 m convergence and 54°×35° FOV;
   • the comfort budget (front ≈0.14 %, behind 0.42 % of width; Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
     The panel's physical size comes from the Windows monitor (DPI and pixels), not from display info.
3. A view-centred frustum option. Kit 110 / Omniverse XR 109 draws nothing for a projection whose frustum doesn't contain the view axis. A true-geometry rig for a viewport above the eye has fov up −0.77°, down −10.4°, and renders black. Unreal and VTK draw the same frustum fine, so this is Kit-specific. For Kit the library should offer either a symmetric frustum with an image-space shift, or a pitched view that keeps the axis inside the frustum.
4. Live depth and convergence as a library call rather than window properties, with one unit: strength relative to the app's nominal.
5. Controllers: the gamepad Touch emulation (Proposal: gamepad EMULATE_PROFILE=touch — pad-driven oculus/touch_controller hands #2333) works with Kit. Kit binds oculus/touch_controller to both hands and exposes trigger, squeeze, a/b/x/y, menu and thumbstick. The per-app choice of profile and pad navigation should be a library setting rather than an environment variable.

Status by OS

• Windows: works.
• Linux: blocked. Kit's XR is Vulkan, which the runtime supports there, but no Linux RTX host was available and there is no display plug-in on Linux.
• macOS and Android: n/a. Kit doesn't ship there.

Pending:

• a seated depth-comfort judgement;
• a hands-on mouse and pad check in 3D (orbit, pan, zoom, picking, button-to-action).

🤖 Generated with Claude Code

[dfattal]

Thanks for the update. Putting each app into one of the two target classes (its own viewport, or a dedicated 3D window), with an explicit list of what it needs from a shared host library, is the right framing. It is the input we're using to design that library.

The overlay PRs (#2332, #2350, #2353) stay as measurement rigs, as you say. We'll post the host-library API here for review when there's a first draft.

[dfattal]

Following up with a result. Woven 3D now runs inside Blender's own 3D View and inside USD Composer's own viewport, with no overlay window. The app keeps its input and presents its own pixels. I checked both seated at a Leia SR panel.

How it works: the app renders two off-axis eyes into textures it owns and hands one D3D11 shared texture to the runtime through the XR_DXR_win32_window_binding v9 texture target. The runtime's display processor weaves into that texture, and the app draws the result back at its viewport rect.

• Blender: gpu.offscreen + draw_view3d for the eyes, and WGL_NV_DX_interop2 for the hand-off.
• Composer: a C++ extension and an omni.ui image layer, D3D12 → D3D11.

The runtime side is on main: #2373 (texture target), #2379 (windowless weave), #2380 (the hub weaves texture sessions). Correct phase inside a sub-rect of the window needs the Leia plug-in's windowless path, DisplayXR/displayxr-leia-plugin#450. That PR is open; until it ships, the released plug-in phases the weave from the window origin.

Measured:

• Composer: placement exact, and the weave runs at 60–72 Hz. The eye images come at 13–14 new frames/s on a laptop RTX 3080, so Kit is limited by the render, not the weave.
• Blender 5.2.3: placement pixel-exact, including after a move or resize. Native per-view resolution. 92 Hz Workbench / 80 Hz EEVEE with both eyes rendered every frame. Clean start/stop/restart. It follows the 3D View into a maximized area or a second Blender window.

On your Kit black-frame finding: it's the same root cause. Kit drops a frustum that doesn't contain the view axis. The prototype works around it with a frustum that contains the axis, cropped back to the true off-axis frustum. A view-centered frustum option on Kit's side would remove the crop.

[joehillthunder]

@dfattal Thanks, that's the route I'd planned for Kit's texture mode, and you've already shown it working seated.

Would you share the Composer prototype (the C++ extension, the omni.ui image layer and the D3D12 → D3D11 hand-off), as a branch or as a start for DisplayXR/displayxr-omniverse? I'd rather build on it than write a second one. displayxr.omniverse would then shrink to what it adds around your hand-off:

• the camera pin (the nominal eye on the viewport camera, so the 3D stays where the 2D view left it);
• the screen-plane and depth controls with the comfort budget (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388);
• display info in place of the remaining constants.

I'd test it here first on runtime 2.55.0 + Leia plug-in 2.18.3 (desktop RTX 5070 Ti, hybrid with Intel; LeiaSR 1.38, so the anchor fallback until 1.39), then on sim-display.

On the crop: an axis-containing frustum cropped back works on Kit with no Kit change, so I'll use that. I'll drop my aperture-offset idea unless you see a reason to keep it.

🤖 Generated with Claude Code

[joehillthunder]

Final update: Omniverse (Kit 110 / USD Composer), 2026-10-27

Windowed 3D in Composer

displayxr.omniverse, on installed runtime v2.55.0 with Leia plug-in 2.18.3.

• Layout: the 3D window docks over Composer's viewport and follows the viewport camera. Screen plane, depth and Fit scene are in a calm main panel; diagnostics are under Advanced.
• Depth fit: fit_depth now places the screen plane and the camera distance from the scene's near and far extents against the comfort budget (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388), not at the bounding-box centre. On a KitBash3D district (Proposal: KitBash3D kits in DisplayXR hosts (content source: Cargo USD → host 3D views at true size, inside the comfort budget) #2435; 55×60×30 m, 0 composition errors, MaterialX surface rendered by Kit RTX) it measured +0.10 to +0.26 % behind against a 0.42 % budget, with nothing in front. The first version measured +0.47 %.
• Content checks in Kit: KitBash3D, and measured materials from Proposal: measured materials at true size in host 3D views (capture device → USD/MDL → Omniverse + Substance 3D Modeler) #2419 and Substance 3D Sampler. Sampler gave 0 errors, textures resolve, OmniPBR binds and DirectX normals are correct. Live updates from the material tools reload in about 1 s while 3D stays active.
• Known issues:
  • Kit's menus are drawn inside its own window, so a menu opening over the viewport is hidden by the 3D window. A fix that hides the 3D window while a menu is open is written but not yet tested.
  • Sampler (Vulkan) and Kit RTX in 3D can lose the GPU device together on a 16 GB card.

Browser stream

Kit App Streaming → DisplayXR Browser + @displayxr/inline3d side-by-side weave.

• Kit renders a side-by-side eye pair, dollied back to frame like the desktop view, and takes the canvas size and depth steps from the page.
• Signaling reaches Kit and the page reaches "3D active", but the WebRTC video leg doesn't connect on streaming client 5.18.11, so there's no video yet. Not a pass yet. Next: align with NVIDIA's web-viewer sample config for Kit 110 / livestream 10.x.

Texture mode (own-viewport 3D, ADR-060): design only, nothing built

• Kit's XR plug-in builds its own XrSessionCreateInfo, so XrWin32WindowBindingCreateInfoDXR can't be chained from Python.
• The workable shape is a small native host session:
  • a D3D12 device on Kit's adapter;
  • Hydra eye cameras posed from xrLocateViews;
  • the woven texture shown at 1:1 in an omni.ui image (CUDA → ByteImageProvider.set_bytes_data_from_gpu).
• Until D3D12 texture sessions: honour the ADR-060 texture-target binding (v9) — sub-rect phase + target state on the hub's client-presents route #2382 lands on D3D12, it needs an anchor window over the viewport.
• Caveats on v2.55:
  • xrWaitFrame doesn't pace texture sessions (xrWaitFrame doesn't pace a session that submits no layers (about 8,700 frames/s before the first image) #2386);
  • the texture-mode atlas is empty (Atlas capture: the per-exe trigger writes nothing for texture-target (client-presents) sessions; window mode works #2421);
  • keep the vFOV canvas-true (XrCameraRigDXR: with an app-framed verticalFov wider than canvas-true, the eyes land far above the app camera and the frustum is entirely below the horizon #2422);
  • a stock install runs hub=0 reason=shared_texture_session.

For the shared host library

1. Frustum: Kit renders an off-axis frustum that lies wholly below the eye axis as black. Render an axis-containing frustum and crop back to the view; no Kit change is needed. Other renderers that assume symmetric frusta will need the same.
2. One host library: Kit, MotionBuilder and Nuke all need the same texture-target core (session, window binding, swapchain copy-in, woven texture out, display info). Each app should be a thin adapter on it, not its own native session.
3. Overlay windows and in-window menus: apps that draw menus inside their own window (omni.ui, ImGui) need the 3D overlay to hide while a menu is open, or an in-place view.
4. VRAM: two GPU-heavy apps beside a 3D session can lose the device on a 16 GB card. The compatible-apps guidance should say so.

🤖 Generated with Claude Code

[joehillthunder]

Readiness on the released runtime: Pass (2026-10-28)

Installed runtime v2.57.1 with the Leia SR plug-in 2.19.1, the shipped extension code, DisplayXR USD Composer (Kit 110.4), default environment.

Check	Result
Launch, "3D active" (OpenXR runtime from Program Files)	Pass
Plug-in: leia-sr 2.19.1 active, no fallback	Pass
Clean exit (2–3 s, no errors in the Kit log)	Pass
Framing at camera pitch ≥ 50° (subject fully in view, no overshoot)	Pass
Depth at 100 %: front −0.111 % / behind +0.272 % of panel width (budget 0.14 % / 0.42 %)	Pass
Framing: subject centred at about half the view height, in Composer's docked viewport	Pass
Seated check at the tested display (Leia SR)	Pass

This supersedes the earlier v2.55 reading that was over the front budget. Not exercised: the live depth slider.