Proposal: DisplayXR for 3D Slicer (medical visualization, Leia SR tested)

[joehillthunder]

Proposal: DisplayXR for 3D Slicer (medical visualization, Leia SR tested)

I've built a 3D Slicer module, DisplayXR 3D, that shows Slicer's 3D view in glasses-free 3D on DisplayXR displays. The 3D sits inside Slicer's own 3D view, next to the 2D axial / coronal / sagittal slices, and weaves at the same time as other 3D apps on the panel. It's tested end to end on a Leia SR laptop. I'd like the maintainers to review it, and to add 3D Slicer to DisplayXR's compatible applications: for example as DisplayXR/displayxr-slicer (experimental), and in a displayxr.org listing once that's hosted and tagged, as with Omniverse in #2331.

Code: joehillthunder/displayxr-slicer (private for now, Apache-2.0 + NOTICE). @dfattal is invited as a collaborator; personal repos can only grant write. I can add anyone else on the maintainers team.

Research and demonstration software. Not a medical device; not for diagnosis.

Why Slicer

3D Slicer is the most widely used open-source medical imaging platform (CT/MRI volume rendering, segmentation, surgical planning). Clinicians and researchers already read anatomy in 3D, and a desk-sized glasses-free display fits that workflow better than a headset: the slice views and the rest of the UI stay readable, and nothing goes on anyone's head.

What it does

Slicer gets OpenXR from the SlicerVirtualReality extension (VTK vtkOpenXRRenderWindow, OpenGL, XR_KHR_opengl_enable), so there's no renderer fork. Slicer runs as a hosted legacy app. The module adds Start 3D; the status reads "3D active".

• Runtime selection per app. The launcher sets XR_RUNTIME_JSON and leaves the system OpenXR runtime alone.
• In the app's own window. DXR_HOSTED_WINDOW=auto (Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332) embeds the woven window over Slicer. The module publishes Slicer's 3D view rect as DXR_HostedViewRect every 100 ms, so the 3D fits the 3D view exactly at any layout or size (measured: 1782×1815 at the view's client rect). F11 or the "Full screen" button toggles the whole panel through the runtime's own F11.
• Normal mouse interaction. Slicer's own rotate, pan, zoom and markups keep working: with DXR_HOSTED_INPUT=app the runtime posts the mouse over the 3D to Slicer (branch hosted/input-passthrough, from the ParaView integration; PR to follow). On older runtimes the module makes the woven window click-through instead. The 3D follows Slicer's 3D camera every frame.
• Camera pinning. Each tick the module calls vtkOpenXRRenderWindow.SetPhysicalToWorldMatrix with columns s·[right, up, −dop] of Slicer's camera, mapping the rig's nominal eye (0, 1.6, 0) onto the camera, pitch included. No pitch/pullback profile is needed. Plain SlicerVirtualReality on DisplayXR leaves the data out of frame.
• True-geometry rig. Per start, DXR_LEGACY_CAMERA_RIG = glass at the 0.6 m nominal viewer, vertical FOV = the 3D view's physical height at 0.6 m, ipdFactor = K · view width / 0.063. Virtual metres map 1:1 onto the panel, so parallax is predictable: p(z) = K·(1 − g/z) as a fraction of view width, with g the glass distance.
• Comfort. "Keep depth comfortable" places the glass from the visible content's depth so it stays inside the calibrated budget (0.14 % of width in front, 0.42 % behind, as in Proposal: DisplayXR for NVIDIA Omniverse (Kit extension, Leia SR tested) #2331). When the user zooms in too close for the budget, it moves the viewpoint back. Example: 2.5× zoom → 585 mm push-back, exactly 0.14 % / 0.42 %. The panel shows the live depth.
• Live depth. Panel sliders Depth (5–200 %, via DXR_HostedDepthRel, d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350) and Screen plane (moves the glass in-module, with no size change). While 3D runs, the module also starts the shared DisplayXR depth slider with -ProcessId, docked in Slicer's menu bar; both stay in sync through DXR_HostedDepthRelNow.
• Omniverse. "Send to Omniverse (USD)" exports visible segmentations and models as .usda (UsdPreviewSurface, vertex normals) in Kit's default layout (Y-up, cm). The scaling is baked in because Kit doesn't convert units on scripted references. Checked in USD Composer: life size and upright.
• Clean exit. Stop 3D restores Slicer's camera view angle, removes the window properties and closes the shared slider.

Tested

3D Slicer 5.12.4 + SlicerVirtualReality c9179ed on DisplayXR 2.53.0 (dev build with #2332 / #2350 / input pass-through), Leia SR plug-in 2.18.0, RTX 5070 Ti, Leia SR laptop panel 3840 × 2400.

• Projection-atlas captures (displayxr_atlas_trigger.SlicerApp-real.projection) on sim-display (DXR_PLUGIN_EXCLUSIVE), with spheres at known depths:
  • the model is within ~15 % and conservative (spheres at 0.8× / 1.4× the glass: −0.26 % / +0.29 % measured vs −0.30 % / +0.34 % predicted);
  • Depth 200 % gives 2.00× parallax;
  • Screen plane +50 % puts the glass at exactly 1.5× (−0.52 % measured vs −0.52 % predicted);
  • a DXR_HostedDepthRel write of 0.5 from outside gives 0.50×, and the panel follows.
• CT chest volume rendering at the reset view: −0.05 % … +0.10 % (block matching). The user's verdict on the panel: "looks good".
• Concurrent weaving: Slicer (embedded), Substance Modeler (embedded) and USD Composer (embedded viewport) wove at the same time; the other apps' logs stayed clean.
• F11 round trip, Stop/Start in one process, a posted mouse drag rotating Slicer's camera through the 3D window, and two Slicer instances each with their own slider all work.
• The USD export opens and references in USD Composer (Kit 110.4, RTX) without errors and renders in the woven 3D.
• Platforms: Windows yes. Linux is blocked: Slicer is OpenGL, DisplayXR on Linux is Vulkan-only, and there's no Leia plug-in. macOS is untested: there's no Leia plug-in there, only sim-display.

Runtime-side findings (for discussion)

1. No viewer → identical eyes. With leia-sr and nobody in front of the panel, xrLocateViews returns eye0 == eye1 (3D-GATE … eye0=(0,0.1,0.6) eye1=(0,0.1,0.6), tracking=0), so every capture shows zero disparity. That's probably intended (fade to 2D), but it makes on-panel measurement impossible without a seated viewer. A documented "fixed nominal eyes" switch for leia-sr, or a note in the capture docs, would help integrators. sim-display is the workaround.
2. Per-exe atlas triggers collide across instances. displayxr_atlas_trigger.<exe> is keyed by exe name, so with two SlicerApp-real processes whichever polls first answers. A PID-suffixed variant would make concurrent-app testing reliable.
3. Focus via convergence isn't a pure glass shift for a camera rig. Already noted in d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350: 0.6 → 0.9 m moved the glass to ≈1.63× (not 1.5×) and about halved parallax-at-infinity. Slicer avoids it by moving the glass in-module.
4. Unmerged dependencies. The windowed experience needs Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332, d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350 and the input pass-through. With released v2.53 the module still works, but the 3D covers the whole panel.

Suggested next step: a displayxr-vendor-template note on what a display plug-in needs for apps like this (nominal viewing distance, panel size, and the comfort budget used here).

Asks

• A review of the approach and code.
• Adding 3D Slicer to DisplayXR's compatible applications: hosting as DisplayXR/displayxr-slicer (experimental, me as maintainer), and a displayxr.org listing after a tagged release.
• Thoughts on findings 1 and 2, and on landing Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332 / d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350 / input pass-through, which the windowed experience depends on.

[joehillthunder]

Final update: 3D Slicer

Code: joehillthunder/displayxr-slicer @ 4ddd8f0 (private, Apache-2.0 + NOTICE; the maintainer has write access). Research and demonstration software; not a medical device.

Status

• Windows: working. The tested display is Leia SR. 3D Slicer 5.12.4 + SlicerVirtualReality (VTK OpenXR, OpenGL). The status reads "3D active".
• Depth: a true-geometry rig per start (glass at the nominal viewing distance, vFOV = the 3D view's physical height), with a per-frame physical-to-world pin to Slicer's 3D camera, plus an automatic screen plane and push-back that keep content inside the comfort budget (0.14 % front / 0.42 % behind of view width). Measured on sim-display: CT chest −0.05 % … +0.10 %; sphere calibration within ~15 %, conservative. On the tested display: "looks good" at the default.
• Input (§5):
  • Mouse: Slicer's own rotate / pan / zoom / markups.
  • Gamepad: XInput drives Slicer's own camera while Slicer is in front (left stick orbit, right stick pan, triggers zoom, A reset). The runtime's pad fly camera and emulated controllers are off for Slicer (DXR_INPUT_GAMEPAD_NAV=0, DXR_INPUT_GAMEPAD_EMULATE_CONTROLLERS=0), because the 3D is pinned to Slicer's camera.
  • Verified with a simulated pad (90°/s orbit, 2×/s zoom, the 3D follows, depth stays in budget). Physical pad test pending.
• Omniverse content: "Send to Omniverse (USD)" writes visible segmentations and models as .usda in Kit's default layout (Y-up, cm). It opens and renders in USD Composer.
• Platforms:
  • Linux: blocked (OpenGL app; Linux: run OpenGL (XR_KHR_opengl_enable) legacy apps — VTK-based ParaView / 3D Slicer, Blender VR #2354).
  • macOS: untested (sim-display only).

Against the architecture direction

• Overlay PRs are test scaffolding. The windowed embed, view rect, live depth and input pass-through (Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332 / d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350 / feat(d3d11): DXR_HOSTED_INPUT=app — pass the mouse through an embedded 3D view #2353) were used only to test and measure. Nothing here depends on them landing.
• Target class: in Slicer's own 3D view, through a VTK texture hand-off. vtkOpenXRRenderWindow already renders both eyes into its own GL framebuffers. The adapter would hand those (or one atlas texture) plus the 3D view's on-screen rect to the shared host library. The vendor plug-in weaves into that rect windowlessly, and Slicer's menus, tooltips and markups stay on top. A dedicated 3D window (e.g. full screen on a second 3D display) is the fallback.
• What the Slicer adapter needs from the shared host library:
  1. A GL texture hand-off (or GL↔D3D interop inside the library) with a per-frame target rect in physical pixels (cf. Design: windowless target-rect binding (decouple position/metrics channel from HWND) #697).
  2. Per-frame tracked eye poses, so the adapter can keep pinning the rig to Slicer's camera.
  3. Display info: panel size (m, px), nominal viewing distance, and the comfort budget (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
  4. Session lifecycle: resize, dock/undock, multiple monitors, start/stop in one process.
  5. A per-app opt-out of the runtime's pad navigation, since the adapter navigates the app's own camera.
• Display values: from display info. Panel size already comes from displayxr-cli status --json. Named gaps, still constants in displayxr_coupling.py / DisplayXR.py:
  • nominal viewing distance (0.6 m);
  • IPD reference (63 mm);
  • comfort budget (0.14 % / 0.42 %);
  • default strength K (1.2 % of view width at infinity);
  • rig eye (0, 1.6, 0);
  • a panel-size fallback.

Runtime findings

• Atlas trigger collision: Proposal: per-instance atlas capture trigger (displayxr_atlas_trigger.<exe>.<pid>) — two copies of one app collide under windowed weaving #2351.
• Linux OpenGL legacy apps: Linux: run OpenGL (XR_KHR_opengl_enable) legacy apps — VTK-based ParaView / 3D Slicer, Blender VR #2354.
• Comfort budget in display info: Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388 (related: Proposal: panel-relative legacy camera profile (converge on the glass) + shipped blender.exe profile + desktop comfort budget #2330).
• No tracked viewer gives identical eyes, so on-panel depth can't be measured: leia-sr: no tracked viewer gives identical eyes, so on-panel depth can't be measured; add a measurement switch or document it #2384.
• Focus via convergence isn't a pure glass shift on a camera rig: Focus on camera rigs: convergence isn't a pure glass shift, and the comfort clamp assumes content at infinity #2385.

Asks

• Review, and hosting as DisplayXR/displayxr-slicer (experimental) once the texture hand-off path exists.
• Slicer as a test case for the shared host library's GL/VTK texture hand-off.

🤖 Generated with Claude Code

[dfattal]

Thanks for the update. Putting each app into one of the two target classes (its own viewport, or a dedicated 3D window), with an explicit list of what it needs from a shared host library, is the right framing. It is the input we're using to design that library.

The overlay PRs (#2332, #2350, #2353) stay as measurement rigs, as you say. We'll post the host-library API here for review when there's a first draft.

[dfattal]

The same in-viewport route now runs in 3D Slicer 5.12.4 and ParaView 6.2, on the texture target. There's no OpenXR render window and no SlicerVirtualReality: two off-axis eyes come from plain vtkCameras (UseOffAxisProjection + screen corners), and one helper DLL is shared with the Blender prototype.

Measured with a simulated viewer; the seated check is still to come:

• Placement is pixel-exact in the app's own 3D/render view, including after a move, a resize and a restore. Slicer also follows a floating (undocked) 3D view and back.
• Picking works with 3D on. The pick has to come from the rig's center eye rather than the app's mono camera.
• Rates: Slicer 66–67 Hz, ParaView 66–67 Hz woven.
• Parallax measured from the runtime's views matches the off-axis prediction: within 0.5 % on Slicer, within 0.09 % of width on ParaView.

Your adapter logic carries over directly: the camera pin (as the model matrix), the true-geometry rig and the comfort solver. On Slicer, your screen-plane solver now drives the rig without ever moving Slicer's camera, and your calibrate-spheres scene is the test oracle.

Two app-side lessons:

• Draw the woven result back in the last vtkRenderer's EndEvent, before VTK copies the frame out. Drawing later lands in Qt's buffer and never reaches the screen.
• On a fractional DPI scale (250 %), Qt composites the view one pixel smaller than the render size, which resamples the weave. The library will need a rule for that.

ParaView: StillRender() renders the same eye twice, so the eye passes call the render window directly.

[joehillthunder]

Seated check: 3D Slicer in its own 3D view (texture target)

3D Slicer 5.12.4 now draws its 3D inside its own 3D view on the released runtime, and the seated check at the tested display (Leia SR) is a Pass.

• Setup: installed DisplayXR v2.55.0 + Leia plug-in 2.18.3, the repo's default launcher, no dev runtime, CT chest volume rendering at the default depth (100 %).
• How: two off-axis vtkCameras rendered by Slicer's own renderer, with each eye's projection pinned to the runtime's eye tangents. The app-neutral host core runs the session through XR_DXR_win32_window_binding v9. In the last vtkRenderer's EndEvent, the woven texture is blitted back through WGL_NV_DX_interop2; that's a framebuffer blit, since VTK's context is core-profile. No OpenXR render window, no SlicerVirtualReality, nothing over Slicer's UI.
• Measured (simulated display, CT chest): the eye projections match the runtime's eyes to 0.001 px. Nearest / farthest content sit at −0.140 % / +0.342 % of panel width (budget 0.14 % / 0.42 %), the glass at 0, and Depth 50 % halves it.
• On the panel: the view is pixel-exact (1689×1815), ~28 eye pairs/s on volume rendering. Anchor fallback (state 0x2) with LeiaSR 1.38; LeiaSR ≥ 1.39 should give the windowless phase.
• Open: picking still uses Slicer's mono camera, not the rig's centre eye. The rate is bound by three renders per pair (two eyes + the draw-back pass).

[joehillthunder]

Recheck on runtime 2.57.1: 3D Slicer in its own 3D view (texture target)

Seated check: Pass. Conditions: installed DisplayXR 2.57.1 + Leia plug-in 2.19.1 (LeiaSR 1.38, so anchor fallback, state 0x2), the repo's default launcher, 3D Slicer 5.12.4, CT chest volume rendering at Depth 100 %, ~27 eye pairs/s, view 1689×1815 drawn pixel-exact.

Measured (simulated display, CT chest, tests/test_viewport.py, displayxr-slicer bb075c1):

• Eye projections vs the runtime's eyes: 0.000 px error. Unchanged from 2.55.0.
• The rig's fit: −0.140 % / +0.366 % of panel width at the nearest / farthest content bounds (budget 0.14 % / 0.42 %). Depth 50 % halves it.
• Projection atlas, now available in texture mode (fix(compositor): texture-class sessions answer the post-compose atlas trigger (#2421) #2426), block-matched: −0.03 % … +0.21 % of panel width (p1 … p99). That's inside the budget. The bounds-based fit is conservative for volume rendering.

No regressions from 2.55.0 + 2.18.3.