Proposal: DisplayXR for ParaView (scientific visualization, Leia SR tested)

[joehillthunder]

Proposal: DisplayXR for ParaView (scientific visualization, Leia SR tested)

I've built DisplayXR for ParaView, which shows ParaView's render view in glasses-free 3D on DisplayXR displays. The 3D sits inside ParaView's own render view, next to the pipeline browser and properties panel. It follows ParaView's camera, and the mouse keeps working as usual. It weaves at the same time as other 3D apps on the panel. It's tested end to end on a Leia SR laptop. I'd like the maintainers to review it, and to add ParaView to DisplayXR's compatible applications: for example as DisplayXR/displayxr-paraview (experimental), and in a displayxr.org listing once that's hosted and tagged, as with Omniverse in #2331 and 3D Slicer in #2352.

Code: joehillthunder/displayxr-paraview (private for now, Apache-2.0 + NOTICE). @dfattal is invited as a collaborator; personal repos can only grant write. I can add anyone else on the maintainers team.

Why ParaView

ParaView is the standard open-source tool for large scientific and engineering data: CFD, simulation, and volume and point-cloud analysis. It's used across national labs, academia and industry. Flow fields, isosurfaces and clipped meshes are hard to read in 2D. A desk-sized glasses-free display adds depth while the pipeline, filters and plots stay readable, and nothing goes on anyone's head.

What it does

ParaView 6.2's Python ships VTK's OpenXR module (vtkOpenXRRenderWindow, OpenGL, XR_KHR_opengl_enable). It drives that module directly. There's no ParaView rebuild, no plug-in to compile and no renderer fork. ParaView runs as a hosted legacy app. The launcher adds Start 3D / Exit 3D buttons to ParaView's Macros toolbar, and the status reads "3D active".

• Not "Send to XR". ParaView's XRInterface plug-in is built for a standing headset user. It frames a 1.6 m room-scale scene around the viewer, runs a blocking render loop and always starts on OpenVR. This module opens its own OpenXR session instead:
  • It renders in ParaView's GL context (as the helper window) with the render view's props, so nothing is copied or reloaded.
  • Frames are driven by a Qt timer on the view's interactor, so ParaView stays fully interactive (~57 fps measured).
• Runtime selection per app. The launcher sets XR_RUNTIME_JSON and leaves the system OpenXR runtime alone.
• In the app's own window. DXR_HOSTED_WINDOW=auto (Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332) embeds the woven window over ParaView. A small follower publishes the render view's rect (QVTKOpenGLNativeWidget) as DXR_HostedViewRect, so the 3D covers exactly the render view and follows layout changes.
• Normal mouse interaction. Rotate, pan, zoom, picking and 3D widgets work through the 3D. This needed a new runtime mode, DXR_HOSTED_INPUT=app (draft PR feat(d3d11): DXR_HOSTED_INPUT=app — pass the mouse through an embedded 3D view #2353, stacked on Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332):
  • The woven window gets WS_EX_NOACTIVATE / MA_NOACTIVATE, so it never takes focus.
  • Mouse messages are posted to the host in host-client coordinates.
  • 3D Slicer has adopted the same mode.
• Camera pinning. Every frame the module calls SetPhysicalToWorldMatrix with columns s·[right, up, −dop] of ParaView's camera, mapping the rig's nominal eye (0, 1.6, 0) onto it. The focal point sits on the glass at the size the 2D view shows it. Head tracking adds look-around parallax on top. The mapping must use the nominal eye, not the tracked eye, or the scene swims.
• True-geometry rig. At Start 3D, DXR_LEGACY_CAMERA_RIG is set as follows:
  • the glass is at 0.6 m;
  • vFOV = 2·atan(canvas height / 2 / 0.6);
  • ipdFactor is fitted so the data uses the comfort budget: 0.14 % of panel width in front, 0.42 % behind (the Proposal: DisplayXR for NVIDIA Omniverse (Kit extension, Leia SR tested) #2331 calibration).
• Comfort, per frame. The module measures the frustum-visible depth of the visible props each frame. When you zoom into a region deeper than the budget allows, the 3D is shown smaller than the 2D view rather than too strong. VTK's OpenXR path has no depth compression, so this is the only safe lever. Pressing Start 3D again re-fits to the current view.
• Live depth. The shared DisplayXR Depth / Focus slider is docked in ParaView's menu bar (DXR_HostedDepthRelNow, d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350):
  • Depth below 100 % flattens the 3D. Above 100 % is written back to 100 %, so the slider can't push ParaView past the budget.
  • Focus moves ParaView's focal point relative to the glass.
• Omniverse. Omniverse Live uses ParaView 6.2's own USD exporter, so no NVIDIA connector is needed (NVIDIA's ParaView Connector lists 5.12 only). It writes paraview_live.usda and re-exports whenever the visible scene, colouring or time changes.
  • The stage is wrapped in Kit's default layout: a /ParaView defaultPrim, Y-up from the camera's view-up, and cm.
  • Material paths are rebased and textures are relative and renamed per export (RTX caches textures by path). Colour-map textures are clamped, because USD's default repeat wrap gives dark, wrong colours at the colour range's ends.
  • Files are replaced atomically, so Kit's layer auto-reload never reads a half-written stage.
• Clean exit. Exit 3D ends the session and restores the 2D view. ParaView keeps running.

Tested

ParaView 6.2.0 (Windows, Python 3.12) on DisplayXR 2.53.0. That's a dev build with #2332, #2350 and #2353, plus the Leia SR plug-in 2.18.0, on an RTX 5070 Ti and a Leia SR laptop panel (3840 × 2400). Demo: ParaView's disk_out_ref.ex2 (a translucent clipped housing coloured by temperature, plus stream tubes).

• Projection-atlas captures (displayxr_atlas_trigger.paraview) on sim-display, with marker spheres at the front, glass and back of the scene:
  • Glass disparity is 0.
  • Front and back land at 85–86 % of the model (conservative).
  • Depth 50 % halves disparity (0.358 → 0.192 mm).
  • Depth 180 % is capped, and the image is identical to 100 %.
  • Focus 0.9 m sinks the scene to the behind budget.
• Interaction: a posted mouse drag over the woven window rotated ParaView's camera, and focus stayed with ParaView.
• Concurrent weaving: ParaView (embedded) wove at the same time as USD Composer, Substance Modeler and 3D Slicer.
• Omniverse: the live USD was checked in USD Composer (Kit 110.4, RTX). It loads upright and life size, with materials and translucency matching ParaView, and reloads in 0.35–0.62 s.
• Platforms:
  • Windows: yes.
  • Linux: blocked. VTK's OpenXR is OpenGL-only, and DisplayXR on Linux has only a Vulkan compositor (filed as Linux: run OpenGL (XR_KHR_opengl_enable) legacy apps — VTK-based ParaView / 3D Slicer, Blender VR #2354, with the GL client compositor route).
  • macOS: blocked. ParaView's macOS builds don't include VTK RenderingOpenXR, and there's no 3D display plug-in there yet.

Runtime-side findings (for discussion)

1. Hosted apps need to keep focus. In embedded mode the woven window took focus and swallowed the mouse, so a DCC app's own navigation stopped working. feat(d3d11): DXR_HOSTED_INPUT=app — pass the mouse through an embedded 3D view #2353 (DXR_HOSTED_INPUT=app) fixes this for ParaView and Slicer, and should help any hosted legacy app with its own viewport navigation.
2. Focus nearer than the glass is clamped. qwerty_set_depth caps convergence via dxr_rig_max_convergence, so the Focus slider can only push the scene back from 0.6 m, not bring it forward. Is that intended for legacy rigs?
3. No viewer → identical eyes with leia-sr (same as Proposal: DisplayXR for 3D Slicer (medical visualization, Leia SR tested) #2352 finding 1). On-panel measurement needs sim-display.
4. Unmerged dependencies. The windowed experience needs Proposal: embed hosted windows in the app's own window (DXR_HOSTED_WINDOW, viewer inset, F11) #2332, d3d11: live depth for embedded apps via DXR_HostedDepth (slider-ready; stacked on #2332) #2350 and feat(d3d11): DXR_HOSTED_INPUT=app — pass the mouse through an embedded 3D view #2353. With released v2.53 the module still works and navigation from the 2D view drives the 3D, but the 3D covers the whole panel.

Asks

• A review of the approach and code.
• Adding ParaView to DisplayXR's compatible applications: hosting as DisplayXR/displayxr-paraview (experimental, me as maintainer), and a displayxr.org listing after a tagged release.
• Review of feat(d3d11): DXR_HOSTED_INPUT=app — pass the mouse through an embedded 3D view #2353 (input pass-through), and thoughts on Linux: run OpenGL (XR_KHR_opengl_enable) legacy apps — VTK-based ParaView / 3D Slicer, Blender VR #2354 (GL legacy apps on Linux), which would bring ParaView, Slicer and Blender VR to Linux.

[joehillthunder]

Final update: DisplayXR for ParaView

Code: joehillthunder/displayxr-paraview @ ec2214e (private, Apache-2.0 + NOTICE; every DisplayXR or ParaView internal it relies on is in python/coupling.py).

Where this sits against the architecture direction

The overlay embed this was built and measured on (#2332 view rect, #2350 live depth, #2353 input pass-through) is test scaffolding. No further work is planned on those PRs; they stay as a measurement rig.

• Target class: in the app's own viewport, through a texture hand-off. ParaView already renders both eye images with VTK's vtkOpenXRRenderWindow inside ParaView's own OpenGL context (ParaView's render window is VTK's helper window). So the adapter needs nothing new on the rendering side. The host library would return the woven result for the render view's rect, and ParaView would draw it into its own QVTKOpenGLNativeWidget. ParaView's toolbars, tooltips, 3D widgets and menus then stay on top, and its mouse handling stays native with no pass-through. The fallback is a dedicated 3D window holding just the render view.
• What this adapter needs from the shared host library:
  1. A GL texture hand-off: VTK renders the eye images, the library returns a woven texture plus the screen rect it lands on, and the vendor plug-in weaves (windowless for the tested display).
  2. Display info as data: panel size and nominal viewing distance (already read from display info), plus the comfort budget (a gap, Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
  3. A way for the app to own navigation: no runtime fly camera or emulated hands for camera-pinned apps.
  4. Live depth / focus as a library call rather than a window property.
  5. A scene depth range the comfort clamp can use (Focus on camera rigs: convergence isn't a pure glass shift, and the comfort clamp assumes content at infinity #2385).
• What stays in the ParaView adapter (application-specific, about 1k lines of Python): pinning the XR rig to ParaView's camera every frame, the comfort-budget depth fit, the frustum-visible depth per frame, the Start 3D / Exit 3D buttons, gamepad navigation, and the USD export.

Since the proposal

• Display values from display info. Panel width and height and the nominal viewing distance come from DisplayXR's display info (displayxr-cli status --json: layout.width_m, layout.nominal_viewer_m.z), not constants. Named gap: display info has no comfort budget, so the front/behind disparity limits (0.14 % / 0.42 % of panel width, calibrated on the tested display) are still a constant in the adapter.
• On-panel depth check: with 3D active at the defaults (Depth 100 %, focal point on the glass), the user's verdict was "pretty good". The defaults are kept.
• Input:
  • Mouse: ParaView's own rotate, pan, zoom, picking and 3D widgets work through the 3D (posted-drag test). Menus and panels sit outside the render view.
  • Gamepad: ParaView has no VR-controller interface outside XRInterface, so an Xbox-style pad drives ParaView's own camera: left stick orbit, right stick pan (in the mouse-drag directions), triggers zoom, A resets. It acts only while ParaView is the foreground window, so a pad used with another 3D app on the panel doesn't move ParaView too. The runtime's pad fly camera and emulated controllers are off for ParaView (DXR_INPUT_GAMEPAD_NAV=0, EMULATE_CONTROLLERS=0), because the fly camera would move the 3D off ParaView's camera.
  • The pad mapping is verified headless with a simulated pad (0.5 s full stick = 45.0° orbit at constant distance; 0.5 s full trigger = ×1.414; ParaView's camera properties stay in sync). Pending: a physical-pad check.
• Omniverse: the live USD export (ParaView 6.2's own exporter, wrapped Y-up in cm under /ParaView) was verified in USD Composer: upright, materials and translucency match, and reloads take 0.35–0.62 s. ParaView's 3D wove at the same time as USD Composer, Substance Modeler and 3D Slicer.

Status

In its own window, concurrent with other 3D apps	yes (overlay scaffolding; target: texture hand-off)
Omniverse content / weaves beside Kit	yes / yes
Windows	yes
Linux	blocked: VTK OpenXR is OpenGL-only; the runtime is Vulkan-only on Linux (#2354)
macOS	blocked: ParaView's macOS builds don't include VTK RenderingOpenXR; no 3D display plug-in there yet
Android	n/a
Live depth control	yes (shared Depth / Focus slider; capped at the comfort budget)
Input	mouse yes; gamepad implemented, physical check pending

Runtime findings

1. Comfort budget missing from display info (Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388). Every adapter hard-codes the front/behind disparity limits. A field such as layout.comfort {front_frac, behind_frac} from the vendor plug-in would remove the last per-display constant.
2. The focus clamp assumes content at infinity (Focus on camera rigs: convergence isn't a pure glass shift, and the comfort clamp assumes content at infinity #2385). The comfort clamp on convergence (dxr_rig_max_convergence, qwerty_device.c) caps the zero-parallax plane at the nominal distance, so the Focus slider can only push ParaView's scene back, never forward. The code notes that a depth-aware caller could pass a scene far plane; there's no way for an app to do so. ParaView knows its frustum-visible depth every frame and could supply it.
3. Camera-pinned legacy apps and the pad fly camera. With NAV=1 (the default), a pad touch moves the rig, and apps that pin the XR views to their own desktop camera drift away from their 2D view. The host library should let an adapter say that it owns navigation. This goes into the cross-app proposal for the host library.
4. No viewer, identical eyes on the tested display (leia-sr: no tracked viewer gives identical eyes, so on-panel depth can't be measured; add a measurement switch or document it #2384). On-panel disparity can't be measured unless someone is seated, so measurements use sim-display.

🤖 Generated with Claude Code

[dfattal]

Thanks for the update. Putting each app into one of the two target classes (its own viewport, or a dedicated 3D window), with an explicit list of what it needs from a shared host library, is the right framing. It is the input we're using to design that library.

The overlay PRs (#2332, #2350, #2353) stay as measurement rigs, as you say. We'll post the host-library API here for review when there's a first draft.

[joehillthunder]

Update: ParaView Glance in the DisplayXR Browser

ParaView's web viewer, ParaView Glance 4.25 (vtk.js, shipped inside ParaView 6.2 under share/paraview-6.2/web/glance), now shows 3D in the DisplayXR Browser.

Why a port was needed. The browser's auto-3D leaves Glance flat ("a WebGL canvas but no supported engine found"): it only recognises three.js and PlayCanvas.

How it works:

• An inline-3D port puts one woven canvas over Glance's 3D view. The canvas ignores the mouse, so Glance's own mouse and camera are untouched.
• Each eye is rendered through vtk.js's own per-eye calls (computeViewParametersFromViewMatrix, setProjectionMatrix, traverseAllPasses).
• The camera rig is pinned to Glance's camera. The glass and eye separation are fitted to the visible depth, inside the comfort budget (0.14 % front / 0.42 % behind of panel width; Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388).
• The status bar reads "3D active", with Depth −/+ in 25 % steps and Reset.

Measured (installed DisplayXR Browser v1.8.0):

• firstWoven: 518 ms cold, 36 ms on reload.
• About 60 fps, with 1 mono frame in 3,700.
• It kept weaving while another app wove in its own window.
• A page drag orbits Glance's camera through the 3D tile.
• Test spheres at known depths: +0.66 / +0.33 / −0.22 % of tile width, against +0.65 / +0.32 / −0.24 % predicted. 50 % depth halves them.

Findings:

• On-panel measurement without a seated viewer isn't possible from the browser either; see the leia-sr: no tracked viewer gives identical eyes, so on-panel depth can't be measured; add a measurement switch or document it #2384 comment.
• auto-3D could gain a vtk.js adapter, but vtk.js has no global announce hook like three.js's __THREE_DEVTOOLS__, so it would need one upstream.

Code: joehillthunder/displayxr-paraview, folder web/, commit 5697464 (private, Apache-2.0 + NOTICE).

Pending: a seated check on the display.

🤖 Generated with Claude Code

[dfattal]

The same in-viewport route now runs in 3D Slicer 5.12.4 and ParaView 6.2, on the texture target. There's no OpenXR render window and no SlicerVirtualReality: two off-axis eyes come from plain vtkCameras (UseOffAxisProjection + screen corners), and one helper DLL is shared with the Blender prototype.

Measured with a simulated viewer; the seated check is still to come:

• Placement is pixel-exact in the app's own 3D/render view, including after a move, a resize and a restore. Slicer also follows a floating (undocked) 3D view and back.
• Picking works with 3D on. The pick has to come from the rig's center eye rather than the app's mono camera.
• Rates: Slicer 66–67 Hz, ParaView 66–67 Hz woven.
• Parallax measured from the runtime's views matches the off-axis prediction: within 0.5 % on Slicer, within 0.09 % of width on ParaView.

Your adapter logic carries over directly: the camera pin (as the model matrix), the true-geometry rig and the comfort solver. On Slicer, your screen-plane solver now drives the rig without ever moving Slicer's camera, and your calibrate-spheres scene is the test oracle.

Two app-side lessons:

• Draw the woven result back in the last vtkRenderer's EndEvent, before VTK copies the frame out. Drawing later lands in Qt's buffer and never reaches the screen.
• On a fractional DPI scale (250 %), Qt composites the view one pixel smaller than the render size, which resamples the weave. The library will need a rule for that.

ParaView: StillRender() renders the same eye twice, so the eye passes call the render window directly.

[joehillthunder]

Addendum (2026-10-27): the pending physical-pad check is done. On the panel with 3D active, an Xbox pad over XInput drove ParaView's own camera: left stick orbits and right stick pans, both at constant distance. The pad was ignored while ParaView wasn't in the foreground. The seated user's verdict was "looks good". Triggers (zoom) and A (reset) are verified headless only.

🤖 Generated with Claude Code

[joehillthunder]

Seated check: ParaView in its own render view (2026-10-28)

ParaView 6.2 now draws its 3D inside its own render view on the texture target, and a seated viewer checked it on the tested display (Leia SR).

• Setup: installed runtime v2.57.1 with the Leia SR plug-in 2.19.1, the default launcher, no dev runtime.
• Result: drawn pixel-exact in the render view, about 25–29 eye pairs/s. Seated check: Pass at 100 % depth and at 140 % (opt-in); 140 % preferred. The screen plane now fills both budgets (front 0.14 %, behind 0.42 % at 100 %). Clean exit.

Earlier today, the hosting-readiness pass on v2.55.0 also passed. The default launcher from a clean shell loads the Program Files runtime with active plug-in: id=leia-sr and reaches "3D active" (54–58 fps). The default depth is the comfort-budget fit at 100 %, Exit 3D switches the lens back to 2D, and ParaView quits cleanly.