Proposal: Off-the-shelf OpenXR apps on DisplayXR: survey, triage, and what unmodified apps need

[joehillthunder]

Why

Many free and open-source OpenXR apps (painting, modelling, engine demos, training) could show 3D in their own window on a 3D display, with no source changes. This post records what happened when we ran them unmodified, what the runtime could do so more of them just work, and what's still to test.

Triage classes

• A: runs hosted; needs only framing and depth.
• B: runs, but needs input mapping (it expects controllers).
• C: runs only full panel, or with breakage.
• D: refuses the runtime or crashes.
• E: OpenVR-only or no XR.

Results so far

Windows 11 x64, sim-display, dev runtime with the hosted-window embed (#2332).

App	Engine / API	Class	Notes
Khronos hello_xr (OpenXR-SDK 1.1.65)	D3D11 / Vulkan / OpenGL	A	Runtime-created window (console app)
Khronos hello_xr	D3D12	D	Device removed on the first frames: #2430
StereoKit test app 0.3.11	D3D11	A	Runtime-created window; ships a win-arm64 build
Open Brush 2.32	Unity OpenXR	A+B	Embeds over its own window; needs controller input
Open Blocks 0.11	Unity OpenXR	A+B	Embeds over its own window; needs controller input
Godot XR Tools demo 4.5.2	Godot 4, GL	D → A+B	Needs GL depth swapchain formats: draft PR #2429

Pending: Shapelab, Gravity Sketch, 3D Organon XR, and two industrial safety-training titles. These wait on store installs. Two flight sims are waiting on disk space.

What we learned

1. Implicit API layers can break apps. A machine-wide hand-tracking implicit layer crashed a Unity player inside xrEndFrame. Setting the layer's own disable_environment variable in the launcher fixed it.
2. Unmodified apps start framed too close. The legacy camera rig puts their start content far too near and too large. What works is a true-geometry rig computed from display info:
   • verticalFov = 2·atan(h / 2 / z);
   • convergenceDiopters = 1 / g;
   • ipdFactor = K·w·g / (0.063·z).
     Here w and h are the 3D view's physical size, z is the nominal viewer distance, g is the app distance that lands on the screen, and K = 0.42 % parallax at infinity. Our launchers compute it at every start from displayxr-cli status --json, so the same launcher fits any panel and window size.
3. Controller-only apps need input. Gamepad Touch emulation (Proposal: gamepad EMULATE_PROFILE=touch — pad-driven oculus/touch_controller hands #2333) or a mouse hand (qwerty: mouse hand under the cursor + XR_LOGITECH_mx_ink_stylus_interaction #2347) makes them usable on a desk.
4. Two window behaviours.
   • Unity and Godot players have a window, and the 3D embeds into it.
   • Console apps (hello_xr, StereoKit) get a runtime-created window, so they need the dedicated-3D-window path rather than the embed.
5. Unity players with runtime depth swapchains on GL need fix(gl): advertise depth swapchain formats on the native GL compositor #2429, and D3D12 users of depth swapchains may hit D3D12: Khronos hello_xr loses its device on the first frame (DXGI_ERROR_INVALID_CALL) #2430.

Ideas for the runtime

• A default legacy rig from display info. Use the formula above, with the comfort budget from Display info: publish the comfort budget (front/behind disparity limits), e.g. layout.comfort {front_frac, behind_frac} #2388. A legacy app would then open at a comfortable depth with no per-app profile.
• Warn about implicit layers. Log a warning, or offer a per-app opt-out, when an implicit API layer is loaded into a legacy session, since it's a common hidden cause of crashes.
• A dedicated 3D window for console-style apps, so they can run windowed and alongside other apps like the embedded ones.

Next

• Calibrate each app on the panel, with depth ratings that record the display's size and resolution.
• Map input, run them side by side with other apps, and check the full-panel round trip (F11).
• Send Open Brush and Open Blocks content to USD Composer (USD and glTF export).
• Test on Windows on ARM: StereoKit and Godot have arm64 builds; the Unity players run x64 under emulation.

Status shown to users is "3D active". Related: #2330 (legacy profile), Discussion #2331.

🤖 Generated with Claude Code

[joehillthunder]

Update: both runtime gaps fixed in v2.56.0, and launchers public

Both runtime gaps found in this survey are fixed in v2.56.0:

• GL depth swapchain formats (fix(gl): advertise depth swapchain formats on the native GL compositor #2429), which lets Godot 4 GL apps start;
• the hello_xr D3D12 depth view (fix(d3d): list D32_FLOAT as the preferred depth swapchain format (#2430) #2448), which closes D3D12: Khronos hello_xr loses its device on the first frame (DXGI_ERROR_INVALID_CALL) #2430.

The launchers are now public:

• Open Brush: https://github.com/joehillthunder/displayxr-openbrush
• Open Blocks: https://github.com/joehillthunder/displayxr-openblocks
• Godot 4 OpenXR exports (XR Tools demo): https://github.com/joehillthunder/displayxr-godot

Each launcher:

• finds the app;
• uses the installed runtime by default (dev builds are opt-in);
• computes the true-geometry rig from display info at every start;
• disables known-bad implicit layers for Unity players;
• docks the depth slider when the 3D is embedded in the app window.

On-panel retests on v2.56.0, plus calibration and depth results, will follow in a later comment.

🤖 Generated with Claude Code