Shell hit-test Phase 2: move cursor per-eye depth ownership off the runtime raycast

[dfattal]

Context

displayxr-shell-pvt #40 Phase 1 ported the eye→cursor ray-vs-window hit-test into the shell for click policy (focus / drag / chrome dispatch / resize) — shipped in displayxr-shell-pvt PR #50, validated on a real Leia SR display (0/0 divergences vs the runtime's enrichment). That was self-contained controller-side work with no runtime change.

This issue tracks Phase 2 — the runtime-side change required to delete workspace_raycast_hit_test entirely.

Why the runtime still raycasts

The runtime renders the mouse cursor sprite into the stereo compositor atlas and needs hit_z_m — the ray-plane intersection depth on the window under the cursor — to compute the cursor's per-eye disparity so it floats at the same depth as what it's pointing at.

Reference (paths/lines as of writing):

• src/xrt/compositor/d3d11_service/comp_d3d11_service.cpp
  • workspace_raycast_hit_test (~L5795) — the ~340-line raycast
  • per-frame hover cache + mc->cursor_hit_z_m publish (~L6407–6478)
  • cursor sprite draw with per-eye disparity off hit_z_m (~L8205–8413)

The shell already computes hit_z_m locally (Phase 1) but has no way to feed it to the runtime cursor renderer.

Options

1. (smaller) New runtime path letting the controller push cursor screen-pos + hit_z each frame; the existing disparity math runs on the shell-supplied depth. The shell becomes the sole hit-test owner.
2. (larger) Controller submits a pre-warped per-eye stereo cursor overlay via the existing cursor swapchain (xrCreateWorkspaceCursorSwapchainEXT / xrSetWorkspaceCursorEXT, spec_version 13).

Once cursor depth is shell-driven, workspace_raycast_hit_test and its non-cursor call sites (POINTER/MOTION event enrichment, the per-frame hover cache that drives POINTER_HOVER, and the public xrWorkspaceHitTestEXT API) can be removed.

Also note

POINTER_HOVER events are themselves generated by the runtime raycast (fired on slot/region transitions). Deleting the raycast means the shell must generate its own hover transitions — which it can only do once it has a per-frame cursor-position feed (Phase 2 gives it one). So hover ownership moves with cursor-depth ownership.

Acceptance

• Cursor per-eye disparity is driven by shell-supplied (or shell-rendered) cursor depth.
• workspace_raycast_hit_test and all four call sites removed from the runtime.
• Shell owns hover-transition generation.

Related: displayxr-shell-pvt #40, PR #50.

[dfattal]

Building & running the shell locally (conflict-free with the installed bundle)

To validate Phase 2 you'll need a dev shell (the workspace controller) running against your dev runtime, without the Program Files bundle interfering. The shell builds standalone — no runtime build needed (it links only OpenXR::openxr_loader + cJSON + Win32).

1. Clone the shell as a sibling

Per the shell repo's CLAUDE.md, clone DisplayXR/displayxr-shell-pvt into the same parent folder as openxr-3d-display.

2. Build

scripts\build-shell.bat build

Output lands in _package\bin\. As of shell main @ e7331bf the script also stages openxr_loader.dll next to the exe, so _package\bin\displayxr-shell.exe runs standalone (the loader is an imported SDK target, so vcpkg's applocal step doesn't copy it). On older shell checkouts, copy it manually:
copy C:\dev\openxr_sdk\x64\bin\openxr_loader.dll _package\bin\

3. Run against your dev runtime, not Program Files

The installed displayxr-service.exe is the OS-registered OpenXR runtime and holds the SR display, so you must redirect and stop it:

1. Stop the installed service so it isn't fighting your dev runtime for the display:
   taskkill /f /im displayxr-service.exe
2. Don't run the installed shell at the same time — there's a singleton mutex Local\DisplayXR.Shell.Singleton, and at most one workspace controller can be active (xrActivateSpatialWorkspaceEXT). taskkill /f /im displayxr-shell.exe if needed.
3. Point the shell's loader at your dev runtime via the OpenXR loader override, set in the shell's own environment before launch:
   set XR_RUNTIME_JSON=<openxr-3d-display>\build\Release\openxr_displayxr-dev.json
   The shell's own connection uses the loader's ActiveRuntime registry by default (which points at Program Files) — XR_RUNTIME_JSON overrides that. The shell also forwards this to the apps it launches (get_runtime_json_path in main.c), so apps hit the same dev runtime.
4. Do not overwrite Program Files. Run from _package\bin\. (The shell repo CLAUDE.md warns that stale Program Files installs bind a ghost MCP pipe; keep the two trees separate.)

4. Cross-repo coupling — Phase 2 is NOT runtime-only

The current shell owns cursor sprite content (shell_cursor.cpp, xrCreateWorkspaceCursorSwapchainEXT / xrSetWorkspaceCursorEXT, spec_version 13) but not cursor position or depth — the runtime raycast still owns those. So to validate a runtime change that consumes shell-supplied hit_z, a shell-side producer must be written first (extend shell_cursor.cpp to push cursor pos + hit_z each frame). A runtime agent can't fully exercise Phase 2 against today's shell.

Also: any new cursor-submission entrypoint must land in DisplayXR/displayxr-extensions (canonical headers) + a spec_version bump before the shell can compile against it — the shell clones displayxr-extensions at build time.

5. Handy validation flags (from shell #40 Phase 1)

The shell can A/B its local hit-test vs the runtime's enrichment:

• DISPLAYXR_HITTEST_DEBUG=1 → logs [hittest] rt{…} shell{…} <DIVERGE> per click (stdout — redirect it, the shell is a GUI subsystem app).
• DISPLAYXR_SHELL_HITTEST=1 → shell dispatches off its own hit-test.

[dfattal]

Done — Phase 2 landed on both sides.

Runtime 9feb1a5f9 (XR_EXT_spatial_workspace spec_version 21→22):

• Deleted workspace_raycast_hit_test + struct and all four call sites (hover cache, LMB handler, POINTER/MOTION enrichment, public hit-test).
• Removed xrWorkspaceHitTestEXT; added xrSetWorkspaceCursorDepthEXT (controller pushes ray-plane depth + over-window flag each frame into the existing per-eye disparity math).
• FRAME_TICK now carries OS cursor position (cursorX/cursorY) so the controller runs its own per-frame hit-test; stopped emitting POINTER_HOVER (controller generates hover transitions).
• Follow-up f4b1656f2 (Retire residual runtime-side input policy in workspace mode (DELETE-close, Ctrl+O launch → shell, cursor dim factor) #376, spec 23) retired the residual runtime-side input policy and made over-window cursor dim controller-pushed.

Shell (displayxr-shell-pvt) 62f1d1c77 — owns hit-test + cursor depth per-frame (the matching producer).

All acceptance criteria met (cursor disparity is shell-driven; raycast + four call sites removed; shell owns hover-transition generation). Closing.