OpenXR: Meta passthrough, environment depth occlusion, and frame state refactor

[Sunderlandkyl]

ENH: Add Meta passthrough and environment depth occlusion support

Adds support for XR_FB_passthrough and XR_META_environment_depth (requires OpenXR SDK 1.1.36).

New API on vtkOpenXRRenderWindow:

• UsePassthrough / PassthroughOn / PassthroughOff — request mixed reality via standard alpha-blend or Meta-specific composition layer
• OccludedOpacity [0,1] — control real-world occlusion strength via depth pre-pass
• ShowEnvDepthDebugVisualization — false-colour depth overlay
• CaptureEnvironmentDepth — force depth capture even at full opacity
• IsPassthroughActive() — query live passthrough state

New class vtkOpenXREnvironmentDepthOcclusionPrePass: fullscreen-triangle pass that reads the META environment depth swapchain and composites real-world occlusion or a debug overlay onto each eye's frame.

New API on vtkOpenXRManager: StartFBPassthrough, StopFBPassthrough, StartEnvironmentDepth, StopEnvironmentDepth, AcquireEnvironmentDepthImage, and related query/accessor methods.

---

ENH: Streamline frame state management into a single pathway

Consolidates per-frame XR state into a single vtkOpenXRFrameContext struct, removing now-redundant standalone members.

Processing loop fixes:

• WaitAndBeginFrame() called at the start of DoOneEvent() so PredictedDisplayTime is valid before PollXrActions()
• SyncActions() moved to just before LocateViews() to ensure action data is always synced before poses are located
• FrameBegan guard prevents double xrWaitFrame/xrBeginFrame calls
• LocateViews() extracted from WaitAndBeginFrame() for clarity

Other fixes:

• Guard hand-pose updates with XR_SPACE_LOCATION_POSITION_VALID_BIT
• Cache last valid view state and fall back on tracking loss (avoids black frames on startup)
• Defer pose-space creation to CreateActionPoseSpaces() after AttachSessionActionSets()
• Fix memory leak: missing delete[] for controller model buffer

[lassoan]

Thanks a lot @Sunderlandkyl, the changes look good at a high level, I'll build and test locally and review it more.

In the meantime, could you please also submit these changes to https://gitlab.kitware.com/vtk/vtk? We don't need to wait for the upstream pull requests to be merged to get these integrated in this repository but in the commit comments we'll need to refer to the upstream pull request(s).

[Copilot]

Pull request overview

This PR enhances VTK’s OpenXR backend by adding Meta passthrough (XR_FB_passthrough) and environment depth occlusion (XR_META_environment_depth) support, and by refactoring the per-frame OpenXR state flow to be managed through a unified frame context.

Changes:

• Add Meta passthrough and META environment-depth acquisition/compositing pathways, including a new GL fullscreen-pass helper for occlusion/debug overlays.
• Refactor frame timing/view state handling into vtkOpenXRFrameContext, moving frame begin/end responsibilities into a single per-frame pathway.
• Bump the required OpenXR SDK version to 1.1.36 and register new extension function dispatch entries.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
Rendering/OpenXR/XrExtensions.h	Registers new extension dispatch-table entries for XR_FB_passthrough and XR_META_environment_depth.
Rendering/OpenXR/vtkOpenXRRenderWindowInteractor.cxx	Moves frame begin/sync/locate/end into the interactor’s per-event loop and adds pose-validity guarding.
Rendering/OpenXR/vtkOpenXRRenderWindow.h	Adds new public API knobs for passthrough and environment-depth occlusion/debugging/capture.
Rendering/OpenXR/vtkOpenXRRenderWindow.cxx	Integrates passthrough startup, updates frame finalization ordering, and hooks environment-depth post-pass into stereo callbacks.
Rendering/OpenXR/vtkOpenXRManager.h	Introduces vtkOpenXRFrameContext and adds FB passthrough + environment depth APIs/accessors and blend-mode preference controls.
Rendering/OpenXR/vtkOpenXRManager.cxx	Implements FB passthrough + environment depth lifecycle, view-location caching, and frame-loop refactor.
Rendering/OpenXR/vtkOpenXREnvironmentDepthOcclusionPrePass.h	Adds new helper class interface for environment-depth occlusion and debug visualization passes.
Rendering/OpenXR/vtkOpenXREnvironmentDepthOcclusionPrePass.cxx	Implements GLSL programs and GL state management for occlusion/debug post-passes.
Rendering/OpenXR/CMakeLists.txt	Requires OpenXR 1.1.36 and adds the new pre-pass class to the module build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lassoan]

@Sunderlandkyl this will be great, but some of the Copilot findings seem to be valid. Please review them and update the code if needed. Thank you!

[Sunderlandkyl]

Thanks @lassoan, I will take a look at them today.

[Sunderlandkyl]

I added the fixes from the review. For some reason when passthrough is enabled, only the back-faces of the mesh seem to be visible. I'm looking into it.

[Sunderlandkyl]

Some more context with the issue:

• If rendering is enabled with occlusion occluded opacity 1.0, then the front-faces are rendered correctly, however the slider doesn't change the occluded opacity anymore.
• If rendering is enabled with occluded opacity < 1.0, then only the back-faces are visible, and the slider works as expected.

[lassoan]

@Sunderlandkyl is this ready to be merged?

[Sunderlandkyl]

Couple of things.

1. I haven't been able to fix the occlusion issue yet. If we don't enable occlusion from the GUI then pass-through still works as intended.
2. I created a build on a different computer (laptop) and ran into a crash when enabling VR. This seems to happen with the latest nightly build as well however so it doesn't seem to be connected.

[Sunderlandkyl]

@lassoan To actually answer your question, yes it can be merged on the understanding that the environmental occlusion support will need to be fixed later.

Should this PR be moved on a new branch, or integrated onto this one?

[lassoan]

It looks good to me, AR works well on Meta Quest 3 (except depth-based occlusion), so this is ready to go. Please also submit these commits to VTK upstream, too.