Doc Request: Add FAQ and/or Roadmap

[tbirdso]

As someone who is interested in Slicer's extended support for XR, it would be helpful to have a high-level "FAQ" section or roadmap of features intended for this module.

Questions to Answer

• Is SlicerMixedReality intended as the OpenXR equivalent of https://github.com/KitwareMedical/SlicerVirtualReality, which is for OpenVR? Is it anticipated that SlicerMixedReality will eventually reach feature parity and replace SlicerVirtualReality as the industry moves towards OpenXR?
• What devices are planned for support by SlicerMixedReality? Is it planned for all devices covered by the VTK::RenderingOpenXR module to eventually be available through SlicerMixedReality, or is only HoloLens remoting support through VTK::RenderingOpenXRRemoting planned?
• To what degree will OpenXR features be leveraged? Will both VR and passthrough AR be available on capable devices such as the Meta Quest headsets? Will AR remoting be available for devices in addition to the Microsoft HoloLens? What input schemes will be supported?
• Are there other planned features that would fit into the scope of this module, such as collaborative XR viewing?

To be clear, I am not necessarily asking for concrete answers or technical details. Any discussion that module developers could contribute to clarify the planned scope of this module and where development may go would be appreciated.

References

To aid in discussion, we have recently expanded documentation for VTK supported XR devices and input mechanisms here:
https://gitlab.kitware.com/vtk/vtk/-/tree/master/Rendering/OpenXR#vtkrenderingopenxr
https://gitlab.kitware.com/vtk/vtk/-/tree/master/Rendering/OpenXRRemoting#vtkrenderingopenxrremoting
https://gitlab.kitware.com/vtk/vtk/-/tree/master/Rendering/OpenVR#vtkrenderingopenvr

[LucasGandel]

@jcfr is probably the right person to confirm the roadmap, but here is my understanding FWIW

1. As the industry moves towards OpenXR, SlicerMixedReality aims to be the OpenXR equivalent of SlicerVirtualReality for VR headsets. But it also allows for leveraging AR, which is a huge potential improvement.
2. So far Hololens2 (hence OpenXRRemoting) is the primary target (see WIP), but adding "plain" OpenXR support is required to fulfil requirement 1. (I am not sure something is planned in this direction, but I think this must definitely be considered)
3. Supporting all devices and feature provided by VTK::RenderingOpenXR would be a good start.
4. Not that I am aware of for now, but the long term would be to have SlicerMixedReality being the standard for medical image processing/visualization in XR, IGT...

Thank you again for the great work around the VTK OpenXR modules documentation! 🥇 🎉 🚀

[jcfr]

Is SlicerMixedReality intended as the OpenXR equivalent of https://github.com/KitwareMedical/SlicerVirtualReality, which is for OpenVR? Is it anticipated that SlicerMixedReality will eventually reach feature parity and replace SlicerVirtualReality as the industry moves towards OpenXR?

Current plan is to refactor SlicerVitualReality to support both OpenVR and OpenXR and provide components re-usable in the SlicerMixedReality extension.

This means that the SlicerMixedReality will depend on SlicerVitualReality:

• Refactoring to support OpenVR and OpenXR, and streamline reuse in SlicerMixedReality  #121

---

What devices are planned for support by SlicerMixedReality? Is it planned for all devices covered by the VTK::RenderingOpenXR module to eventually be available through SlicerMixedReality, or is only HoloLens remoting support through VTK::RenderingOpenXRRemoting planned?

The virtual reality devices integrating with OpenXR will be supported through the SlicerVirtualReality extension (itself depending on vtkRenderingOpenXR), the SlicerMixedReality will support devices based on the VTK support (currently provided through vtkRenderingOpenXRRemoting).

---

To what degree will OpenXR features be leveraged? Will both VR and passthrough AR be available on capable devices such as the Meta Quest headsets? Will AR remoting be available for devices in addition to the Microsoft HoloLens? What input schemes will be supported?

See @LucasGandel comment in #127

---

Are there other planned features that would fit into the scope of this module, such as collaborative XR viewing?

Similarly to collaborative VR ¹, adding support for collaborative XR is indeed a sensible path forward.

Footnotes

1. https://www.youtube.com/watch?v=rG9ST6xv6vg ↩

[tbirdso]

The virtual reality devices integrating with OpenXR will be supported through the SlicerVirtualReality extension, the SlicerMixedReality will support devices based on the VTK support.

@jcfr Would you please clarify this point? As it is written this seems to imply that SlicerVirtualReality will not depend on VTK, is that correct?

Maybe it is too early now, but eventually it would be extremely helpful to have a README section explicitly stating what features are supported in SlicerMixedReality that are not supported or planned for support in SlicerVirtualReality.

[tbirdso]

Current plan is to refactor SlicerVirtualReality to support both OpenVR and OpenXR and provide components re-usable in the SlicerMixedReality extension.

IMO under the current naming scheme it will be unclear to developers which module to start from or install, especially for developers who are familiar with "OpenXR" or "mixed reality" as the newer standard. It is understandable that the scope of this module may have changed since its inception. I suggest considering other names for this module to clarify that it will contain extended feature support for the "SlicerVirtualReality" module. "SlicerAdvancedVirtualReality" may be sufficient.

[jcfr]

The virtual reality devices integrating with OpenXR will be supported through the SlicerVirtualReality extension, the SlicerMixedReality will support devices based on the VTK support.

Would you please clarify this point? As it is written this seems to imply that SlicerVirtualReality will not depend on VTK, is that correct?

The comment #127 has been updated to explicitly mention vtkRenderingOpenXR and vtkRenderingOpenXRRemoting

---

IMO under the current naming scheme it will be unclear to developers which module to start from or install

Users usually discover extensions (and associated modules) though the extension manager and descriptions reported there should help clarify.

SlicerVirtualReality, SlicerAdvancedVirtualReality and SlicerMixedReality

Once we have functional integration, we could revisit the split to reduce confusion. In the meantime, I suggest we discuss proposal described in:

• Refactoring to support OpenVR and OpenXR, and streamline reuse in SlicerMixedReality  #121

Maybe it is too early now, but eventually it would be extremely helpful to have a README section explicitly stating what features are supported in SlicerMixedReality that are not supported or planned for support in SlicerVirtualReality.

Ditto. A README similar to the one added the the SlicerVirtualReality extension¹ will be added.

In the meantime, I envision that the SlicerMixedReality extension will provide a module allowing to integrate with the mixed reality headset currently supported by VTK through vtkRenderingOpenXRRemoting.

Footnotes

1. https://github.com/KitwareMedical/SlicerVirtualReality#slicervirtualreality ↩

[tbirdso]

Thanks @jcfr . Responding to the edits in #127:

The virtual reality devices integrating with OpenXR will be supported through the SlicerVirtualReality extension (itself depending on vtkRenderingOpenXR), the SlicerMixedReality will support devices based on the VTK support (currently provided through vtkRenderingOpenXRRemoting).

In the meantime, I envision that the SlicerMixedReality extension will provide a module allowing to integrate with the mixed reality headset currently supported by VTK through vtkRenderingOpenXRRemoting.

I suggest renaming this module to SlicerVirtualRealityRemoting if that is the differentiating functionality.