Skip to content

OpenXR compact binding for creating extended reality applications on NVIDIA Omniverse

License

Notifications You must be signed in to change notification settings

Toni-SM/semu.xr.openxr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OpenXR compact binding for creating extended reality applications on NVIDIA Omniverse

This extension provides a compact python binding (on top of the open standard OpenXR for augmented reality (AR) and virtual reality (VR)) to create extended reality applications taking advantage of NVIDIA Omniverse rendering capabilities. In addition to updating views (e.g., head-mounted display), it enables subscription to any input event (e.g., controller buttons and triggers) and execution of output actions (e.g., haptic vibration) through a simple and efficient API for accessing conformant devices such as HTC Vive, Oculus and others...


Target applications: Any NVIDIA Omniverse app with the omni.syntheticdata extension installed (e.g., Isaac Sim, Code, etc.)

  • Tested in Ubuntu 18.04/20.04, STEAMVR beta 1.24.3, Omniverse Code 2022.1.3 and Isaac Sim 2022.1.1

Supported OS: Linux

Changelog: CHANGELOG.md

Table of Contents:


showcase


Extension setup

  1. Add the extension using the Extension Manager or by following the steps in Extension Search Paths

    • Git url (git+https) as extension search path

      git+https://github.com/Toni-SM/semu.xr.openxr.git?branch=main&dir=exts
      
    • Compressed (.zip) file for import

      semu.xr.openxr.zip

  2. Enable the extension using the Extension Manager or by following the steps in Extension Enabling/Disabling

  3. Import the extension into any python code and use it...

    from semu.xr.openxr import _openxr
  4. Or use the GUI launcher to directly dislpay the current stage in the HMD


Diagrams

High-level overview of extension usage, including the order of function calls, callbacks and the action and rendering loop

Typical OpenXR application showing the grouping of the standard functions under the compact binding provided by the extension (adapted from openxr-10-reference-guide.pdf)

openxr-application


Sample code

The following sample code shows a typical workflow that configures and renders on a stereo headset the view generated in an Omniverse application. It configures and subscribes two input actions to the left controller to 1) mirror on a simulated sphere the pose of the controller and 2) change the dimensions of the sphere based on the position of the trigger. In addition, an output action, a haptic vibration, is configured and executed when the controller trigger reaches its maximum position

A short video, after the code, shows a test of the OpenXR application from the Script Editor using an HTC Vive Pro

import omni
from pxr import UsdGeom
from semu.xr.openxr import _openxr

# get stage unit
stage = omni.usd.get_context().get_stage()
meters_per_unit = UsdGeom.GetStageMetersPerUnit(stage)

# create a sphere (1 centimeter radius) to mirror the controller's pose
sphere_prim = omni.usd.get_context().get_stage().DefinePrim("/sphere", "Sphere")
sphere_prim.GetAttribute("radius").Set(0.01 / meters_per_unit)

# acquire interface
xr = _openxr.acquire_openxr_interface()

# setup OpenXR application using default parameters
xr.init()
xr.create_instance()
xr.get_system()

# action callback
def on_action_event(path, value):
    # process controller's trigger
    if path == "/user/hand/left/input/trigger/value":
      # modify the sphere's radius (from 1 to 10 centimeters) according to the controller's trigger position
      sphere_prim.GetAttribute("radius").Set((value * 9 + 1) * 0.01 / meters_per_unit)
      # apply haptic vibration when the controller's trigger is fully depressed
      if value == 1:
        xr.apply_haptic_feedback("/user/hand/left/output/haptic", {"duration": _openxr.XR_MIN_HAPTIC_DURATION})
    # mirror the controller's pose on the sphere (cartesian position and rotation as quaternion)
    elif path == "/user/hand/left/input/grip/pose":
        xr.teleport_prim(sphere_prim, value[0], value[1])

# subscribe controller actions (haptic actions don't require callbacks) 
xr.subscribe_action_event("/user/hand/left/input/grip/pose", callback=on_action_event, reference_space=_openxr.XR_REFERENCE_SPACE_TYPE_LOCAL)
xr.subscribe_action_event("/user/hand/left/input/trigger/value", callback=on_action_event)
xr.subscribe_action_event("/user/hand/left/output/haptic")

# create session and define interaction profiles
xr.create_session()

# setup cameras and viewports and prepare rendering using the internal callback
xr.set_meters_per_unit(meters_per_unit)
xr.setup_stereo_view()
xr.set_frame_transformations(flip=0)
xr.set_stereo_rectification(y=0.05)

# execute action and rendering loop on each simulation step
def on_simulation_step(step):
    if xr.poll_events() and xr.is_session_running():
        xr.poll_actions()
        xr.render_views(_openxr.XR_REFERENCE_SPACE_TYPE_LOCAL)

physx_subs = omni.physx.get_physx_interface().subscribe_physics_step_events(on_simulation_step)
snippet.mp4

GUI launcher

The extension also provides a graphical user interface that helps to launch a partially configurable OpenXR application form a window. This interface is located in the Add-ons > OpenXR UI menu

The first four options (Graphics API, Form factor, Blend mode, View configuration type) cannot be modified once the OpenXR application is running. They are used to create and configure the OpenXR instance, system and session

The other options (under the central separator) can be modified while the application is running. They help to modify the pose of the reference system, or to perform transformations on the images to be rendered, for example.


Extension API

Acquiring extension interface

  • Acquire OpenXR interface

    _openxr.acquire_openxr_interface() -> semu::xr::openxr::OpenXR
  • Release OpenXR interface

    _openxr.release_openxr_interface(xr: semu::xr::openxr::OpenXR) -> None

API

The following functions are provided on the OpenXR interface:

  • Init OpenXR application by loading the necessary libraries

    init(graphics: str = "OpenGL", use_ctypes: bool = False) -> bool

    Parameters:

    • graphics: str

      OpenXR graphics API supported by the runtime (OpenGL, OpenGLES, Vulkan, D3D11, D3D12). Note: At the moment only OpenGL is available

    • use_ctypes: bool, optional

      If true, use ctypes as C/C++ interface instead of pybind11 (default)

    Returns:

    • bool

      True if initialization was successful, otherwise False

  • Get OpenXR session's running status

    is_session_running() -> bool

    Returns:

    • bool

      Return True if the OpenXR session is running, False otherwise

  • Create an OpenXR instance to allow communication with an OpenXR runtime

    create_instance(application_name: str = "Omniverse (XR)", engine_name: str = "", api_layers: list = [], extensions: list = []) -> bool

    Parameters:

    • application_name: str, optional

      Name of the OpenXR application (default: Omniverse (XR))

    • engine_name: str, optional

      Name of the engine (if any) used to create the application (empty by default)

    • api_layers: list of str, optional

      API layers to be inserted between the OpenXR application and the runtime implementation

    • extensions: list of str, optional

      Extensions to be loaded. Note: The graphics API selected during initialization (init) is automatically included in the extensions to be loaded. At the moment only the graphic extensions are configured

    Returns:

    • bool

      True if the instance has been created successfully, otherwise False

  • Obtain the system represented by a collection of related devices at runtime

    get_system(form_factor: int = XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY, blend_mode: int = XR_ENVIRONMENT_BLEND_MODE_OPAQUE, view_configuration_type: int = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO) -> bool

    Parameters:

    • form_factor: {XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY, XR_FORM_FACTOR_HANDHELD_DISPLAY}, optional

      Desired form factor from XrFormFactor enum (default: XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY)

    • blend_mode: {XR_ENVIRONMENT_BLEND_MODE_OPAQUE, XR_ENVIRONMENT_BLEND_MODE_ADDITIVE, XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND}, optional

      Desired environment blend mode from XrEnvironmentBlendMode enum (default: XR_ENVIRONMENT_BLEND_MODE_OPAQUE)

    • view_configuration_type: {XR_VIEW_CONFIGURATION_TYPE_PRIMARY_MONO, XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO}, optional

      Primary view configuration type from XrViewConfigurationType enum (default: XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO)

    Returns:

    • bool

      True if the system has been obtained successfully, otherwise False

  • Create an OpenXR session that represents an application's intention to display XR content

    create_session() -> bool

    Returns:

    • bool

      True if the session has been created successfully, otherwise False

  • Event polling and processing

    poll_events() -> bool

    Returns:

    • bool

      False if the running session needs to end (due to the user closing or switching the application, etc.), otherwise False

  • Action polling

    poll_actions() -> bool

    Returns:

    • bool

      True if there is no error during polling, otherwise False

  • Present rendered images to the user's views according to the selected reference space

    render_views(reference_space: int = XR_REFERENCE_SPACE_TYPE_LOCAL) -> bool

    Parameters:

    • reference_space: {XR_REFERENCE_SPACE_TYPE_VIEW, XR_REFERENCE_SPACE_TYPE_LOCAL, XR_REFERENCE_SPACE_TYPE_STAGE}, optional

      Desired reference space type from XrReferenceSpaceType enum used to render the images (default: XR_REFERENCE_SPACE_TYPE_LOCAL)

    Returns:

    • bool

      True if there is no error during rendering, otherwise False

  • Create an action given a path and subscribe a callback function to the update event of this action

    subscribe_action_event(path: str, callback: Union[Callable[[str, object], None], None] = None, action_type: Union[int, None] = None, reference_space: Union[int, None] = XR_REFERENCE_SPACE_TYPE_LOCAL) -> bool

    If action_type is None the action type will be automatically defined by parsing the last segment of the path according to the following policy:

    Action type (XrActionType) Last segment of the path
    XR_ACTION_TYPE_BOOLEAN_INPUT /click, /touch
    XR_ACTION_TYPE_FLOAT_INPUT /value, /force
    XR_ACTION_TYPE_VECTOR2F_INPUT /x, /y
    XR_ACTION_TYPE_POSE_INPUT /pose
    XR_ACTION_TYPE_VIBRATION_OUTPUT /haptic, /haptic_left, /haptic_right, /haptic_left_trigger, /haptic_right_trigger

    The callback function (a callable object) should have only the following 2 parameters:

    • path: str

      The complete path (user path and subpath) of the action that invokes the callback

    • value: bool, float, tuple(float, float), tuple(pxr.Gf.Vec3d, pxr.Gf.Quatd)

      The current state of the action according to its type

      Action type (XrActionType) python type
      XR_ACTION_TYPE_BOOLEAN_INPUT bool
      XR_ACTION_TYPE_FLOAT_INPUT float
      XR_ACTION_TYPE_VECTOR2F_INPUT (x, y) tuple(float, float)
      XR_ACTION_TYPE_POSE_INPUT (position (in stage unit), rotation as quaternion) tuple(pxr.Gf.Vec3d, pxr.Gf.Quatd)

    XR_ACTION_TYPE_VIBRATION_OUTPUT actions will not invoke their callback function. In this case the callback must be None

    XR_ACTION_TYPE_POSE_INPUT also specifies, through the definition of the reference_space parameter, the reference space used to retrieve the pose

    The collection of available paths corresponds to the following interaction profiles:

    Parameters:

    • path: str

      Complete path (user path and subpath) referring to the action

    • callback: callable object (2 parameters) or None for XR_ACTION_TYPE_VIBRATION_OUTPUT

      Callback invoked when the state of the action changes

    • action_type: {XR_ACTION_TYPE_BOOLEAN_INPUT, XR_ACTION_TYPE_FLOAT_INPUT, XR_ACTION_TYPE_VECTOR2F_INPUT, XR_ACTION_TYPE_POSE_INPUT, XR_ACTION_TYPE_VIBRATION_OUTPUT} or None, optional

      Action type from XrActionType enum (default: None)

    • reference_space: {XR_REFERENCE_SPACE_TYPE_VIEW, XR_REFERENCE_SPACE_TYPE_LOCAL, XR_REFERENCE_SPACE_TYPE_STAGE}, optional

      Desired reference space type from XrReferenceSpaceType enum used to retrieve the pose (default: XR_REFERENCE_SPACE_TYPE_LOCAL)

    Returns

    • bool

      True if there is no error during action creation, otherwise False

  • Apply a haptic feedback to a device defined by a path (user path and subpath)

    apply_haptic_feedback(path: str, haptic_feedback: dict) -> bool

    Parameters:

    • path: str

      Complete path (user path and subpath) referring to the action

    • haptic_feedback: dict

      A python dictionary containing the field names and value of a XrHapticBaseHeader-based structure. Note: At the moment the only haptics type supported is the unextended OpenXR XrHapticVibration

    Returns:

    • bool

      True if there is no error during the haptic feedback application, otherwise False

  • Stop a haptic feedback applied to a device defined by a path (user path and subpath)

    stop_haptic_feedback(path: str) -> bool

    Parameters:

    • path: str

      Complete path (user path and subpath) referring to the action

    Returns:

    • bool

      True if there is no error during the haptic feedback stop, otherwise False

  • Setup Omniverse viewport and camera for monoscopic rendering

    setup_mono_view(camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim] = "/OpenXR/Cameras/camera", camera_properties: dict = {"focalLength": 10}) -> None

    This method obtains the viewport window for the given camera. If the viewport window does not exist, a new one is created and the camera is set as active. If the given camera does not exist, a new camera is created with the same path and set to the recommended resolution of the display device

    Parameters:

    • camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional

      Omniverse camera prim or path (default: /OpenXR/Cameras/camera)

    • camera_properties: dict

      Dictionary containing the camera properties supported by the Omniverse kit to be set (default: {"focalLength": 10})

  • Setup Omniverse viewports and cameras for stereoscopic rendering

    setup_stereo_view(left_camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim] = "/OpenXR/Cameras/left_camera", right_camera: Union[str, pxr.Sdf.Path, pxr.Usd.Prim, None] = "/OpenXR/Cameras/right_camera", camera_properties: dict = {"focalLength": 10}) -> None

    This method obtains the viewport window for each camera. If the viewport window does not exist, a new one is created and the camera is set as active. If the given cameras do not exist, new cameras are created with the same path and set to the recommended resolution of the display device

    Parameters:

    • left_camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional

      Omniverse left camera prim or path (default: /OpenXR/Cameras/left_camera)

    • right_camera: str, pxr.Sdf.Path or pxr.Usd.Prim, optional

      Omniverse right camera prim or path (default: /OpenXR/Cameras/right_camera)

    • camera_properties: dict

      Dictionary containing the camera properties supported by the Omniverse kit to be set (default: {"focalLength": 10})

  • Get the recommended resolution of the display device

    get_recommended_resolutions() -> tuple

    Returns:

    • tuple

      Tuple containing the recommended resolutions (width, height) of each device view. If the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye

  • Set the pose of the origin of the reference system

    set_reference_system_pose(position: Union[pxr.Gf.Vec3d, None] = None, rotation: Union[pxr.Gf.Vec3d, None] = None) -> None

    Parameters:

    • position: pxr.Gf.Vec3d or None, optional

      Cartesian position (in stage unit) (default: None)

    • rotation: pxr.Gf.Vec3d or None, optional

      Rotation (in degress) on each axis (default: None)

  • Set the angle (in radians) of the rotation axes for stereoscopic view rectification

    set_stereo_rectification(x: float = 0, y: float = 0, z: float = 0) -> None

    Parameters:

    • x: float, optional

      Angle (in radians) of the X-axis (default: 0)

    • y: float, optional

      Angle (in radians) of the Y-axis (default: 0)

    • x: float, optional

      Angle (in radians) of the Z-axis (default: 0)

  • Specify the meters per unit to be applied to transformations (default: 1.0)

    set_meters_per_unit(meters_per_unit: float) -> None

    Parameters:

    • meters_per_unit: float

      Meters per unit. E.g.: 1 meter is 1.0, 1 centimeter is 0.01

  • Specify the transformations to be applied to the rendered images

    set_frame_transformations(fit: bool = False, flip: Union[int, tuple, None] = None) -> None

    Parameters:

    • fit: bool, optionl

      Adjust each rendered image to the recommended resolution of the display device by cropping and scaling the image from its center (default: False). OpenCV resize method with INTER_LINEAR interpolation will be used to scale the image to the recommended resolution

    • flip: int, tuple or None, optionl

      Flip each image around vertical (0), horizontal (1), or both axes (0,1) (default: None)

  • Teleport the prim specified by the given transformation (position and rotation)

    teleport_prim(prim: pxr.Usd.Prim, position: pxr.Gf.Vec3d, rotation: pxr.Gf.Quatd, reference_position: Union[pxr.Gf.Vec3d, None] = None, reference_rotation: Union[pxr.Gf.Vec3d, None] = None) -> None

    Parameters:

    • prim: pxr.Usd.Prim

      Target prim

    • position: pxr.Gf.Vec3d

      Cartesian position (in stage unit) used to transform the prim

    • rotation: pxr.Gf.Quatd

      Rotation (as quaternion) used to transform the prim

    • reference_position: pxr.Gf.Vec3d or None, optional

      Cartesian position (in stage unit) used as reference system (default: None)

    • reference_rotation: pxr.Gf.Vec3d or None, optional

      Rotation (in degress) on each axis used as reference system (default: None)

  • Subscribe a callback function to the render event

    subscribe_render_event(callback=None) -> None

    The callback function (a callable object) should have only the following 3 parameters:

    • num_views: int

      The number of views to render: mono (1), stereo (2)

    • views: tuple of XrView structure

      A XrView structure contains the view pose and projection state necessary to render a image. The length of the tuple corresponds to the number of views (if the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye)

    • configuration_views: tuple of XrViewConfigurationView structure

      A XrViewConfigurationView structure specifies properties related to rendering of a view (e.g. the optimal width and height to be used when rendering the view). The length of the tuple corresponds to the number of views (if the tuple length is 2, index 0 represents the left eye and index 1 represents the right eye)

      The callback function must call the set_frames function to pass to the selected graphics API the image or images to be rendered

      If the callback is None, an internal callback will be used to render the views. This internal callback updates the pose of the cameras according to the specified reference system, gets the images from the previously configured viewports and invokes the set_frames function to render the views

    Parameters:

    • callback: callable object (3 parameters) or None, optional

      Callback invoked on each render event (default: None)

  • Pass to the selected graphics API the images to be rendered in the views

    set_frames(configuration_views: list, left: numpy.ndarray, right: numpy.ndarray = None) -> bool

    In the case of stereoscopic devices, the parameters left and right represent the left eye and right eye respectively. To pass an image to the graphic API of monoscopic devices only the parameter left should be used (the parameter right must be None)

    This function will apply to each image the transformations defined by the set_frame_transformations function if they were specified

    Parameters:

    • configuration_views: tuple of XrViewConfigurationView structure

      A XrViewConfigurationView structure specifies properties related to rendering of a view (e.g. the optimal width and height to be used when rendering the view)

    • left: numpy.ndarray

      RGB or RGBA image (numpy.uint8)

    • right: numpy.ndarray or None

      RGB or RGBA image (numpy.uint8)

    Returns:

    • bool

      True if there is no error during the passing to the selected graphics API, otherwise False

Available enumerations

  • Form factors supported by OpenXR runtimes (XrFormFactor)

    • XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY = 1

    • XR_FORM_FACTOR_HANDHELD_DISPLAY = 2

  • Environment blend mode (XrEnvironmentBlendMode)

    • XR_ENVIRONMENT_BLEND_MODE_OPAQUE = 1

    • XR_ENVIRONMENT_BLEND_MODE_ADDITIVE = 2

    • XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND = 3

  • Primary view configuration type (XrViewConfigurationType)

    • XR_VIEW_CONFIGURATION_TYPE_PRIMARY_MONO = 1

    • XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO = 2

  • Reference spaces (XrReferenceSpaceType)

    • XR_REFERENCE_SPACE_TYPE_VIEW = 1

    • XR_REFERENCE_SPACE_TYPE_LOCAL = 2

    • XR_REFERENCE_SPACE_TYPE_STAGE = 3

  • Action type (XrActionType)

    • XR_ACTION_TYPE_BOOLEAN_INPUT = 1

    • XR_ACTION_TYPE_FLOAT_INPUT = 2

    • XR_ACTION_TYPE_VECTOR2F_INPUT = 3

    • XR_ACTION_TYPE_POSE_INPUT = 4

    • XR_ACTION_TYPE_VIBRATION_OUTPUT = 100

Available constants

  • Graphics API extension names

    • XR_KHR_OPENGL_ENABLE_EXTENSION_NAME = "XR_KHR_opengl_enable"

    • XR_KHR_OPENGL_ES_ENABLE_EXTENSION_NAME = "XR_KHR_opengl_es_enable"

    • XR_KHR_VULKAN_ENABLE_EXTENSION_NAME = "XR_KHR_vulkan_enable"

    • XR_KHR_D3D11_ENABLE_EXTENSION_NAME = "XR_KHR_D3D11_enable"

    • XR_KHR_D3D12_ENABLE_EXTENSION_NAME = "XR_KHR_D3D12_enable"

  • Useful constants for applying haptic feedback

    • XR_NO_DURATION = 0

    • XR_INFINITE_DURATION = 0x7fffffffffffffff

    • XR_MIN_HAPTIC_DURATION = -1

    • XR_FREQUENCY_UNSPECIFIED = 0