Skip to content

Latest commit

 

History

History
106 lines (78 loc) · 4.28 KB

README.md

File metadata and controls

106 lines (78 loc) · 4.28 KB
Raw

dVRK Mixed Reality Extension

This repository has code related to da Vinci Research Kit Mixed Reality extension.

Overview

Video Examples:

dVRK-XR in Unity Editor dVRK-XR on Windows Mixed Reality Headset
dVRK-XR with Android ARCore (3rd view) dVRK-XR with Google Cardboard (screen rec.)

dVRK-XR facilitates the integration of the da Vinci Research Kit (or da Vinci robot with research API) with extended reality platforms. To run dVRK-XR, you first need some knowledge about Unity development and the dVRK. I also wrote a blog post about the rationale behind the development of this project.

Structure of Project

  • dvrk_unity:
    • A unity project that contains prefabs, models and scripts of PSM, ECM and MTMs of da Vinci robot.
    • The virtual robots can be controlled by UDP messages that specify joint values.
    • The scene can be adapted and deployed to various mixed reality platforms.
  • sample_server:
    • A python-based sample server that sends UDP messages to the mixed reality platform.
    • Format of the JSON messages, for debugging use

Integration with dVRK

Dependencies

How to Run

  • sawIntuitiveResearchKit needs to run with sawSocketStreamer. This can be done via a component configuration file with -m option, e.g.
# Navigate to share folder
cd sawIntuitiveResearchKit/share

# Run dVRK console with a configuration file for socket streamer. 
# This example streams the MTML and MTMR joint states to remote UDP client. 
# The client IP address, UDP port number, message rate, message to send, can be specified in the \
# configuration file. Make sure the IP and port match that on the mixed reality platform. 
sawIntuitiveResearchKitQtConsoleJSON -j CONSOLE_CONFIG -m socket-streamer/manager-socket-streamer-console.json
  • Multiple component configuration files can be attached to the main program, e.g.
# This example streams both the surgeon console and patient-side arms to the remote UDP client
sawIntuitiveResearchKitQtConsoleJSON -j CONSOLE_CONFIG -m socket-streamer/manager-socket-streamer-console.json -m socket-streamer/manager-socket-streamer-patient-cart.json
  • Different configuration files can also specify different destinations (IP address), supporting multiple headsets viewing at the virtual robot at the same time.

Integration with da Vinci Robot

Dependencies

  • da Vinci robot (Classic, S or Si) with research API enabled
  • sawIntuitiveDaVinci 1.2.0 or higher
  • sawSocketStreamer
  • dvrk-xr running on a mixed reality platform
    • Unity Editor, Standalone build (Windows, Ubuntu), HoloLens, or other platforms

How to Run

Same as dVRK, but main program is sawIntuitiveDaVinciConsoleQt.

Academic Citation

@inproceedings{qian2019dvrkxr,
  title={{dVRK-XR: Mixed Reality Extension for da Vinci Research Kit}},
  author={Qian, Long and Deguet, Anton and Kazanzides, Peter},
  booktitle={Hamlyn Symposium on Medical Robotics},
  year={2019}
}

Acknowledgment

dvrk-xr was developed with support from NSF NRI 1637789.

License

JHU CISST Software License