Skip to content

A HoloLens 2 application that enables users (especially amputy people) to control the Boston Dynamics Spot robot’s movement and arm

License

Notifications You must be signed in to change notification settings

dehezhang2/holo-spot

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

41 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

HoloSpot: Accessible robot control in Mixed Reality

logo

Demo

Here is a demostration of follow mode in our project:

holospot.mp4

For other modes, please see our project website.

Overview

Course project of Mixed Reality Fall 2022 in ETHZ.

motivation

A HoloLens 2 application that enables users (especially amputy people) to control the Boston Dynamics Spot robot’s movement and arm using:

  • Eye tracking.
  • Head movements.
  • Voice control.

We implements three control modes, including follow mode (robot follow eye gaze), select mode (robot directly goes to the selected location), and arm mode (robot arm mimic the user’s head pose). Besides, we also implements user friendly interface including voice control, arm camera visualization, and a help panel to show detailed command for the users.

More details about this project can be found in our project website and report.

Authors: Ganlin Zhang *, Deheng Zhang *, Longteng Duan *, Guo Han *

(* Equally contribute to this project)

System Overview

system

Set Up

Requirements:

  • Windows for the unity application, Ubuntu 20.04 & ROS Noetic for the spot robot.
  • The Unity version is really important (recommend 2020.3.40 )

Clone this repository:

git clone https://github.com/dehezhang2/holo-spot.git

Spot Robot Connection

  1. Connecting the robot using ssh.
  2. Follow the instruction in ROS_ws.

Unity App Deployment

  1. Open this project using Unity

  2. In the unity project File > Build Settings > Universal Windows Platform, use the following settings:

    Target Device: Hololens 
    Architecture: ARM64
    

    Then click Switch Platform.

    Screen Shot 2023-01-17 at 2.40.04 PM
  3. Switch the anchor user information in Assets>Scenes>SampleScene, edit the game object AzureSpatialAnchors, fill the Spatial Anchors Account id, Spatial Anchors Account and Spatial Anchors Account Domain in Credentials.

    Screen Shot 2023-01-17 at 2.41.16 PM
  • Please don’t move the project folder once you create it!!
  • You can refer to the MRTK Tutorial to build and deploy the project.
    • Make sure the unity version is consist.
    • Connecting the Hololens2 to the USB can make the deployment faster.

Acknowledgement

We thank our supervisor Eric Vollenweider from Microsoft Mixed Reality & AI Lab Zurich for the help and tons of useful advice for this project.

We also thank Boyang Sun for the support for the usage of the Spot robot from CVG lab in ETH Zurich.

Reference

@article{zhang2023accessible,
  title={Accessible Robot Control in Mixed Reality},
  author={Zhang, Ganlin and Zhang, Deheng and Duan, Longteng and Han, Guo},
  journal={arXiv preprint arXiv:2306.02393},
  year={2023}
}

License

The code is released under the GPL-3.0 license.

About

A HoloLens 2 application that enables users (especially amputy people) to control the Boston Dynamics Spot robot’s movement and arm

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published