MRCozmo

This project explore to operate the Anki Cozmo robot in mixed reality (MR) and virtual reality (VR) using OpenXR supported head mounted display (HMD) such as Microsoft Hololens and Oculus Quest 2.

This document demonstrates some examples and record the future development steps.

[! Please note that this repository is under continuous development and improvement. Only the functionalities in the CozmoMRTest scene is working. But please feel free to explore other work in progress code base as well!]

Table of Contents

---

Application Demos

System architecture

Demos

---

Raycast way point selection and path following

---

MR Robot Path Planning using Oculus Quest 2 Passthrough API

---

Free hand path planning and drawing in VR:

---

Free Hand Lift Control Using Hololens One Hand Ruler

In simulation in Unity editor or in VR:

---

Demo of live control of Cozmo via Hololens 2:

---

Lift Control with VR controller

---

Turning head angle with VR headset

---

Live 2D video streaming

---

System Architecture

---

The robot model was imported to Unity using URDF Importer

The communcication between ROS and HMD is built on top of the Unity ROS TCP Connector

---

Learn more about this project from our paper at the 5th International Workshop on Virtual, Augmented, and Mixed Reality for HRI ( VAM-HRI 2022): https://openreview.net/pdf?id=HYIes841hJc