FetchRelaxedIKHoloLens2

This is a Unity project I created to show how you can control a robot arm in Unity using inverse kinematics calculations from ROS. Specifically, I use relaxedIK to compute the joint positions given an end-effector position, and send that to an AR visualization on the HoloLens 2.

Note: If you compile relaxedIK using windows, it will output a dll that you can use in Unity rather than sending and receiving information using ROS. The performance is much better. However, I was unable compile the repository. It should work with most other robots that they already compiled/built it for in their Unity repository. Setup will probably be slightly different.

Requirements

• Unity 2020.3.38f1
• MRTK v2.8.3 with OpenXR, Examples, Extensions, Foundation, Standard Assets, Test Utilities, and Tools
• ROS TCP Connector v0.7.0
• URDF Importer v0.5.2
• ROS TCP Endpoint
• Holographic remoting on the HoloLens 2
• ROS Noetic
• relaxedIK

Setup

RelaxedIK

Create a new ROS workspace folder.

mkdir relaxed_ik
cd relaxed_ik/

Clone the following repositories into a src directory

git clone --recuse-submodules https://github.com/uwgraphics/relaxed_ik_ros1.git src 
cd src
git clone https://github.com/Unity-Technologies/ROS-TCP-Endpoint.git

If in WSL2 install rust like so:

sudo apt-get update
sudo apt-get install curl
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
echo 'source $HOME/.cargo/env' >> $HOME/.bashrc

If in Ubuntu install rust like so:

sudo apt-get update
sudo apt-get install curl
curl https://sh.rustup.rs -sSf | bash -s -- -y
source $HOME/.cargo/env
echo 'source $HOME/.cargo/env' >> $HOME/.bashrc

Build relaxed_ik_core:

cd relaxed_ik_ros1/relaxed_ik_core/
cargo build

Use fetch.yaml settings:

cp configs/example_settings/fetch.yaml configs/settings.yaml

Build the directory

cd ..
catkin build

Source the directory

source devel/setup.bash

Unity

• Open up the Unity project
• Open up the FetchRelaxedIKScene
• Open up the FetchRelaxedIKScene
• Update your IP address in the Robotics tab
• Enable holographic remoting for play mode under the Mixed Reality tab, adding the IP address of your Hololens holographic play mode app

Running the Application

ROS

Launch the Relaxed IK solver by typing the following command:

roslaunch relaxed_ik_ros1 demo.launch

Launch the ros_tcp_endpoint by typing the following command adding in your own IP address:

roslaunch ros_tcp_endpoint endpoint.launch tcp_ip:=127.0.0.1 

Unity

• Press play on the Unity application
• Grab the Fetch robot's end effector and move it around.