This document describes the Rviz2AR Project as well as how to prepare your evironment to use Rviz2AR in your Unity 3D project with ROS. Rviz2AR is a ROSIN funded project.
What is Rviz2AR
The goal of this project is to provide a stable communication for data exchange between ROS and the Microsoft Hololens as well as an integration for the game engine Unity for the creation and visualization of robot data. The project should be upscaleable for different hardware, allowing the use of AR and VR devices for different tasks. An interface toolkit for robot data visualization will be developed and the capabilities of the application will be demonstrated with a showcase using industrial robot hardware running ROS.
The project is currently in Milestone 1 where we are working on the software module for a stable communication link between the Unity application running on AR/VR gear and ROS, enabling to send and receive live data.
How does Rviz2AR work?
The communication infrastructure of Rviz2AR is based on the ROS-Sharp(ROS#) module originally developed by Siemens AG. ROS# is using rosbridge_suite to provide ROS-Unity communication via a JSON API using ROSBridge Server
It should be mentioned that, even though our Rvis2AR now provides a working data communication between Unity and ROS it is not without shortcomings. While the underlying JSON API is sufficient in supporting the exchange of small data, larger data files like camera feeds or point cloud data are not handled efficiently. To compensate for those limitations we plan to incorporate another communication link between Unity and ROS based on ROS2. To this end, we are in contact with the team of ARViz: Augmented Reality Visualizer for ROS2 by Francisco Martin Rico. Our goal would be to enhance our currently implementation with a ROS2 DDS or larger data files. Since the ARViz Project is still in active development, we will incorporate it into our project in the near future, when it is available.
What is Rviz2AR capable of right now?
Rviz2AR is still in active development and and is not feature complete at the moment.
The current build serves the purpose of demonstrating the general communication infrastructue for ROS-Unity data exchange running on the Microsoft Hololense.
A preview for the visualization toolbox, that gives the user the option to interact with the ROS data in AR-space is also implemented.
1. Setting up your development environment
Check your Windows 10 version
It is recommended to be running the Windows 10 Fall Creators update for modern Windows Universal apps targeting Mixed Reality Headsets
This can be verified by running the "WinVer" application from the Windows Run command
Windows Key + R -> WinVer
Which will display a new window as follows:
If you are not running the Windows 10 Fall Creators update, then you will need to Update your version of Windows.
Set up your development environment
Be sure to enable Developer mode for Windows 10 via:
Action Center -> All Settings -> Update & Security -> For Developers -> Enable Developer mode
If you have not already, download and install Visual Studio 2017 and these required components:
- Windows Universal Platform Development Workload
- Windows SDK 10.16299.10
- Visual Studio Tools for Unity
- Nuget Package Manager
You can install more components and UWP SDK's as you wish.
Note: Be sure to include the Windows Store .NET scripting backend components.
3. Ubuntu on Oracle VM VirtualBox
If you want to use ROS from a gazebo Simulator tis chapter explains how to set up Oracle VM VirtualBox to run ROS in a Gazebo simulation
Download the latest version of Virtual Box from here from here and install it
Setting up the Network
In Oracle VirtualBox, make sure the Ubuntu VM is powered off. Then open the settings for the Ubuntu VM. In the Network tab, add two new network adapters:
- Host-only Adapter
- Network Bridge
In Ubuntu type
$ ifconfig to check the network configuration and to verify your Ethernet connection to the Windows OS. The IP address (enp0s8) of the Ubuntu system will be used by rosbridge_suite and RosBridgeClient.
These settings are needed so that the RosBridgeClient running in Windows, and the ROSBridge Server running on Ubuntu can communicate.
Setting up ROS on Ubuntu VM
- Follow this tutorial to install ROS Kinetic
- Follow this tutorial to install and configure the ROS environment
- Install rosbridge-suite via
$ sudo apt-get install ros-kinetic-rosbridge-server
See http://wiki.ros.org/rosbridge_suite for further information.
Gazebo Setup for VM with Turtlebot24.
This tutorial shows how to install and setup Gazebo, which is used for robot simulation in ROS.
- Using the following commands, setup your Ubuntu VM to accept software from packages.osrfoundation.org and install the Gazebo7 packages
$ sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list' $ wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add - $ sudo apt-get update $ sudo apt-get install gazebo7
- Install the required packages
$ sudo apt-get install libgazebo7-*
This tutorial explains how to set up the example robot TurtleBot2.
- Install the required TurtleBot2 sources on the Ubuntu VM
$ sudo apt-get install ros-kinetic-turtlebot ros-kinetic-turtlebot-apps ros-kinetic-turtlebot-interactions ros-kinetic-turtlebot-simulator ros-kinetic-kobuki-ftdi ros-kinetic-ar-track-alvar-msgs ros-kinetic-turtlebot-gazebo
Gazebo Simulation Example Setup
- create a new Catkin workspace
$ mkdir -p ~/catkin_ws/src $ cd ~/catkin_ws/src $ catkin_init_workspace $ cd .. $ catkin_make
Place the folder
gazebo_simulation_scene(from here) inside the src folder of your Catkin workspace and rebuild your workspace.
In the directory g
gazebo_simulation_scene/scriptsmake the file
joy_to_twist.pyexecutable by running
chmod +x joy_to_twist.py
- rebuild your Catkin workspace
Run Rviz2Ar with Gazebo Simulation Example
- Run the following command in the Ubuntu terminal
$ roslaunch gazebo_simulation_scene gazebo_simulation_scene.launch
- Open a ner tab in the termina and type
- Look for the line
- There ´, look for the line
- Copy the ip displayed there (for example: 192.168.2.198)
Open the Rviz2AR Project in your Unity Editor
- In the Unity scene look for the RosConnector
- In the RosConnector look for the script Ros Connector wih the field
Ros Bridge Server Url
- Past your coppied IP there so it has the form:
ws://yourIP:9090(so if your coppied IP was
192.168.2.198it would be
- Press play in the inspector to start the application
Run Rviz2Ar with Gazebo Simulation on Hololense
- Make sure, that the Hololense is conencted to the same network as the Computer/VM
In Unity Editor go to
- Make sure that Universal WIndoes Platform is selected
- Click on
- Go to
Other Settingand make sure, that in the tab
Scripting Runtime version,
Api Compatibility Levelare selected with
.NET 4.x Equivaent,
- Go to
Publish Settingand make sure, that in the tab
Go Back to the Build Settings and press
Build and select a Folder where the Project should be build
Go the the build Folder and select the
.sln File to open the Project in Visual Studio
Make sure your Hololense is connected and select
Device in the Visual Studio deploy settings
Click on the
> Device- Button to deploy the project to the Hololense
Gazebo Setup for VM with UR5 Robot5.
In your catkin workspace go to
src -> universal_robot -> ur_gazebo -> launch and open the
In the lauch file, add the following lines, which will start the rosbridge websocket:
<include file="$(find rosbridge_server)/launch rosbridge_websocket.launch"> <param name="port" value="9090"/> </include>
Supported by ROSIN - ROS-Industrial Quality-Assured Robot Software Components. More information: rosin-project.eu)
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 732287.