Skip to content

IRIM Workshop Wiki

Jump to bottom
Simone Macciò edited this page Oct 18, 2023 · 2 revisions

Welcome to the RICO-MR wiki! Here's the step to build the soccer simulation combining the power of Unreal Engine, ROS and Kafka.

STEP 1: Clone Repo and Open Project

  1. Clone RICO-MR repo via git clone https://github.com/TheEngineRoom-UniGe/RICO-MR --recursive
  2. Checkout to irim-demo branch via git checkout irim-demo
  3. Navigate to the root folder of the project and locate MR_HRI.uproject
  4. Right click on the file, press Show More Options and select Generate Visual Studio Project Files
  5. Once generated, open the MR_HRI.sln and compile the solution
  6. Wait for successful completion and UE editor should open automatically

STEP 2: Build Soccer Scene in Editor

  1. In the editor, navigate to the RobotSoccerDemo folder in the content browser, where most the assets needed are located

  2. Open a new level by selecting (on the top-left of the screen) File -> New Level -> Default. Save the new level to make sure a persistent reference is created

  3. Let’s add our first assets:

    • Under RobotSoccerDemo/Blueprints locate the GoalBP asset and drag it inside the editor scene to spawn the corresponding actor

    • Select it and in the Details panel (on the right of the screen) set the actor’s location to X=70, Y=315, Z=20, leave rotations to 0 and the scale set to 1

    • Repeat the process to spawn a second goal: this time set the location to X=220, Y=-315, Z=20 and set the rotation around Z to 180 degrees.

    • Locate the RobotSoccerBallBP and place the ball similarly in the scene by drag and drop, setting its initial location to X=150, Y=0, Z=25

  4. Now it’s time to add the robots:

    • Locate the Franka Emika Panda robot in the content browser (bottom left of the screen), under the folder Robots/Panda. Locate the panda data asset and drag and drop it in the editor scene
    • Set the robot’s position to X=150, Y=200, Z=20 and the rotation around Z to -90 degrees.
    • For the Universal Robot 5 (UR5), instead, we will use a default asset which ships with the URoboSim plugin, a fundamental tool that we employed to build our architecture upon. In the content browser, scroll down until you locate the URoboSim Content folder. If such a folder is not visible, locate the View Options button in the browser, as indicated by the following picture, click it and select Show Plugin Content.
    • Under URoboSim Content/UR5_WSG50, select the corresponding data asset and drag and drop it in the scene as before to spawn UR5
    • Select the robot and set its location to X=150, Y=-200, Z=20 and its rotation around Z to 90 degrees.
    • Fantastic! We have built our soccer scene. However, should you choose to Play our simulation now, you will find that the robots will immediately fall apart. That’s because we have not yet set any robot controller!

View Options

  1. Add Joints Controller to robots. In order to do that, navigate to RobotSoccerDemo/DataAssets and locate the RobotSoccer- BaseController asset. Open it via double click and expand the various menus to inspect the parameters. Specifically, note that under Robot Names (first row), there is an empty array. Here’s press twice the + icon to add two names to the array. Now simply add the names of the robots instances. To locate such names:

    • Close the asset and return to the scene
    • In the World Outliner (top right of the screen), locate the actors corresponding to the two robots and hover the mouse on them to pop up a small widget referencing the ID Name of the actor itself.
    • Copy paste those two IDs (e.g., panda_2 and UR5_WSG50_5) in the two empty spots we just created in the RobotSoccer-BaseController asset.
    • Save the controller asset as it is (we employ a simple Kinematic simulation with joint controller’s values set to the default) and drag and drop it in the scene as if it was an actor.
    • Nothing will pop up in the editor, but now the two robots have a joints controller attached to them and you can immediately notice the difference because by playing the simulation now the robots will be still and not fall apart. That means our robots have proper controllers keeping them in place!

  2. Our scene is now ready, however there is no logic attached to the robot, thus they don’t know how to move to intercept the ball and play soccer. That’s where ROS and Kafka come into play!

STEP 3: Run Local Kafka Server

  1. Clone the Kafka-Docker repository via git clone https://github.com/TheEngineRoom-UniGe/kafka-docker-compose
  2. Edit **PUBLIC_IP **in env.dev to match current network configuration
  3. Consistently, locate file RobotSoccer-KafkaConnectionParam under RobotSoccerDemo/DataAssets (in the UE project) and modify the servers list accordingly (change just the IP values)
  4. Run Docker configuration via #docker-compose --env-file ./configs/.env.dev -f KAFKA_SASL_PLAINTEXT-docker-compose.yml up
  5. Go back to the UE editor and Play to start the simulation and test connection to Kafka
  6. Analyze RobotSoccerGameInstanceBP and KafkaHandler to understand how connection with Kafka is established and how topics are exposed to control the robots’ logic

STEP 4: Control Robot through ROS

We provide a complementary repository for implementing the robots’ logic and control them throughout the soccer simulation. However, keep in mind that this step needs to be performed on a computer with Ubuntu and ROS installed.

  1. Clone repo as usual via git clone https://github.com/TheEngineRoom-UniGe/ros-kafka-demo

  2. The ROS package employs MoveIt! for planning motions for the two robots and at the same time provides a ROS-Kafka interface to make sure that we can control our soccer simulation running in the UE editor directly from ROS scripts.

  3. Follow the steps on the GitHub page to launch the MoveIt scripts and the ROS-Kafka interface successfully

STEP 5: Play Soccer on HoloLens

To migrate the soccer simulation we just developed to its Mixed Reality counterpart and properly deploy the application on HoloLens2, we need some additional steps:

  1. Open RobotSoccerMRLevel, which is already configured for spawning the whole scenario in its holographic form exploiting Unreal’s QR code tracking capabilities
  2. We need to use a QR code as anchor for the HoloLens to spawn the MR scene consistently
  3. To achieve such behavior, the level includes an actor of type RobotSoccerDynamicSpawner, which is in charge of implementing the QR code tracking pipeline and spawns the various holographic assets once the marker has been detected.
  4. RobotSoccerDynamicSpawner is also in charge of spawning the two robot models as holographic assets and exposes the Kafka topics associated with each robot
  5. Before proceeding to packaging the app for HoloLens, make sure RobotSoccerMRLevel is selected as Default Game Map under Edit -> Project Settings -> Maps & Modes
  6. Package the application for HoloLens through File -> Package Project -> HoloLens
  7. Once the executable is ready, connect the HoloLens via USB to the development computer and open the Windows Device Portal (click here for further information)
  8. Under Views -> Apps, select the newly packaged executable and install it on the device (again, make sure to check the aforementioned link for further information on how to install an application via Device Portal)
  9. Wear HoloLens and play app. Happy Mixed Reality Soccer!

Clone this wiki locally