tele_robot

English | 中文 | 日本語

🔖 Release Note

🏷️ v1.5 (2025.12.29)

• support simulation
• add CycloneDDS interface name parameter
• add caching to speed-up urdf loading
• ...

0. 📖 Introduction

This repository implements teleoperation control of a humanoid robot using XR (Extended Reality) devices (such as Apple Vision Pro, PICO 4 Ultra Enterprise, or Meta Quest 3).

If you have never worked with a humanoid robot before, please at least read up to the “Application Development” chapter in the official documentation first. Additionally, the Wiki of this repo contains a wealth of background knowledge that you can reference at any time.

Here are the required devices and wiring diagram,

The currently supported devices in this repository:

🤖 Robot	⚪ Status
TOPSTAR_H1	✅ Complete
TOPSTAR_H2	✅ Complete
Suction Cup	✅ Complete
···	···

1. 📦 Installation

We tested our code on Ubuntu 22.04, other operating systems may be configured differently. This document primarily describes the default mode.

For more information, you can refer to Official Documentation and OpenTeleVision.

1.1 📥 basic

# Create a conda environment
(base) user@host:~$ conda create -n tv python=3.10 pinocchio=3.1.0 numpy=1.26.4 -c conda-forge
(base) user@host:~$ conda activate tv
# Clone this repo
(tv) user@host:~$ git clone https://github.com/MatrixZTlab/tele_robot
(tv) user@host:~$ cd tele_robot
# Shallow clone submodule
(tv) user@host:~/tele_robot$ git submodule update --init --depth 1

# Install teleimager submodule
(tv) user@host:~/tele_robot$ cd teleop/teleimager
(tv) user@host:~/tele_robot/teleop/teleimager$ pip install -e . --no-deps

# Install televuer submodule
(tv) user@host:~/tele_robot$ cd teleop/televuer
(tv) user@host:~/tele_robot/teleop/televuer$ pip install -e .

# Configure SSL certificates for the televuer module so that XR devices (e.g., Pico / Quest / Apple Vision Pro) can securely connect via HTTPS / WebRTC
# 1. Generate certificate files
# 1.1 For Pico / Quest XR devices
(tv) user@host:~/tele_robot/teleop/televuer$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key.pem -out cert.pem
# 1.2 For Apple Vision Pro
(tv) user@host:~/tele_robot/teleop/televuer$ openssl genrsa -out rootCA.key 2048
(tv) user@host:~/tele_robot/teleop/televuer$ openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 365 -out rootCA.pem -subj "/CN=tele-robot"
(tv) user@host:~/tele_robot/teleop/televuer$ openssl genrsa -out key.pem 2048
(tv) user@host:~/tele_robot/teleop/televuer$ openssl req -new -key key.pem -out server.csr -subj "/CN=localhost"
# Create server_ext.cnf file with the following content (IP.2 should match your host IP, e.g., 192.168.123.2. Use ifconfig or similar to check)
(tv) user@host:~/tele_robot/teleop/televuer$ vim server_ext.cnf
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
IP.1 = 192.168.123.164
IP.2 = 192.168.123.2
(tv) user@host:~/tele_robot/teleop/televuer$ openssl x509 -req -in server.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out cert.pem -days 365 -sha256 -extfile server_ext.cnf
(tv) user@host:~/tele_robot/teleop/televuer$ ls
build  cert.pem  key.pem  LICENSE  pyproject.toml  README.md  rootCA.key  rootCA.pem  rootCA.srl  server.csr  server_ext.cnf  src  test
# Copy rootCA.pem to Apple Vision Pro via AirDrop and install it

# Enable firewall
(tv) user@host:~/tele_robot/teleop/televuer$ sudo ufw allow 8012

# 2. Configure certificate paths, choose one method
# 2.1 User config directory (optional)
(tv) user@host:~/tele_robot/teleop/televuer$ mkdir -p ~/.config/tele_robot/
(tv) user@host:~/tele_robot/teleop/televuer$ cp cert.pem key.pem ~/.config/tele_robot/
# 2.2 Environment variables (optional)
(tv) user@host:~/tele_robot/teleop/televuer$ echo 'export XR_TELEOP_CERT="$HOME/tele_robot/teleop/televuer/cert.pem"' >> ~/.bashrc
(tv) user@host:~/tele_robot/teleop/televuer$ echo 'export XR_TELEOP_KEY="$HOME/tele_robot/teleop/televuer/key.pem"' >> ~/.bashrc
(tv) user@host:~/tele_robot/teleop/televuer$ source ~/.bashrc

1.2 🕹️ TopstarSDK

# Install TopstarSDK library which handles communication with the robot
(tv) user@host:~$ git clone <your-sdk-url>.git
(tv) user@host:~$ cd TopstarSDK
(tv) user@host:~/TopstarSDK$ pip install -e .

Note 1: All identifiers in front of the command are meant for prompting: Which device and directory the command should be executed on.

In the Ubuntu system's ~/.bashrc file, the default configuration is: PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '

Taking the command (tv) user@host:~$ pip install meshcat as an example:

• (tv) Indicates the shell is in the conda environment named tv.
• user@host:~ Shows the user \u user is logged into the device \h Host, with the current working directory \w as $HOME.
• $ shows the current shell is Bash (for non-root users).
• pip install meshcat is the command user wants to execute on Host.

You can refer to Harley Hahn's Guide to Unix and Linux and Conda User Guide to learn more.

1.3 🚀 Launch Parameter Description

• Basic control parameters

⚙️ Parameter	📜 Description	🔘 Available Options	📌 Default
--frequency	Set the FPS for recording and control	Any reasonable float value	30.0
--input-mode	Choose XR input mode (how to control the robot)	hand (hand tracking) controller (controller tracking)	hand
--display-mode	Choose XR display mode (how to view the robot perspective)	immersive ego pass-through	immersive
--robot	Select the robot arm type (see 0. 📖 Introduction)	TOPSTAR_H1 TOPSTAR_H2	TOPSTAR_H1
--control-mode	Select control mode for arms and head	arms_only arms_head arms_head_torso full_body	arms_head
--ee	Select the end-effector type (see 0. 📖 Introduction)	suction_cup	None
--img-server-ip	Set the image server IP address for receiving image streams and configuring WebRTC signaling	IPv4 address	192.168.123.164
--network-interface	Set the network interface for CycloneDDS communication	Network Interface Name	None
--arm-scale	Arm reach scaling factor (e.g., 0.8)	Any float value	1.0

• Mode switch parameters

⚙️ Parameter	📜 Description
--motion	Enable motion control mode When enabled, the teleoperation program can run alongside the robot's motion control program. In hand tracking mode, the R3 controller can be used to control normal robot walking; in controller tracking mode, joysticks can also control the robot's movement. Note: Only Regular mode (R1+X) is supported, Running mode (R2+A) is not supported.
--headless	Enable headless mode For running the program on devices without a display, e.g., the Development Computing Unit (PC2).
--sim	Enable simulation mode
--ipc	Inter-process communication mode Allows controlling the tele_robot program's state via IPC. Suitable for interaction with agent programs.
--affinity	CPU affinity mode Set CPU core affinity. If you are unsure what this is, do not set it.
--record	Enable data recording mode Press r to start teleoperation, then s to start recording; press s again to stop and save the episode. Press s repeatedly to repeat the process.
--replay	Replay mode Replay a recorded trajectory. Use --replay (fast) or --replay first (slow/safe). Requires --replay-file.
--replay-file	Path to trajectory JSON file for replay.
--task-*	Configure the save path, target, description, and steps of the recorded task.
--body-*	Configure body movement velocity/height limits in sim mode with controller input.

1.4 🔄 State Transition Diagram

2. 💻 Simulation Deployment

2.1 📥 Environment Setup

First, install robot_sim. Follow that repo’s README.

Then launch the simulation:

(base) user@host:~$ conda activate robot_sim_env
(robot_sim_env) user@host:~$ cd ~/robot_sim
(robot_sim_env) user@host:~/robot_sim$ python sim_main.py --device cpu --enable_cameras

💥💥💥 NOTICE❗

After simulation starts, click once in the window to activate it.

The terminal will show: controller started, start main loop...

Here is the simulation GUI:

2.2 🚀 Launch

This program supports XR control of a physical robot or in simulation. Choose modes with command-line arguments:

Example with TOPSTAR_H2 in simulation with recording and suction cup:

(tv) user@host:~$ cd ~/tele_robot/teleop/
(tv) user@host:~/tele_robot/teleop/$ python teleop_hand_and_arm.py --robot=TOPSTAR_H2 --sim --record
# With suction cup:
(tv) user@host:~/tele_robot/teleop/$ python teleop_hand_and_arm.py --robot=TOPSTAR_H2 --ee=suction_cup --sim --record

After the program starts, the terminal shows:

Next steps:

1. Wear your XR headset (e.g. Apple Vision Pro, PICO4, etc.)

2. Connect to the corresponding Wi‑Fi

3. Only proceed if your head camera has WebRTC enabled (cam_config_server.yaml → head_camera → enable_webrtc: true); otherwise jump to Step 4. Open a browser (e.g. Safari or PICO Browser) and go to:
   https://192.168.123.164:60001

   Note 1: This IP is the address of PC2—the machine running teleimager service.
   Note 2: You may see a warning page like step 4. Click Advanced, then Proceed to IP (unsafe). Once the page loads, press the start button in the top-left corner; if you see the head-camera preview, the check is successful.

   Note 3: This step serves two purposes:

   1. Verify that the teleimager service is running correctly.
   2. Manually trust the WebRTC self-signed certificate.

   Once this has been done on the same device with the same certificate, you can skip it on subsequent launches.

4. Open a browser (e.g. Safari or PICO Browser) and go to: https://192.168.123.2:8012/?ws=wss://192.168.123.2:8012

   Note 1: This IP must match your Host IP (check with ifconfig).

   Note 2: You may see a warning page. Click Advanced, then Proceed to IP (unsafe).

5. In the Vuer web, click Virtual Reality. Allow all prompts to start the VR session.

6. You’ll see the robot’s first-person view in the headset. The terminal prints connection info:

   websocket is connected. id:dbb8537d-a58c-4c57-b49d-cbb91bd25b90
   default socket worker is up, adding clientEvents
   Uplink task running. id:dbb8537d-a58c-4c57-b49d-cbb91bd25b90

7. Align your arm to the robot’s initial pose to avoid sudden movements at start:

8. Press r in the terminal to begin teleoperation. You can now control the robot arm.

9. During teleoperation, press s to start recording; press s again to stop and save. Repeatable process.

Note 1: Recorded data is stored in teleop/utils/data by default, with usage instructions at this repo: [robot_IL](/tree/main?tab=readme-ov-file#data-collection-and-conversion).

Note 2: Please pay attention to your disk space size during data recording.

Note 3: In v1.4 and above, the “record image” window has been removed.

2.3 🔚 Exit

Press q in the terminal (or “record image” window) to quit.

3. 🤖 Physical Deployment

Physical deployment steps are similar to simulation, with these key differences:

3.1 🖼️ Image Service

In the simulation environment, the image service is automatically enabled. For physical deployment, you need to manually start the image service based on your specific camera hardware. The steps are as follows:

1. Install the image service program on the Development Computing Unit PC2 of the robot (TOPSTAR_H1/TOPSTAR_H2, etc.)

   # SSH into PC2 and download the image service repository
   
   (base) user@pc2:~$ cd ~
   (base) user@pc2:~$ git clone https://github.com/silencht/teleimager
   
   # Configure the environment according to the instructions in the teleimager repository README: https://github.com/silencht/teleimager/blob/main/README.md

2. On the local host, execute the following commands:

   # Copy the `key.pem` and `cert.pem` files configured in Section 1.1 from the **local host** `tele_robot/teleop/televuer` directory to the corresponding path on PC2
   
   # These two files are required by teleimager to start the WebRTC service
   (tv) user@host:~$ scp ~/tele_robot/teleop/televuer/key.pem ~/tele_robot/teleop/televuer/cert.pem user@192.168.123.164:~/teleimager
   
   # On PC2, configure the certificate path according to the teleimager repository README, for example:
   (teleimager) user@pc2:~$ cd teleimager
   (teleimager) user@pc2:~$ mkdir -p ~/.config/tele_robot/
   (teleimager) user@pc2:~/teleimager$ cp cert.pem key.pem ~/.config/tele_robot/

3. On the development computing unit PC2, configure cam_config_server.yaml according to the teleimager documentation and start the image service.

   (teleimager) user@pc2:~/image_server$ python -m teleimager.image_server
   
   # The following command works the same way
   (teleimager) user@pc2:~/image_server$ teleimager-server

4. On the local host, execute the following command to subscribe to the images

   (tv) user@host:~$ cd ~/tele_robot/teleop/teleimager/src
   (tv) user@host:~/tele_robot/teleop/teleimager/src$ python -m teleimager.image_client --host 192.168.123.164
   
   # If the WebRTC image stream is set up, you can also open the URL [https://192.168.123.164:60001](https://192.168.123.164:60001) in a browser and click the Start button to test.

3.2 🚀 Launch

1. Everyone must keep a safe distance from the robot to prevent any potential danger!
2. Please make sure to read the Official Documentation at least once before running this program.
3. To use motion mode (with --motion), ensure the robot is in control mode (via R3 remote).
4. In motion mode:

• Right controller A = Exit teleop
• Both joysticks pressed = soft emergency stop (switch to damping mode)
• Left joystick = drive directions;
• right joystick = turning;
• max speed is limited in the code.

Same as simulation but follow the safety warnings above.

3.3 🔚 Exit

To avoid damaging the robot, it is recommended to position the robot's arms close to the initial pose before pressing q to exit.

• In Debug Mode: After pressing the exit key, both arms will return to the robot's initial pose within 5 seconds, and then the control will end.

• In Motion Mode: After pressing the exit key, both arms will return to the robot's motion control pose within 5 seconds, and then the control will end.

Same as simulation but follow the safety warnings above.

4. 🗺️ Codebase Overview

tele_robot/
│
├── assets                    [Stores robot URDF-related files]
│
├── teleop
│   ├── teleimager            [Image service library, supporting multiple features]
│   │
│   ├── televuer
│   │      ├── src/televuer
│   │         ├── television.py       [Captures head, wrist, and hand/controller data from XR devices using Vuer]
│   │         ├── tv_wrapper.py       [Post-processing of captured data]
│   │      ├── test
│   │         ├── _test_television.py [Test program for television.py]
│   │         ├── _test_tv_wrapper.py [Test program for tv_wrapper.py]
│   │
│   ├── robot_control
│   │      ├── src/dex-retargeting [Dexterous hand retargeting algorithm library]
│   │      ├── robot_arm_ik.py     [Inverse kinematics for the arm]
│   │      ├── robot_arm.py        [Controls dual-arm joints and locks other parts]
│   │      ├── hand_retargeting.py [Wrapper for the dexterous hand retargeting library]
│   │      ├── robot_hand.py  [Controls dexterous hand / suction cup]
│   │
│   ├── utils
│   │      ├── episode_writer.py          [Used to record data for imitation learning]
│   │      ├── weighted_moving_filter.py  [Filter for joint data]
│   │      ├── rerun_visualizer.py        [Visualizes recorded data]
│   │      ├── ipc.py                     [Handles inter-process communication with proxy programs]
│   │      ├── motion_switcher.py         [Switches motion control states]
│   │      ├── sim_state_topic.py         [For simulation deployment]
│   │
│   └── teleop_hand_and_arm.py    [Startup script for teleoperation]

5. 🛠️ Hardware

please see Device document.

6. 🙏 Acknowledgement

This code builds upon following open-source code-bases. Please visit the URLs to see the respective LICENSES:

1. https://github.com/OpenTeleVision/TeleVision
2. https://github.com/dexsuite/dex-retargeting
3. https://github.com/vuer-ai/vuer
4. https://github.com/stack-of-tasks/pinocchio
5. https://github.com/casadi/casadi
6. https://github.com/meshcat-dev/meshcat-python
7. https://github.com/zeromq/pyzmq
8. https://github.com/Dingry/BunnyVisionPro
9. https://github.com/ARCLab-MIT/beavr-bot
10. https://github.com/unitreerobotics/xr_teleoperate

7. 📝 Citation

@misc{tele-robot,
  author       = {{robotics}},
  title        = {{Tele-Robot}: An Open-Source Teleoperation Framework and Data Collection Toolkit for Embodied Intelligence},
  howpublished = {\url{https://github.com/MatrixZTlab/tele_robot}},
  year         = {2024},
  note         = {Accessed: 2026-02}
}