Robot Realtime Control Interfaces

Robots Realtime is a research codebase supporting modular software stacks for realtime control, teleoperation, and policy integration on real-world robot embodiments including bi-manual I2RT YAM arms, Franka Panda, (more to come...).

It provides extensible pythonic infrastructure for low-latency joint command streaming, agent-based policy control, visualization, and integration with inverse kinematics solvers like pyroki developed by Chung-Min Kim!

Shown is a headless-capable web-based real-time visualizer and controller for viewing commanded joint state and actual robot state, built with Viser

Installation

Clone the repository and initialize submodules:

git clone --recurse-submodules https://github.com/uynitsuj/robots_realtime.git
# Or if already cloned without --recurse-submodules, run:
git submodule update --init --recursive

Install the main package and I2RT repo for CAN driver interface using uv:

cd yam_realtime
curl -LsSf https://astral.sh/uv/install.sh | sh
source .venv/bin/activate

uv venv --python 3.11
uv pip install -e .
uv pip install dependencies/i2rt/

Configuration

If using YAM arms, configure YAM arms CAN chain according to instructions from the I2RT repo

Launch

Then run the launch entrypoint script with an appropriate robot config file. For Bimanual YAMS:

uv run robots_realtime/envs/launch.py --config_path configs/yam_viser_bimanual.yaml

For Franka Panda:

uv run robots_realtime/envs/launch.py --config_path configs/franka_viser_osc.yaml

Extending with Custom Agents

To integrate your own controller or policy:

Subclass the base agent interface:

from robots_realtime.agents.agent import Agent

class MyAgent(Agent):
    ...

Add your agent to your YAML config so the launcher knows which controller to instantiate.

Examples of agents you might implement:

• Leader arm or VR controller teleoperation
• Learned policy (e.g., Diffusion Policy, ACT, PI0)
• Offline motion-planner + scripted trajectory player

Linting

If contributing, please use ruff (automatically installed) for linting (https://docs.astral.sh/ruff/tutorial/#getting-started)

ruff check # lint
ruff check --fix # lint and fix anything fixable
ruff format # code format

Roadmap/Todos

• Add data logging infrastructure
• Implement a Diffusion Policy agent controller
• Implement a Physical Intelligence π0 agent controller