OpenArm ROS2 Bimanual Control & Vision — Setup Package

Complete setup package for the OpenArm v10 bimanual robot control system with Intel RealSense D435i vision integration, using ROS 2 Humble.

System Requirements

OS: Ubuntu 22.04 LTS (Jammy Jellyfish)
Hardware: Two OpenArm v10 7-DOF arms + grippers, PCAN-USB Pro FD dual-channel adapter (can1 = right, can2 = left)
Camera: Intel RealSense D435i (USB 3.0 required — use a blue USB port)
Disk: ~5 GB free for ROS 2 + vision packages

Important: Source ROS 2 in Every Terminal

Before running any ros2 or colcon command, you must source all three workspaces. Run these in every new terminal (or add them to ~/.bashrc):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash

Tip: Add all three lines to your ~/.bashrc so every new terminal is ready automatically:
echo 'source /opt/ros/humble/setup.bash' >> ~/.bashrc
echo 'source ~/ros2_ws/install/setup.bash' >> ~/.bashrc
echo 'source ~/openarm_ws/install/setup.bash' >> ~/.bashrc

Workspace	Location	Provides
ROS 2 Humble	`/opt/ros/humble`	Core ROS 2 framework
`ros2_ws`	`~/ros2_ws`	`openarm_bringup`, `openarm_description`, `bimanual_control`
`openarm_ws`	`~/openarm_ws`	`vision_advanced`, `realsense2_camera`

Key File Locations

File	Path
This README	`~/Openarm/ROS2_control/README.md`
After-reboot CAN script	`~/Openarm/ROS2_control/scripts/04_after_reboot.sh`
Vision install script	`~/Openarm/ROS2_control/scripts/install_vision.sh`
Hand-eye calibration data	`~/openarm_ws/src/Openarm_ROS2_Vision/config/calibration_result.yaml`
Camera config	`~/openarm_ws/src/Openarm_ROS2_Vision/config/camera_config.yaml`
Robot arm source code	`~/ros2_ws/src/`
Vision source code	`~/openarm_ws/src/Openarm_ROS2_Vision/`
RealSense driver source	`~/openarm_ws/src/realsense-ros/`

Quick Install (One Command)

cd ~/Openarm/ROS2_control/scripts
bash full_install.sh

This runs all stages in sequence. You'll need your sudo password at the start.

Step-by-Step Install

If you prefer to run stages separately:

Script	What It Does
`01_install_ros2_humble.sh`	Install ROS 2 Humble Desktop + dev tools
`02_install_ros2_control_packages.sh`	Install ros2_control, MoveIt2, joint_state_publisher_gui, OpenArm CAN library
`03_setup_workspace.sh`	Clone all repos, build workspace
`04_after_reboot.sh`	Post-reboot CAN setup + quick-reference commands

cd ~/Openarm/ROS2_control/scripts
bash 01_install_ros2_humble.sh
bash 02_install_ros2_control_packages.sh
bash 03_setup_workspace.sh

After Every Reboot — Robot Arm Control (Real Hardware)

⚠️ IMPORTANT: You must run the source commands in EVERY new terminal tab you open. If you skip this, you will get command not found or package not found errors.

Step-by-Step Real Robot Testing

Step 1 — CAN setup (run once after every reboot):

bash ~/Openarm/ROS2_control/scripts/04_after_reboot.sh

Expected output: both can1 and can2 report ✓ active and OK.

Step 2 — Terminal 1: Start hardware + RViz:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch openarm_bringup openarm.bimanual.launch.py

Expected: RViz opens showing the bimanual robot. The console should print Configured and activated left_joint_trajectory_controller and Configured and activated right_joint_trajectory_controller with no errors.

Step 3 — Verify arm mapping in RViz:

Gently move the physical right arm → the right arm in RViz should mirror the movement.
Gently move the physical left arm → the left arm in RViz should mirror the movement.
If left/right are still swapped, see the CAN Interface Mapping section below.

Step 4 — Terminal 2: Start preview slider + ghost robot:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch bimanual_control bimanual_control.launch.py mode:=real start_rviz:=false

Step 5 — Terminal 3: Start capture & execute:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run bimanual_control capture_and_execute

What you see in RViz (Real Mode):

Solid robot (alpha=1.0) = real hardware position from /joint_states
Semi-transparent ghost robot (alpha=0.4) = slider preview from /preview_joint_states
Drag sliders → ghost moves → confirm in Terminal 3 → solid robot follows

Vision-Based Demo (After Every Reboot)

Requires the Intel RealSense D435i camera plugged into a USB 3.0 port (blue port).

⚠️ IMPORTANT: You must run the 3 source commands in EVERY new terminal tab you open. If you skip this, you will get Package 'vision_advanced' not found errors.

Vision — Simulation Mode (No Robot Arms Connected)

Use this when you want to test the vision pipeline without physical robot arms.

Terminal 1 — Perception pipeline (robot sim + camera + YOLO detector + RViz):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch vision_advanced perception_demo.launch.py mode:=simulation

Terminal 2 — Grasp planner (generates pick-place waypoints from detected objects):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run vision_advanced grasp_planner

Terminal 3 — Pick-place state machine (coordinates the full workflow):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run vision_advanced pick_place_demo

Vision — Real Hardware Mode

Terminal 1 — CAN setup (run once after reboot, then close this terminal):

bash ~/Openarm/ROS2_control/scripts/04_after_reboot.sh

Terminal 2 — Perception pipeline (real hardware + camera + YOLO):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch vision_advanced perception_demo.launch.py mode:=real

Terminal 3 — Grasp planner:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run vision_advanced grasp_planner

Terminal 4 — Pick-place demo:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run vision_advanced pick_place_demo

What to Expect

Hold a bottle, cup, bowl, apple, or orange in front of the camera
Terminal 2 (simulation) or Terminal 3 (real) prints detected object 3D positions
The pick-place terminal prints === PICK-PLACE SEQUENCE READY === with a 7-step plan

Useful Debug Commands

Open a new terminal (remember to source!) and run any of these:

# See annotated camera feed with bounding boxes + class labels
ros2 run rqt_image_view rqt_image_view /object_detection_viz

# See raw object positions
ros2 topic echo /object_poses

# Check camera is connected
lsusb | grep Intel

RViz Performance Tip

If RViz is laggy, uncheck PointCloud in the Displays panel on the left. The vision pipeline does not need the point cloud visualization to work.

MoveIt Production-Line Stress Cycle (Autonomous, Bimanual)

Moves both arms simultaneously through configurable waypoints (A → B → C → A …) in an infinite loop using MoveIt 2 + OMPL motion planning. All 14 joints are engaged.

Telemetry is logged to CSV at /home/nirvana-ai/Documents/openarm_stresstest/:

Column	Description
`cmd_pos_deg`	Last commanded joint position (degrees)
`actual_pos_deg`	Measured joint position (degrees)
`temperature_C`	Motor temperature (°C)
`P_m_W`	Mechanical power = \|τ × ω\| (W)
`P_e_W`	Electrical power = V_bus × \|τ / Kt\| (W)

Workspace folder to open in your IDE: ~/ros2_ws/src/openarm_moveit_stress/

Simulation (Fake Hardware)

Terminal 1 — Fake hardware backend:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_bringup openarm.bimanual.launch.py use_fake_hardware:=true

Terminal 2 — Bimanual motion + RViz:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_moveit_stress stress_cycle.launch.py

Real Hardware

Step 1 — CAN setup (once after reboot):

bash ~/Openarm/ROS2_control/scripts/04_after_reboot.sh

Step 2 — Real hardware backend (no use_fake_hardware flag):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_bringup openarm.bimanual.launch.py

Step 3 — Motion script (same as simulation, in another Terminal):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_moveit_stress stress_cycle.launch.py

⚠️ CAUTION: On real hardware, start with slow speeds (max_velocity_scaling_factor = 0.1) and small joint deltas. Keep your hand on the E-stop.

Stopping the Test

Press Ctrl+C in the stress cycle terminal (Terminal 2). The shutdown sequence:

Both arms return to zero position via MoveIt trajectory
Final telemetry sample is written
CSV and JSON stats are saved to ~/Documents/openarm_stresstest/
You can then Ctrl+C Terminal 1 (bringup) to disable motors

Test Duration & Rest Cycles

Parameter	Default	Description
`MAX_HOURS`	8.0	Total test runtime (0 = unlimited)
`COOLDOWN_EVERY`	40	Take a rest every N cycles
`COOLDOWN_S`	60.0	Rest duration (seconds)
`DWELL_S`	1.5	Pause at each waypoint

Customizing the Waypoints Interactively

You can now use RViz to intuitively drag the arms and record your own custom waypoint sequence without touching code!

Step 1: Start the hardware simulator in Terminal 1 (this ensures points stay solidly and don't snap to zero):

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_bringup openarm.bimanual.launch.py use_fake_hardware:=true

(If you haven't already, add the MotionPlanning tool by clicking [Add] at the bottom left panel of the open RViz window -> select moveit_ros_visualization -> MotionPlanning).

Step 2: Connect the MoveIt planner to it in Terminal 2:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 launch openarm_bimanual_moveit_config move_group.launch.py

Step 3: Run the waypoint recorder in Terminal 3:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 run openarm_moveit_stress record_waypoints

Step 4: Record Your Poses:

In the RViz MotionPlanning panel, select left_arm or right_arm in the Planning Group drop-down.
Drag the interactive ball marker to move the ghost robot arm.
Click Plan & Execute. (You will see the solid robot physically move permanently to the new spot).
Switch to Terminal 3 and press [Enter] to officially save that pose. Repeat for as many poses as you want!
Type save to export waypoints.json.

Next time you run stress_cycle.launch.py, it will automatically load and loop your custom waypoints!

Real-Time Telemetry Plotter

You can monitor all 14 motor temperatures simultaneously in a real-time graph:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
ros2 run openarm_moveit_stress plot_temps

The plot includes dynamic bars with visual warnings at 65°C and an ESTOP threshold at 75°C.

Each entry is a dict of {joint_name: radians}. You can have 2, 3, or any number of waypoints:

LEFT_WAYPOINTS = [
    {   # ── Pose A ──
        "openarm_left_joint1": -0.6,
        ...
    },
    {   # ── Pose B ──
        "openarm_left_joint1": -1.2,
        ...
    },
]

RIGHT_WAYPOINTS = [
    {   # ── Pose A (mirrored) ──
        "openarm_right_joint1":  0.6,
        ...
    },
    ...
]

After editing, rebuild:

cd ~/ros2_ws && colcon build --symlink-install --packages-select openarm_moveit_stress

Using VLA Mock Poses Instead of Hardcoded Waypoints

To have the VLA mock service generate the target poses at runtime:

Start the VLA mock services (Terminal 3):

source ~/ros2_ws/install/setup.bash
ros2 run openarm_vla_mock camera_tf &
ros2 run openarm_vla_mock vla_infer &
ros2 run openarm_vla_mock vla_pose_bridge

Use the full stress test orchestrator instead of ab_motion:
```
ros2 launch openarm_moveit_stress stress_test.launch.py
```
The orchestrator calls /vla/infer to get grasp/place poses from the VLA mock, transforms them to world frame, and plans/executes motions to those poses.

Tip: The stress_cycle.py file also contains a commented-out Cartesian (XYZ + quaternion) planning mode. Uncomment _plan_and_execute_cartesian if you want to specify goal positions in world-frame meters.

No-Hardware Modes (Robot Arm Only, No Vision)

Simulation Mode — Quick Test (No Ghost Robot)

Simplest way to test. 1 terminal, sliders directly control the robot:

Terminal 1:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch bimanual_control bimanual_control.launch.py mode:=simulation

✅ Sliders move the robot directly in RViz
❌ No ghost robot — there is only one robot model, and it moves with the sliders
❌ No capture_and_execute workflow

Fake Hardware Mode — Full Workflow with Ghost Robot

If you want the same experience as real hardware (ghost robot + solid robot + capture & execute) but without physical arms, use use_fake_hardware:=true. Open 3 terminals:

Terminal 1 — Start fake hardware + RViz:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch openarm_bringup openarm.bimanual.launch.py use_fake_hardware:=true

Terminal 2 — Start preview slider + ghost robot:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 launch bimanual_control bimanual_control.launch.py mode:=real start_rviz:=false

Terminal 3 — Start capture & execute:

source /opt/ros/humble/setup.bash
source ~/ros2_ws/install/setup.bash
source ~/openarm_ws/install/setup.bash
ros2 run bimanual_control capture_and_execute

Note: Terminal 1's RViz already includes the ghost robot display. Use start_rviz:=false in Terminal 2 to avoid opening a duplicate RViz window.

What you see in RViz (Fake Hardware Mode):

Solid robot stays at its current position (starts at home/zero pose)
Semi-transparent ghost robot moves as you drag the sliders
Press Enter in Terminal 3 → confirm → solid robot moves to match the ghost

⚠️ Do NOT use mode:=real without Terminal 1 running. Real mode expects the hardware bringup (real or fake) to already be providing /joint_states. If you do, you will see red "No transform" errors and RViz will be very laggy.

Architecture

Manual Mode (Slider Control)

joint_state_publisher_gui (sliders)
  → /preview_joint_states
    ├── preview_robot_state_publisher → ghost TF in RViz
    └── capture_and_execute → FollowJointTrajectory
          → Real robot (via ros2_control)
              → /joint_states → solid robot TF in RViz

Autonomous Mode (Stress Cycle)

stress_cycle (Python)        ← editable WAYPOINTS list + timing params
  → MoveItPy (OMPL planner + KDL IK)
    → FollowJointTrajectory action
      → ros2_control (fake or real hardware)
        → /joint_states → robot TF in RViz
        → /tmp/stress_cycle_stats_*.json (stats)

CAN Interface Mapping

This system uses a PCAN-USB Pro FD dual-channel adapter. Because the laptop also has an unrelated gs_usb CAN device on can0, the PCAN channels enumerate as can1 and can2:

CAN Interface	Adapter	Arm	State after setup
`can0`	gs_usb (unrelated)	—	DOWN (ignored)
`can1`	PCAN-USB Pro FD ch1	Right arm	UP
`can2`	PCAN-USB Pro FD ch2	Left arm	UP

Tip: Run ip -details link show can1 to verify your adapter type. If your PCAN adapter enumerates differently (e.g., can0/can1 on a system without the extra gs_usb device), update the defaults in:

~/Openarm/ROS2_control/scripts/04_after_reboot.sh

~/ros2_ws/src/openarm_ros2/openarm_bringup/launch/openarm.bimanual.launch.py

~/ros2_ws/src/openarm_description/urdf/robot/openarm_robot.xacro

~/ros2_ws/src/openarm_description/urdf/ros2_control/openarm.bimanual.ros2_control.xacro

Then rebuild: cd ~/ros2_ws && colcon build --symlink-install --packages-select openarm_description openarm_bringup

URDF Body Orientation (Left / Right Arm Fix)

The URDF model's "front" must match the physical robot's facing direction. If the left and right arms are swapped in RViz compared to the real robot, the fix is a 180° yaw rotation on the body mount joint — NOT swapping arm positions.

The relevant args in v10.urdf.xacro:

<!-- 180° yaw rotates the model to match physical robot orientation -->
<xacro:arg name="rpy" default="0 0 3.14159" />

<!-- Arm base positions — must match upstream (do NOT swap these) -->
<xacro:arg name="right_arm_base_xyz" default="0.0 -0.031 0.698" />
<xacro:arg name="right_arm_base_rpy" default="1.5708 0 0" />
<xacro:arg name="left_arm_base_xyz"  default="0.0  0.031 0.698" />
<xacro:arg name="left_arm_base_rpy"  default="-1.5708 0 0" />

Note: The reflect parameter in openarm_arm.xacro must be passed to <xacro:openarm-kinematics> and <xacro:openarm-limits> for all 7 joints. The upstream repo was missing this for joints 3–7, causing the left arm meshes to render with right-arm geometry from J3 downward. This has been fixed locally.

After any URDF or launch file change, rebuild:

cd ~/ros2_ws && source /opt/ros/humble/setup.bash && colcon build --symlink-install --packages-select openarm_description openarm_bringup

Troubleshooting

Problem	Cause	Fix
`ros2: command not found`	ROS 2 not sourced in this terminal	`source /opt/ros/humble/setup.bash`
`No module named 'ament_package'` during build	ROS 2 not sourced before `colcon build`	Source ROS 2 first, then rebuild
Red "No transform from [preview/...]" in RViz	Launched `mode:=real` without Terminal 1 running	Start Terminal 1 first, or use `mode:=simulation`
RViz is very laggy	Failed TF lookups from missing transforms	Fix the "No transform" issue above
No joint slider window	`ros-humble-joint-state-publisher-gui` not installed	`sudo apt install -y ros-humble-joint-state-publisher-gui`
No ghost robot in simulation mode	Expected — simulation mode has only one robot	Use Fake Hardware Mode (see above) for ghost + solid
Second RViz window opens	Terminal 2 also starts RViz by default	Add `start_rviz:=false` to Terminal 2 command
Left/right arms swapped in RViz	URDF model faces opposite direction from real robot	Set body `rpy` yaw to `3.14159` in `v10.urdf.xacro` — see URDF section above
J3+ meshes look swapped between arms	Missing `reflect` parameter in `openarm_arm.xacro`	Ensure `reflect="${reflect}"` is passed for all joints — see URDF section above
Left arm not enabled after reboot	CAN interface mismatch (e.g., `can0` is a different adapter)	Run `ip link show` and update CAN interface names — see CAN Interface Mapping above
Right arm controls left arm in RViz	CAN interfaces are swapped	Verify CAN → arm mapping in launch file defaults — see CAN Interface Mapping above
URDF/launch changes have no effect	Workspace not rebuilt after editing source files	Run `colcon build --packages-select openarm_description openarm_bringup` then re-source

Repositories Included

Repo	Source	Workspace	Purpose
`openarm_ros2`	enactic/openarm_ros2	`ros2_ws`	ROS2 bringup, hardware interface
`openarm_can`	enactic/openarm_can	`ros2_ws`	CAN bus communication library
`openarm_description`	enactic/openarm_description	`ros2_ws`	URDF/xacro robot description
`openarm_teleop`	enactic/openarm_teleop	`ros2_ws`	Teleoperation
`bimanual_control`	Sazabi06/Openarm-ROS2-robot-control	`ros2_ws`	Bimanual control package
`openarm_moveit_stress`	local	`ros2_ws`	Production stress cycle, MoveIt orchestrator
`realsense-ros`	IntelRealSense/realsense-ros	`openarm_ws`	RealSense camera ROS2 driver
`vision_advanced`	Sazabi06/Openarm_ROS2_Vision	`openarm_ws`	YOLO detection, calibration, grasp planning

Folder Structure

Openarm/ROS2_control/
├── scripts/
│   ├── 01_install_ros2_humble.sh       # ROS 2 Humble + dev tools
│   ├── 02_install_ros2_control_packages.sh  # Control + MoveIt2 + CAN lib
│   ├── 03_setup_workspace.sh           # Clone repos + build
│   ├── 04_after_reboot.sh              # CAN setup + quick-reference
│   └── full_install.sh                 # One-shot full installer
├── config/
│   └── safety_config.yaml              # Safety parameters reference
└── README.md                           # This file

~/ros2_ws/src/                          # Robot arm control workspace
├── openarm_ros2/                       # Hardware bringup + controllers
├── openarm_description/                # URDF/xacro models
├── bimanual_control/                   # Slider + capture/execute
└── openarm_moveit_stress/              # Production stress cycle + orchestrator
    └── openarm_moveit_stress/stress_cycle.py  # ← edit WAYPOINTS + timing here

~/openarm_ws/src/                       # Vision workspace (overlay)
├── realsense-ros/                      # RealSense camera ROS2 driver
└── Openarm_ROS2_Vision/                # YOLO, calibration, grasp planning
    └── config/calibration_result.yaml  # Saved hand-eye calibration

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
bimanual_control		bimanual_control
config		config
launch		launch
openarm_hw_control		openarm_hw_control
openarm_moveit_stress		openarm_moveit_stress
openarm_msgs		openarm_msgs
openarm_teleop		openarm_teleop
openarm_vla_mock		openarm_vla_mock
resource		resource
rviz		rviz
scripts		scripts
ARCHITECTURE.md		ARCHITECTURE.md
README.md		README.md
arm_control_gui.py		arm_control_gui.py
bimanual_arm_controller.py		bimanual_arm_controller.py
bimanual_control.launch.py		bimanual_control.launch.py
install.sh		install.sh
package.xml		package.xml
setup.cfg		setup.cfg
setup.py		setup.py
setup_can.sh		setup_can.sh
test_system.py		test_system.py

Folders and files

Latest commit

History

Repository files navigation