UR Robotic Arm with Robotiq 2-Finger Gripper for ROS 2

Blog post (dev journey, engineering details): How I'm Building an Autonomous Pick-and-Place System with ROS 2 Jazzy and Gazebo Harmonic

UR3 + Robotiq 2-Finger Gripper on ROS 2 Humble + Gazebo Harmonic: URDF/ros2_control, MoveIt Task Constructor pick-and-place, vision-based detection, LLM task planning (Ollama), RL (SAC), and demo recording for behavior cloning.

New here? Start with Full MTC Pick-and-Place Demo — one command to run.

Requires ROS 2 Humble and Gazebo Harmonic (gz-sim 8.x) — Ignition Fortress (gz-sim 6) will not work.

git clone https://github.com/darshmenon/UR3_ROS2_PICK_AND_PLACE.git
cd UR3_ROS2_PICK_AND_PLACE

export ROS_DISTRO=humble   # or jazzy
sudo apt install ros-$ROS_DISTRO-rviz2 ros-$ROS_DISTRO-joint-state-publisher \
  ros-$ROS_DISTRO-robot-state-publisher ros-$ROS_DISTRO-ros2-control \
  ros-$ROS_DISTRO-ros2-controllers ros-$ROS_DISTRO-controller-manager \
  ros-$ROS_DISTRO-joint-trajectory-controller ros-$ROS_DISTRO-position-controllers \
  ros-$ROS_DISTRO-gz-ros2-control ros-$ROS_DISTRO-ros2controlcli \
  ros-$ROS_DISTRO-moveit ros-$ROS_DISTRO-moveit-ros-perception \
  ros-$ROS_DISTRO-simple-grasping ros-$ROS_DISTRO-cv-bridge \
  ros-$ROS_DISTRO-tf2-ros ros-$ROS_DISTRO-tf2-geometry-msgs ros-$ROS_DISTRO-pcl-ros

# Jazzy only, also install: ros-jazzy-ros-gz-sim ros-jazzy-ros-gz-bridge ros-jazzy-moveit-planners-stomp
# (STOMP isn't packaged for Humble — leave it out, the planner init fails silently and is harmless)

pip3 install -r requirements.txt
pip3 install py-trees          # required for ur_bt_planner

# Ollama, for the LLM planner: install from https://ollama.com, then
ollama pull llama2:latest

colcon build --symlink-install
source install/setup.bash

MTC source is already patched and vendored in src/moveit_task_constructor/ for both Humble and Jazzy — colcon build above covers it, no extra steps.

MTC needs MongoDB (warehouse_ros_mongo) running:

curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | \
  sudo gpg -o /usr/share/keyrings/mongodb-server-7.0.gpg --dearmor
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/7.0 multiverse" | \
  sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt-get update && sudo apt-get install -y mongodb-org
sudo systemctl enable --now mongod

Verify with mongosh. Humble/Jazzy API differences: ur_mtc_pick_place_demo/README.md.

Run MTC with one command (Gazebo + MoveIt first, MTC started after it is up), then trigger a pick and place:

ros2 launch ur_gazebo mtc_demo.launch.py gripper:=robotiq_2f_85        # default gripper is $UR_GRIPPER, else onrobot_rg2
ros2 service call /mtc_bt/run std_srvs/srv/Trigger {}                  # retries on failure; progress on /mtc_bt/status

OnRobot RG2, 5 cm cylinder

Robotiq 2F-85, 5 cm cube

_{MTC plans the whole pick and place before moving; the long planning pause is cut from these GIFs.}

Status: works on the OnRobot RG2 (7 of 8 trials in a row at 1-19 mm) and, less reliably, on the Robotiq 2F-85. The RG6 and 2F-140 settings exist but are not verified yet. Each run ends by moving the arm to the view_clear pose, so the next run's point cloud does not contain the arm.

source install/setup.bash
bash ur_mtc_pick_place_demo/scripts/robot.sh              # GUI
USE_GAZEBO_GUI=false bash ur_mtc_pick_place_demo/scripts/robot.sh   # headless

That script just runs three launches in order (sim+MoveIt → planning scene server → pick/place task). Success looks like Task executed successfully. Occasional execute -3 is RRTConnect random-seed variance, not a bug — rerun, or use the MTC retry wrapper for automatic retry.

get_planning_scene_server turns the head depth cloud into MoveIt collision objects: crops to the workspace, RANSAC-fits the support plane and object, and serves them via /get_planning_scene_ur. Falls back to a known table+cylinder if perception fails or force_fallback_scene:=true.

ros2 launch ur_gazebo ur.gazebo.launch.py                       # default, Robotiq 2F-85
ros2 launch ur_gazebo ur.gazebo.launch.py gripper:=robotiq_2f_140
ros2 launch ur_gazebo ur.gazebo.launch.py gripper:=onrobot_rg2   # or onrobot_rg6
ros2 launch ur_gazebo ur.gazebo.launch.py wrist_camera:=true     # eye-in-hand instead of head cam

Launch args: use_rviz, use_move_group, use_gazebo_gui (all default true), gripper (default robotiq_2f_85), world_file, table_height, rviz_config (default moveit_config/rviz/moveit.rviz — see Visualization & Logging for a leaner alternative).

Starts move_group + RViz with ompl (default) and pilz_industrial_motion_planner pipelines.

1. Planning Group → arm
2. Context → Planning Library → OMPL, Planner → RRTConnectkConfigDefault
3. Drag the interactive marker to a goal, click Plan & Execute

Cartesian (straight-line) moves: RViz Cartesian Path tab, or Pilz LIN/PTP/CIRC planners. Configs: moveit_config/config/ompl_planning.yaml, pilz_cartesian_limits.yaml.

To attach RViz/MoveIt to a sim already running headless, use moveit_config move_group.launch.py with use_move_group/use_rviz/use_sim_time flags instead.

Wait ~10-15s after launch, then confirm:

ros2 control list_controllers   # arm_controller, gripper_controller, joint_state_broadcaster — all active

Default gripper: onrobot_rg2 (best benchmark result, see below). Every single-arm launch file reads UR_GRIPPER, so export UR_GRIPPER=robotiq_2f_85 switches the whole stack (Gazebo, MoveIt, MTC, MotionExecutor) at once; the gripper:= launch argument still overrides it. The dual-arm launches stay on the Robotiq grippers.

All use position_controllers/GripperActionController. Mimic joints are state-only — Gazebo Harmonic enforces <mimic> constraints at the physics level.

ros2 action send_goal /gripper_controller/gripper_cmd control_msgs/action/GripperCommand \
  "{command: {position: 0.5, max_effort: 50.0}}"

Useful debug commands: ros2 topic echo /joint_states, ros2 topic hz /camera_head/depth/color/points, ros2 run tf2_ros tf2_echo base_link tool0.

ros2 action send_goal /arm_controller/follow_joint_trajectory control_msgs/action/FollowJointTrajectory \
'{"trajectory": {"joint_names": ["shoulder_pan_joint","shoulder_lift_joint","elbow_joint","wrist_1_joint","wrist_2_joint","wrist_3_joint"],
  "points": [{"positions": [0.0, -1.57, 1.57, 0.0, 1.57, 0.0], "time_from_start": {"sec": 2, "nanosec": 0}}]}}'

Automation script: python3 ur_system_tests/scripts/arm_gripper_loop_controller.py. GUI: python3 ur_system_tests/scripts/gui.py.

full_demo.launch.py brings up Gazebo, MoveIt, perception, grasp detection, and a selectable brain (mutually exclusive):

ros2 launch ur_gazebo full_demo.launch.py brain:=llm      # Ollama, via /llm_planner/command
ros2 launch ur_gazebo full_demo.launch.py brain:=rl model_path:=ur_rl_training/models/checkpoints/<run>/best_model.zip
ros2 launch ur_gazebo full_demo.launch.py brain:=openvla task:="pick the red block and place it in the bin"
ros2 launch ur_gazebo full_demo.launch.py brain:=none     # perception + grasp only

Startup sequence: Gazebo+MoveIt → perception (60s) → grasp (62s) → brain (65s).

Camera/Depth  → ur_perception → /detected_objects        → LLM planner / RL policy
PointCloud2   → ur_grasp      → /ur_grasp/grasp_pose      → RL policy (overrides perception)
Camera        → OpenVLA       → /arm_controller/joint_trajectory

Estimates grasp poses from the D435 point cloud — simple_grasping (PCL RANSAC, primary) or a built-in HSV-centroid fallback.

ros2 launch ur_grasp grasp_detection.launch.py colour:=red
python3 testing/test_grasp.py --colour red --execute

For eye-in-hand: launch sim with wrist_camera:=true, then point grasp_detection.launch.py at camera_topic:=/camera_wrist/depth/color/points. Publishes /ur_grasp/grasp_pose and /ur_grasp/grasp_marker.

Trains SAC in MuJoCo, deploys to Gazebo — reach → grasp → lift → place curriculum (auto-advances at eval reward ≥ 400), with domain randomization on mass/friction/size/noise.

cd ur_rl_training
python3 scripts/train.py --timesteps 3000000
python3 scripts/train.py --resume models/checkpoints/<run>/best_model   # resume from checkpoint

Deploy to Gazebo:

ros2 launch ur_gazebo ur.gazebo.launch.py world_file:=rl_policy_demo.world
ros2 launch ur_rl_training rl_policy.launch.py model_path:=ur_rl_training/models/checkpoints/<run>/best_model.zip

Key params: action_scale (0.1), step_dt (0.01), control_rate_hz (100), phase (0-3); object position auto-overridden by perception/grasp topics when running.

Headless eval: python3 ur_rl_training/scripts/eval_headless.py --model <path> --episodes 20.

Compare training runs (reads all ur_rl_training/logs/*/evaluations.npz, no launch needed):

python3 testing/plot_rl_run_comparison.py
# -> testing/rl_learning_curves.png, testing/rl_best_reward_bar.png

Gripper contact — monitors finger_joint effort to stop before crushing:

ros2 launch ur_force_control ft_monitor.launch.py

/ft/finger_effort (effort), /ft/contact_detected (bool), service /ft/compliant_close. Also via MotionExecutor.compliant_close_gripper() / .compliant_pick().

Arm-level wrench estimation (no F/T sensor — inferred from joint effort via F ≈ pinv_damped(Jᵀ)·τ_ext, gravity via Pinocchio):

ros2 launch ur_force_control wrench_estimator.launch.py
ros2 topic echo /ft/estimated_wrench
ros2 service call /ft/zero_wrench std_srvs/srv/Trigger {}

Params: force_threshold_n (15.0), damping_lambda (0.05). Note: /joint_states effort readback doesn't reliably track commanded torque, so absolute wrench values aren't trustworthy yet — relative/contact detection is fine.

Joint impedance control (torque mode, τ = K(q_des−q) − D·q̇ + g(q)):

ros2 launch ur_force_control joint_impedance.launch.py
ros2 control switch_controllers --deactivate arm_controller --activate forward_command_controller_effort
ros2 topic pub --once /joint_impedance_controller/target_positions std_msgs/msg/Float64MultiArray "{data: [0.0, -1.57, 1.57, 0.0, 1.57, 0.0]}"
ros2 control switch_controllers --deactivate forward_command_controller_effort --activate arm_controller   # hand back

Use arm_controller for MoveIt; switch to forward_command_controller_effort only for compliant hold/contact. Service: /joint_impedance_controller/hold_current_pose.

Hierarchical py_trees pick-and-place with IK-failure retry via a Selector fallback:

ros2 launch ur_bt_planner bt_planner.launch.py pick_x:=0.35 pick_y:=0.0 pick_z:=0.05 place_x:=0.15 place_y:=0.30 place_z:=0.08
ros2 service call /bt/run_pick_place std_srvs/srv/Trigger {}

Wraps the MTC demo with automatic retry (py_trees.decorators.Retry) since occasional -3 execute failures resolve on a fresh attempt:

ros2 launch ur_gazebo ur.gazebo.launch.py world_file:=pick_and_place_demo.world gripper:=robotiq_2f_85 \
  use_rviz:=false use_move_group:=true use_gazebo_gui:=true
ros2 launch ur_bt_planner mtc_retry.launch.py
ros2 service call /mtc_bt/run std_srvs/srv/Trigger {}
ros2 topic echo /mtc_bt/status

max_attempts (default 5) is a launch arg.

Spawns random-color boxes at the belt entry, moves them to the pick zone, publishes /conveyor/object_ready.

ros2 launch ur_conveyor conveyor.launch.py spawn_interval_s:=6.0 belt_speed:=0.03
ros2 service call /conveyor/start std_srvs/srv/Trigger {}

Topics/services: /conveyor/object_ready, /conveyor/picked, /conveyor/start, /conveyor/stop.

The arm picks each box off the belt and drops it on the bin of its colour (python3 testing/conveyor_pick_demo.py).

Two independent UR3+gripper arms (left_/right_ prefixed) on one shared world frame, one Gazebo model, one controller_manager.

ros2 launch ur_gazebo dual_ur.gazebo.launch.py            # GUI
ros2 launch ur_gazebo dual_ur.gazebo.launch.py use_gazebo_gui:=false gripper:=robotiq_2f_85

Args: gripper (all 4 grippers), table_height, left_y/right_y (base offsets, default ±0.45 m), world_file, use_move_group, use_rviz, use_dual_cameras (per-arm D435, off by default).

MoveIt groups: left_arm, right_arm, left_gripper, right_gripper and both_arms; move_group and RViz start by default.

Independent per-arm pipelines (namespaced left/right, fallback planning scene — no live perception dependency yet):

ros2 launch ur_gazebo dual_mtc_pick_place.launch.py     # two mtc_node instances
ros2 launch ur_bt_planner dual_mtc_retry.launch.py      # + BT auto-retry per arm
ros2 launch ur_sorting_demo dual_sorting_demo.launch.py # workspace-split sorting

Whole-body coordination (ur_dual_coordination): both hands hold one object and all 12 joints are solved together. Priorities: keep the hands' relative pose and the arms apart, move the object to its target, stay near a comfortable posture. Joints at their limits are frozen.

ros2 launch ur_dual_coordination dual_coordination.launch.py               # with the dual-arm sim running
ros2 service call /dual/grasp std_srvs/srv/Trigger {}                      # freeze the hands' relative pose
ros2 topic pub --once /dual/object_target geometry_msgs/msg/PoseStamped \
  "{header: {frame_id: world}, pose: {position: {x: 0.2, y: 0.0, z: 0.31}, orientation: {w: 1.0}}}"
ros2 service call /dual/release std_srvs/srv/Trigger {}

Note: the object/place/bin/handover poses in the per-arm pipelines are placeholders from the single-arm demo and are not yet confirmed reachable.

RViz. The default config (moveit_config/rviz/moveit.rviz, also used by dual_ur.gazebo.launch.py via dual_ur.rviz) now starts framed on the workspace instead of zoomed out to the whole world, has the MTC Motion Planning Tasks panel pre-expanded with computation times shown, and carries a Detection Image display (/detection_image, off by default — enable it in the Displays panel when ur_perception's object detector is running).

For a monitoring-only view (no planning controls, no MTC debug panel), pass an alternate config:

ros2 launch ur_gazebo ur.gazebo.launch.py \
  rviz_config:=$(ros2 pkg prefix moveit_config)/share/moveit_config/rviz/operator_view.rviz

Web dashboard (ur_web_dashboard). Browser UI with a live camera feed, joint-state bars, detected-object list, LLM command box (with voice input), and a filterable /rosout log panel — searchable by node/message, filterable by severity, light/dark toggle (persisted).

ros2 launch ur_web_dashboard dashboard.launch.py   # rosbridge (9090) + web_video_server (8080)
# then open ur_web_dashboard/web/index.html in a browser

Native log/plot viewers. rqt_console and PlotJuggler, wired up as a convenience launch:

ros2 launch ur_web_dashboard view_logs.launch.py                    # both
ros2 launch ur_web_dashboard view_logs.launch.py plotjuggler:=false # rqt_console only

Robotiq 2F-85, 5 cm cylinder	Robotiq 2F-140, 5 cm cube
OnRobot RG2 (default), 5 cm cube	OnRobot RG6, 5 cm cube

tall box

5 cm cylinder

The same runs as still contact sheets (arm at home, approach, grasp, lift, carry, placed)

Robotiq 2F-85, 5 cm cube	Robotiq 2F-140, 5 cm cube
OnRobot RG2, 5 cm cube	OnRobot RG6, 5 cm cube

python3 testing/benchmark_grippers.py --trials 5 --domain 91 --output results.csv   # all grippers x objects
python3 testing/benchmark_grippers.py --grippers onrobot_rg2 --objects cube_5cm --record-dir frames/   # + PASS/FAIL image strips
python3 testing/benchmark_grippers.py --summarise-csv results.csv                   # print the table again

Latest full run (4 objects x 5 trials each, pick & place): RG2 80%, 2F-85 60%, 2F-140 35%, RG6 30%.

• Real robot deployment (swap Gazebo hardware interface for the live UR3 driver)
• Fine-tune OpenVLA on collected demonstrations for sim-to-real
• 6-DoF object pose estimation from depth for grasp orientation
• Dual-arm MTC task graph for coordinated bimanual pick
• Multi-object sorting: BT planner + conveyor + perception together
• Vision (ur_perception): color/YOLO detection works; multi-view reconstruction (reconstruct.launch.py) is inspection-only, not yet consumed by the pick pipeline
• LLM planner (ur_llm_planner): relative moves (move the gripper up by 10 centimeters) and named poses work; pick-by-description doesn't yet — the color detector reports zero objects for this scene

Contributions welcome, especially around sim stability, transfer learning, and perception-to-action integration.