HuRo

A staged pipeline that turns raw egocentric (first-person) human video into robot-ready training data. Each stage is a standalone, resumable script named stage<N>_<phase>_<name>.py; stages run in order, and a later stage consumes only the output of the stage immediately before it.

Layout

pipeline/        stage scripts (run in order)
  stage1_annot_intrinsics.py       # camera intrinsics from raw video
  stage2_annot_contact.py          # hand detection (100DoH detector, hand class only)
  stage3_annot_contact_refine.py   # track-based hand-side refinement (BoT-SORT)
  stage4_annot_hand.py             # 3D hand keypoints + MANO + hand masks (HAWOR)
  stage5_annot_extrinsics.py       # per-frame camera extrinsics (DROID-SLAM + MoGe-2 + GeoCalib)
  stage6_annot_segment.py          # per-frame arm masks (Detectron2 + SAM2)
  segmentation/                    # stage-6 model wrappers (ViTDet-H person detection, SAM2)
  stage7_annot_narr.py             # per-segment narration + chunked videos (Qwen3.5 VLM)
  captioning/                      # stage-7 Qwen3.5 VLM subsystem (caption, prompts, validate, tracker)
  stage8_annot_inpaint.py          # per-segment arm-removal inpainting (ProPainter)
  stage9_robot_retarget.py         # human hand keypoints -> robot joints + EEF (PyRoKi IK)
  retargeting/                     # stage-9 IK subsystem (two-stage solver + FK helpers)
  stage10_robot_overlay.py         # render the robot's arms into the arm-removed video
  overlay/                         # stage-10 Isaac Sim rendering subsystem (+ URDF import)
  stage11_lerobot_convert.py       # pack the overlay output into LeRobot V2.0 datasets
configs/         per-robot YAML (joint groups -> state layout, keypoint mapping, home pose)
common/          importable shared library, grouped by broad concern
  camera.py      # camera intrinsics: validation, ensemble selection, undistortion
  geometry.py    # bbox crop coverage, pinhole 3D->2D projection, SE(3) interp, SLERP smoothing
  stats.py       # generic numeric helpers (MAD-robust median, Gaussian kernel)
  video.py       # H.264 segment encode + output-file validation
  io.py          # shared per-frame Parquet I/O (schemas, chunk writer, ParquetReader)
  paths.py       # repo-root paths + the --part a/b sharding helper
  robot_config.py # robot YAML loader: state layout, EEF dims, schema metadata
submodules/      pristine upstream vision backends (git submodules)
  droidcalib/    # DROID-SLAM + self-calibration (its droid_backends is reused at stage 5)
  anycalib/      # AnyCalib single-view calibration
  100doh/        # 100DoH hand-object contact detector
  hawor/         # HAWOR hand-mesh reconstruction; also bundles the stage-5 DROID-SLAM frontend
  moge-2/        # MoGe-2 metric monocular depth (stage 5 scale)
  geocalib/      # GeoCalib gravity/calibration (stage 5)
  detectron2/    # Detectron2 ViTDet-H person detection (stage 6)
  sam2/          # SAM 2 video segmentation (stage 6 arm masks)
  propainter/    # ProPainter video inpainting (stage 8 arm removal)
  pyroki/        # PyRoKi kinematic optimization (stage 9 JAX IK; used as shipped)
  allex_model/   # the Allex robot model, used as shipped (URDF: stage 9 + stage 10)
submodules_patches/   patches applied on top of the pristine submodules

Each backend's model weights live inside its own submodule (kept out of git); the stage-7 Qwen3.5 VLM, an HF model rather than a submodule, auto-downloads into the user's HF cache ($HF_HOME). Neither is stored in this repo — the Model weights section below has the download commands.

The robot model is used exactly as shipped: submodules/allex_model is pinned upstream, and its patch only appends the fixed convention frames the pipeline needs (a mano-convention wrist frame and a palm anchor per side, and an OpenCV-convention camera frame). Stage 10 needs a USD, so it imports that same URDF once into build/ and renders it. Everything the pipeline itself decides — joint groups, keypoint mapping, home pose, which links to hide when rendering — lives in configs/.

From stage 2 on, every stage's output is a chunked per-frame Parquet table sharing one cumulative schema (common/io.py): each stage fills the fields it computes and leaves later-stage fields None, so stage N reads only stage N-1's table.

Setup

Linux, an NVIDIA GPU with a driver supporting CUDA 12.8, git. Pick Option A (conda) or Option B (docker) and follow only that one.

Step 0: clone (both options)

git clone <HuRo-repo-url> HuRo
cd HuRo
git submodule update --init --recursive

Option A: conda

./setup.sh                # conda env `huro` + CUDA extensions + import checks
./download.sh             # model weights (see below)
conda activate huro       # before every run

If internet and GPU are on different machines:

./setup.sh env                          # where there is internet
./setup.sh build && ./setup.sh check    # where there is a GPU

Option B: docker

The image carries the environment only; your checkout stays on the host. docker_run.sh mounts the directory containing the checkout at /workspace, so the repo is at /workspace/HuRo (and its siblings stay reachable) and the shell starts there.

docker build -t huro .    # ~2 hours: the CUDA extensions compile ahead of time
./docker_run.sh

First time inside the container, in this order:

./setup.sh dev       # patch the submodules in this checkout + editable re-install
./setup.sh check     # verify every stage stack imports on this GPU
./download.sh        # model weights (see below)

dev comes first: the image ships the environment but no code, so the submodule patches have to be applied to the checkout you mounted.

python is already the environment, so there is nothing to activate. Edits under /workspace are edits to your real checkout, and they apply on the next run with no rebuild.

Keep your clips under that same parent directory and they show up beside the repo. If they live somewhere else entirely, mount a higher directory: HURO_MOUNT=/media/ssd1 ./docker_run.sh.

Model weights (both options)

./download.sh fetches every checkpoint (idempotent; needs internet, no python environment). Two items need manual steps:

• 100DoH checkpoint: while this repo is private, run gh auth login first
• MANO models: register at https://mano.is.tue.mpg.de, download mano_v*_*.zip, and place

submodules/hawor/_DATA/data/mano/MANO_RIGHT.pkl
submodules/hawor/_DATA/data_left/mano_left/MANO_LEFT.pkl

The stage-7 Qwen3.5-9B VLM (~18 GB) auto-downloads on first run into $HF_HOME (default ~/.cache/huggingface). To pre-fetch: hf download Qwen/Qwen3.5-9B.

Run

CUDA_VISIBLE_DEVICES=0 python pipeline/stage1_annot_intrinsics.py \
    --input_dir /path/to/clips --part 1/1 --no_tqdm

Run the stages in order (stage1 … stage11); each stage script's header documents its I/O contract and flags, and --part a/b shards clips across processes. Stage 10 renders through Isaac Sim: the node needs a working Vulkan ICD (the stage checks and fails loudly), and in batch jobs point the shader caches at node-local disk and accept the Kit EULA:

export HOME=/tmp/huro_ov_cache_$USER && mkdir -p "$HOME"
export __GL_SHADER_DISK_CACHE=1 __GL_SHADER_DISK_CACHE_PATH="$HOME/nvgl" CUDA_CACHE_PATH="$HOME/cuda"
export OMNI_KIT_ACCEPT_EULA=Y

Using the dataset

Stage 11 writes plain LeRobot V2.0; per-frame features and their conventions:

feature	dim	meaning
observation.state / action	48	active joint angles in the config's group order; action[t] = state[t+1]
observation.state_eef_{left,right} / action_eef_*	9	wrist pose in the robot base frame: position (3) + rotation as the first two rows of the rotation matrix (6)
observation.state_cam_frame	9	camera pose in the same base frame and 3+6 layout, OpenCV axes (x right, y down, z forward)

Joint names and per-side hand-joint layouts are recorded in each episode parquet's schema metadata. meta/stats.json (exact per-dimension min/max/mean/std for every float feature, shared across the dataset's resolution groups) is written at the end of every stage-11 run; re-running stage 11 on a finished dataset refreshes it.

examples/load_dataset.py is the reference for consuming the data — a minimal PyTorch Dataset that joins the parquet vectors, video frames and task strings per timestep and applies min-max normalization from stats.json:

python examples/load_dataset.py /path/to/clips_lerobot/allex/192x342

Pipeline stages

#	Script	Output
1	stage1_annot_intrinsics.py	per-clip camera intrinsics JSON (fx, fy, cx, cy, xi, H, W, model)
2	stage2_annot_contact.py	per-frame Parquet: hand boxes + side (carries raw intrinsics + intr_model)
3	stage3_annot_contact_refine.py	per-frame Parquet: track-consistent hand sides
4	stage4_annot_hand.py	per-frame Parquet: + 3D hand keypoints, MANO rotations, hand masks
5	stage5_annot_extrinsics.py	per-frame Parquet: + camera pose (4x4 T_cam2world, OpenCV)
6	stage6_annot_segment.py	per-frame Parquet: + arm mask (bit-packed) + person count; drops per-hand mask
7	stage7_annot_narr.py	per-segment chunked (undistorted) videos + Parquet: + narr caption (Qwen3.5 VLM)
8	stage8_annot_inpaint.py	per-segment arm-removed inpainted videos (ProPainter); no Parquet written
9	stage9_robot_retarget.py	per-segment Parquet: robot state_qpos + per-side EEF + IK diagnostics; cam_pose rebased to the robot's frame
10	stage10_robot_overlay.py	per-segment overlay videos (robot arms rendered into the arm-removed frames) + Parquet: + overlay_valid
11	stage11_lerobot_convert.py	LeRobot V2.0 dataset(s): one episode per segment — remapped joints + EEF + resized video + meta/ incl. stats.json

(One environment runs all eleven stages, whether built on the host by ./setup.sh or baked into the Docker image. The robot stages support the Allex robot only.)