Generating Robot Hands from Human Demonstrations

Code release for the paper Generating Robot Hands from Human Demonstrations (project page).

We generate tree-structured robot hands (thumb + index) that reproduce human fingertip motion using the same simple controller employed after fabrication: inverse kinematics on target fingertip positions. The framework produces both a high-DoF fully actuated generalist hand and low-DoF task-specific hands with spatial four-bar (Bennett) mimic joints, and exports one-piece 3D-printable meshes with print-in-place joints.

The release contains three components:

1. Design optimization (gradient descent) — script/ktree_rotbnt_offsetfix.py
2. Low-DoF hand generation with a learned actor — script/load_conditional_actor.py (uses the shipped checkpoints), plus training code script/conditional_best_actor.py and script/train_latent.py
3. Mesh (STL) generation — design_gen/finger_gen_offset.py for mimic-joint hands, script/ktree_to_urdf.py + design_gen/treemesh_gen.py for fully actuated hands

Installation

Python ≥ 3.10 with a CUDA-capable GPU (the optimization and the actor run on GPU).

pip install -r requirements.txt
pip install -e .            # makes the `script` and `design_gen` packages importable

python-fcl is required by trimesh for the mesh collision checks.

Data

The demonstration data is not included. Place per-sequence .npz files under data/oakink_contact_tip/ (relative to the repo root), one file per trajectory, containing at least:

• contact_points: (T, 2, 3) float — thumb and index fingertip positions (meters) in the wrist frame, per frame.

We extracted these from OakInk right-hand sequences (file names ending in _rh_contact). Any data source with the same format works. A synthetic example trajectory (thumb circle + index square, used in the paper) is included at script/tools/synthetic_circle_square.npz; see script/tools/generate_squarecircle.py for how to create your own.

Scripts resolve bare directory names under data/ by default. Pass --data_dir (or --traj_dir for the actor trainer) with either a bare name like oakink_contact_tip or an absolute/relative path to point at your own trajectory folder.

--seq_name is the trajectory file name without the .npz extension. For example, for the dataset file data/oakink_contact_tip/scene_01__A008++seq__6803df6444210e558b06__2023-04-23-11-33-27_rh_contact.npz:

--seq_name scene_01__A008++seq__6803df6444210e558b06__2023-04-23-11-33-27_rh_contact

When using --contact_file_path (e.g. for the synthetic trajectory), --seq_name is only used to name the output, so any short label works, e.g. --seq_name synthetic_circle_square.

Optimized designs are saved to data/<out_dir>/ as <seq>_seed<seed>_tree<tree>_mimic<mimic>_loss<loss>_conv<converged>.npz, e.g. synthetic_circle_square_seed42_tree1-2-2_mimic0101_loss43.0460_conv209.npz. These design files are the input to the mesh generators.

1. Design optimization (gradient descent)

ktree_rotbnt_offsetfix.py jointly optimizes the hand morphology (link lengths, joint axes, mimic parameters) and per-frame joint angles to track the fingertip trajectory of a sequence. The tree structure is given as root-branch1-branch2 joint counts, and the mimic array marks which joints are passively coupled to their parent (one 0/1 digit per non-root joint). Its output .npz is what both mesh generators and the actor pipeline consume.

Fully actuated 6-DoF hand (tree 1-2-3, no mimic joints), here on the included synthetic trajectory:

cd script
python ktree_rotbnt_offsetfix.py \
    --seq_name synthetic_circle_square \
    --contact_file_path tools/synthetic_circle_square.npz \
    --tree_struct 1-2-3 --mimic_arr 00000 \
    --steps 20000 --save

For a dataset sequence, drop --contact_file_path and use the file name as --seq_name, e.g. --seq_name scene_01__A008++seq__6803df6444210e558b06__2023-04-23-11-33-27_rh_contact. For optimizing the design on the entire dataset, use --seq_name all. I recommend around 50000 steps and smaller learning rate for the entire dataset.

Low-DoF hand with mimic joints:

python ktree_rotbnt_offsetfix.py \
    --seq_name synthetic_circle_square \
    --contact_file_path tools/synthetic_circle_square.npz \
    --tree_struct 1-2-2 --mimic_arr 0101 \
    --steps 20000 --save

Useful flags: --seed -1 (random seed per run), --out_dir (subfolder of data/), --data_dir (trajectory folder; default oakink_contact_tip under data/), --contact_file_path to point at an arbitrary trajectory file (e.g. tools/synthetic_circle_square.npz) instead of a dataset sequence.

To sweep the optimization over many random sequences (this is how we generated designs at scale), use the batch script:

./run_experiments.sh

2. Low-DoF hand generation with the actor

Searching low-DoF mimic designs from scratch is slow, so we train a trajectory-conditioned actor that proposes hand designs and initial joint angles. At generation time the actor's proposal warm-starts a small CEM search, and the best sample is refined with a final long gradient-descent run.

The checkpoints we use are included:

• Actor: script/runs_ckpt_cond_actor_dir/cond_best_actor_1-2-2_0101_dir_k8_instep500_2026-02-24_19-22-13_seed42_std1.0_final.pt
• Contact-trajectory latent encoder: script/contact_ae_model/ae_best_mlp_L512_z128_mix.pt

Generate a 3-DoF hand (tree 1-2-2, mimic 0101) for a dataset sequence:

cd script
python load_conditional_actor.py \
    --actor_ckpt runs_ckpt_cond_actor_dir/cond_best_actor_1-2-2_0101_dir_k8_instep500_2026-02-24_19-22-13_seed42_std1.0_final.pt \
    --latent_ckpt contact_ae_model/ae_best_mlp_L512_z128_mix.pt \
    --seq_name scene_01__A008++seq__6803df6444210e558b06__2023-04-23-11-33-27_rh_contact \
    --gpu_ids 0 1 2 3 --procs_per_gpu 4 \
    --episodes 1000 --k_samples 8 --inner_steps 500 --design_lr 0.005 \
    --tree_struct 1-2-2 --mimic_arr 0101

For a synthetic/custom trajectory, add --contact_file_path tools/synthetic_circle_square.npz. The resulting design .npz is written to data/ktree_rotbnt/ (override with --out_dir).

Retraining (optional)

Train the contact-trajectory autoencoder used as the actor's context encoder:

python train_latent.py \
    --real_dir ../data/oakink_contact_tip \
    --synthetic_path <dir_with_synthetic_npz> \
    --model mlp --L 512 --z_dim 128

Train the actor (samples a random trajectory each episode, evaluates K design proposals in parallel with the inner-loop optimization + collision check, and regresses towards the best sample):

python conditional_best_actor.py \
    --traj_dir oakink_contact_tip \
    --gpu_ids 0 1 2 3 --procs_per_gpu 4 \
    --tree_struct 1-2-2 --mimic_arr 0101 \
    --episodes 5000 --k_samples 8 --inner_steps 500 --fixed_std 1.0 \
    --latent_ckpt contact_ae_model/ae_best_mlp_L512_z128_mix.pt

--traj_dir accepts a bare name under data/ or a full path to your trajectory folder.

3. Mesh (STL) generation

Mimic-joint hands

finger_gen_offset.py converts an optimized mimic design into a printable mesh: each mimic joint is realized as a spatial four-bar Bennett linkage solved from the optimized joint axes and mimic ratios, and print-in-place revolute joints are added. It also reports self-collisions of the generated linkage.

cd design_gen
python finger_gen_offset.py \
    --input ../data/ktree_rotbnt/<design>.npz \
    --max_J 6 --use_idx 0 --verbose

This writes finger.stl. --use_idx selects which frame's joint angles the mesh is posed at (or random); --test_fk cross-checks the mesh against the kinematic-tree forward kinematics.

Fully actuated hands

First convert the design to a URDF, then render the printable mesh from it:

cd script
python ktree_to_urdf.py --input ../data/ktree_rotbnt/<design>.npz --output kinematic_tree.urdf

cd ../design_gen
python treemesh_gen.py --urdf ../script/kinematic_tree.urdf --out tree.stl

For the fully actuated hands, each joint is a simple disk in the generated mesh. The user needs to align the designated motor holder (eg Dynamixel XL330) to that disk in a CAD software (eg Fusion 360). For the mimic hands, the passive joints have print-in-place joints, but they also need to be cleaned up in CAD software so that the joints do not fuse. Specifically, the meshes need to be cut with the outer wall of the cylinder and only keep the outer part of the links. Example Fusion files are provided in designs/.

Visualization (optional)

script/visualize_contacts_warp.py renders a contact trajectory together with an optimized design as an animated USD file using NVIDIA Warp: the design is replayed as an articulated kinematic tree (boxes for links, spheres for fingertips) tracking the contact-point spheres frame by frame.

It needs two extra packages. Note the version pin — warp.sim was removed from Warp in v1.10:

pip install "warp-lang<1.10" usd-core

Visualize an optimized design. Here --seq_name is the design file name inside data/<seq_dir>/, without the .npz extension (default --seq_dir is ktree_rotbnt). For example, for a design optimized on the synthetic trajectory and saved as data/ktree_rotbnt/synthetic_circle_square_seed42_tree1-2-2_mimic0101_loss43.0460_conv209.npz:

cd script
python visualize_contacts_warp.py \
    --seq_name synthetic_circle_square_seed42_tree1-2-2_mimic0101_loss43.0460_conv209 \
    --contact_file_path tools/synthetic_circle_square.npz \
    --ktree --save_usd contact_visualization.usd

For a design optimized on a dataset sequence, --contact_file_path can be omitted — the trajectory is loaded from data/oakink_contact_tip/<seq_name up to _seed>.npz (i.e. the sequence the design was optimized on):

python visualize_contacts_warp.py \
    --seq_name scene_01__A008++seq__6803df6444210e558b06__2023-04-23-11-33-27_rh_contact_seed42_tree1-2-2_mimic0101_loss2.1004_conv300 \
    --ktree --save_usd contact_visualization.usd

Omit --ktree to render only the contact trajectory. For OakInk sequences, pass --oakink_dir pointing at the OakInk-v2 dataset root to also render the manipulated object meshes; otherwise objects are skipped.

The output .usd animation can be opened in usdview, NVIDIA Omniverse, or Blender.

Citation

@article{yi2026generating,
  title  = {Generating Robot Hands from Human Demonstrations},
  author = {Yi, Sha and Hansen, Nicklas and Bai, Xueqian and
            Sferrazza, Carmelo and Tolley, Michael T. and Wang, Xiaolong},
  year   = {2026},
  url    = {https://arxiv.org/abs/2606.20549}
}