Review AnyDexRT

In-Depth Review — AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance

Paper: "AnyDexRT: Calibration-Free Dexterous Hand Retargeting with Few-Shot Human Guidance" — arXiv 2607.08341 (Jul 9 2026) Authors: Chenxi Wang, Ying Feng, Hongjie Fang, Shangning Xia, Lixin Yang, Chuan Wen, Cewu Lu · Shanghai Jiao Tong University (Cewu Lu group) What it is: a calibration-free, cross-hand human→robot retargeting method — the load-bearing L4 bridge of the Dexterous-Hand Data Pyramid, made cheap and hand-agnostic.

Companions: Dexterous-Hand Data Pyramid · Do As I Do (the source-data counterpart) · One-Hand (cross-hand canonicalization) · Dexterous Manipulation.

---

1. TL;DR

1. Retargeting without per-hand calibration or hand-crafted objectives. AnyDexRT maps human fingertip motion to any dexterous hand by learning task-relevant fingertip correspondences rather than assuming geometric similarity — no precise calibration, no global shape matching.
2. Two-stage mapping: a self-supervised fingertip position mapper (f_m) + inverse kinematics (f_s), trained with three losses — Partial Chamfer (asymmetric human→robot fingertip mapping), Distance Preservation, and Local Motion Preservation (directional consistency in local frames — the key to calibration-robustness).
3. Few-shot human guidance: operators imitate a tiny set of reference gestures (~5 sampled configs per anchor type, interpolated to ~150–200 paired anchors per hand) to ground the mapping in task-relevant regions.
4. Pinch is handled specially: a contact classifier detects human pinch and searches the neighborhood of the mapped robot position for a valid pinch pose — fixing the grasp-critical configs generic mapping misses.
5. Results: across 7 dexterous hands, Local Motion Consistency 90.2% (GeoRT 59.8, optimization 52.2) and pinch success 62.0% (GeoRT 29.2); only 3 hyperparameters, runs at 293 Hz, stable under ±90° frame rotations.

---

2. Why it matters

• It attacks the pyramid's real bottleneck. The data pyramid argues L4 retargeting fidelity — not raw data volume — is what converts abundant human data into 5-finger skill. AnyDexRT makes L4 calibration-free and cross-hand, so the same pipeline serves Inspire, Allegro, Shadow, LEAP, etc. without per-hand tuning.
• Local-motion preservation is the calibration-robust trick. By enforcing directional consistency in local coordinate frames instead of matching global geometry, it stays stable under large frame-rotation errors (±90°) — precisely the calibration burden that breaks prior retargeters.
• Complements the source-data side. Pairs naturally with Do As I Do (which produces the human hand-object trajectories) and One-Hand (which canonicalizes hand morphology): AnyDexRT is the cheap universal mapper between them.

---

3. Method

flowchart LR
  H[Human fingertip motion] --> FM[Fingertip mapper f_m<br/>self-supervised]
  subgraph L[losses]
    PC[Partial Chamfer<br/>asymmetric human→robot]
    DP[Distance Preservation]
    LMP[Local Motion Preservation<br/>directional, calibration-robust]
  end
  FM --- L
  GUIDE[Few-shot human guidance<br/>~5 configs/anchor → ~150–200 anchors/hand] --> FM
  FM --> IK[Inverse kinematics f_s]
  CC[Contact classifier<br/>detect pinch → search robot pinch pose] --> IK
  IK --> OUT[Robot hand joints · 293 Hz]

Loading

• Fingertip mapping (f_m): learns human→robot fingertip correspondence self-supervised. Partial Chamfer maps human fingertip space into the feasible robot fingertip space without requiring full coverage; Distance Preservation keeps pairwise fingertip distances; Local Motion Preservation enforces local-frame directional consistency (less sensitive to calibration than global methods).
• Few-shot guidance: operators imitate a small set of reference gestures giving paired human-robot fingertip anchors; an alignment loss minimizes mapped-vs-anchor distance over M pairs. Only two anchor types (lateral rotation, bending); K₀=5 initial configs interpolated to K=50 (rotation) / K=100 (bending) per finger.
• Contact classifier: binary classifier on finger contact signals; on detected human pinch, searches the neighborhood of the mapped robot position for a valid robotic pinch pose.
• IK (f_s): maps target fingertip positions to joint angles.

---

4. Results (paper-reported)

Motion consistency, averaged across 7 hands (Inspire 6-DoF · Ability · XHand · Wuji · Allegro 16-DoF · LEAP · Shadow 24-DoF):

Metric	AnyDexRT	GeoRT	Optimization
Global Motion Consistency	79.9%	78.3%	62.0%
Local Motion Consistency	90.2%	59.8%	52.2%
Pinch success rate	62.0%	29.2%	39.6%

• Efficiency: 3 hyperparameters; 293 Hz.
• Robustness: stable Local Motion Consistency under ±90° frame rotations; orders-of-magnitude better training stability than GeoRT across initializations.
• Real world: teleoperation on Flexiv Rizon 4 + Wuji Hand, task times e.g. Spray-Bottle 10.6 s, Light-Bulb 17.0 s, Steak-Shoveling 28.0 s, Small-Ball Picking 105.8 s; 8 operators of varying experience.

---

5. Significance & limitations

Significance. AnyDexRT turns the pyramid's L4 bridge from a per-hand, calibration-heavy chore into a cheap, hand-agnostic module — the enabling piece for scaling human data to any 5-finger hardware. Local-motion preservation + contact-aware pinch refinement are the transferable ideas.

Limitations (authors').

1. Still needs a few human-guided anchors — future work: automate anchor selection or adapt online from operator feedback.
2. Contact refinement is pinch-only — broader contact-rich behaviors need richer contact models.
3. Evaluated mainly as teleoperation — training downstream manipulation policies on AnyDexRT-collected data is not yet validated (the true test of its data-collection value).

---

6. Links

• Paper: arXiv 2607.08341
• Pyramid placement: L4 (the retargeting bridge) — Dexterous-Hand Data Pyramid
• Counterpart source-data: Do As I Do · cross-hand canonicalization: One-Hand · glove capture: DexUMI
• Dexterous Manipulation · Cross-Embodiment

← Back to Reviews · Home