A Hand with Places to Be: Teaching an Anthropomorphic Hand to Walk with Isaac Lab

[amrkzp]

Hi everyone,

We used NVIDIA Isaac Lab to train task-specific reinforcement-learning policies for an anthropomorphic 20-DoF hand that supports its own weight on its fingers.

The same hand can crawl untethered, steer, recover from falls, push a cube to targets, and press keyboard keys without visual feedback. The policies run onboard on a Raspberry Pi Zero 2 W, with an IMU and battery mounted on the hand.

The main challenge is contact-rich: every finger used for a task is also one less contact supporting the hand. We first calibrated the simulator against hardware measurements of stiffness, friction, and delays, then applied domain randomization before transferring the policies to the real system.

In the hardware demos, the hand crawls across a range of indoor and outdoor surfaces and can recover from falls.

Video
Project page
Paper

It still has a few quirks, including a preference for turning one way. If you’re working on Isaac Lab, contact-rich tasks, or sim-to-real transfer, we’d be happy to hear your thoughts.