How To do position retarget in real world?

[BIT-wh]

Hi:
it's a awesome work. I have a leap hand in my Lab. I want to use my origin laptop's camera to capture my hand's finger pose, then use dex-retargeting to retarget my finger pose to my leap hand. However my os is Win11. Can I use it? How can I do it? Maybe my question is so easy, but I really need your help. Thank you so much : )
(I am now able to achieve joint control of dexterous hands, but I dont know how to deploy your codes) :)

[yzqin]

Your use case is perfect for this code. Check out the vector_retargeting examples/tutorials and follow the step-by-step tutorial to learn how to retarget a hand, like a Leap Motion hand, using a laptop's camera. The examples and tutorial will guide you through the process. If you have any questions, feel free to reach out. Happy coding!

[BIT-wh]

sorry to bother again. When I install the dex-retargeting, there are some mistakes about cmeel_*, eg. cmeel_boost. When I tried to solve it, I realized that cmeel-boost doesn't seem to support windows?

[yzqin]

First, the issue you met should be related to the installation of pin (Pinocchio) package.
Based on the information provided in the Pinocchio documentation, it appears that Windows should be supported, but the installation process can be a bit tricky.

According to user feedback from previous issues, using conda to install Pinocchio instead of pip seems to be a more reliable solution on Windows.

Here's a suggestion for installing the library on Windows:

1. Create and activate a conda environment.
2. Install Pinocchio using conda-forge:

   conda install pinocchio -c conda-forge

3. Install the dex_retargeting library using pip:

   pip install dex_retargeting

For more detailed instructions and additional information, please refer to the official Pinocchio tutorials, which provide comprehensive guidance on getting started with Pinocchio.

[BIT-wh]

Your use case is perfect for this code. Check out the vector_retargeting examples/tutorials and follow the step-by-step tutorial to learn how to retarget a hand, like a Leap Motion hand, using a laptop's camera. The examples and tutorial will guide you through the process. If you have any questions, feel free to reach out. Happy coding!

Thanks for your reply, I have now installed an ubuntu 20.04 VM and can now successfully install dex_retargeting. I went through the vector_retargeting examples/tutorials and only saw how to retarget the hand and render it into sapien, but my computer is not good enough to run sapien. I would like to retarget directly to a realistic leaphand, but i didn't see how to do it?

[yzqin]

The SAPIEN example demonstrates how to control a simulated robot to visualize retargeting results, as most users don't have access to a real robot hand. Real-world robotics code heavily depends on hardware-specific drivers, which is beyond the scope of this repo. This repo focuses solely on the retargeting computation algorithm.

The retargeting algorithm outputs joint positions for each robot joint. To control your robot, use these joint positions as input for your robot's joint position controller, whether in simulation or the real world.

[BIT-wh]

Thank you for your reply, I have solved the problem above. Now the query I am facing is that qpos is the joint angle obtained by the RETARGET, how do the 16 joint angles here correspond to the joint nodes of the mediapipe? Or how does it correspond to the joints of leaphand? (because I want to apply it on the actual leaphand)
Overall, I would like to know the joint correspondence of qpos. Looking forward to your reply, thanks!

[yzqin]

This is a good question!

I have updated the readme for joint orders, you can also use a similar method to handle the joint order mapping between retargeting and leap hand driver: README HERE

[BIT-wh]

Thank you so much! What an amazing project.

I have now successfully applied it to my leap hand. May I ask, is the repositioning of the robotic arm ready to be open source? Or any recommended open source projects? Our group's robotic arm is arriving in about two weeks, and I'd like to try to put the leap hand on the end of the robotic arm as an actuator. If you can help me, I will be thankful to the best of my ability.

You are such a helpful and selfless expert!

[yzqin]

Hi @BIT-wh

I would like to keep this repo simple and focused only on hands. For motion control of robot arm for teleoperation, you can check our other project here: https://github.com/Dingry/BunnyVisionPro

It is targeted at bimanual robots with two arms and two hands.

[BIT-wh]

Thank you for your generous reply! Bunny-VisionPro is also an amazing project!

I looked at your Bunny-VisionPro project and it seems like I have to have Apple VisionPro. But I only have my laptop's camera and an rgb-d camera. I don't seem to be able to deploy Bunny-VisionPro.

Possibly do you have any other recommendations for a lightweight open source project for robotic arm wrist retargeting?