Multimodal iCub Simulation in MuJoCo (Vision, Touch, Proprioception)

[2103simon]

Hello @everyone,

There has been great work bringing iCub into MuJoCo already. Building on the work of @fedeceola, I’ve extended the simulator to include full-body tactile sensing (taxels) and optional event-driven outputs:

https://github.com/event-driven-robotics/neuromorphic_body_schema

Key features:

• Stereo vision (standard or event-based camera output)
• Joint encoders (default)
• Full-body skin (taxel-level touch sensing)
• Optional event-driven streams for vision, touch, and proprioception (AER)

The simulator is designed as a flexible platform for multimodal robotics, including tasks such as tactile interaction, reach-and-grasp, manipulation, and sensorimotor learning.

This is still work in progress. Known limitations include sim-to-real gaps (e.g., PID tuning, skin response magnitude, and cross-talk).

I’d really appreciate feedback from the community:

• Would you use this for your research or applications?
• Which use cases or features would make it more relevant for you?
• What should be prioritized next?

If there’s interest, I’m considering scaling up development effort in this direction.

Thanks!

[pattacini]

Awesome work @2103simon!

We may consider these steps for the near / mid terms:

• Include it in the superbuild
• Possibly port the simulator to robotology
• Collaborate on churning out some useful YARP wrappers

[2103simon]

Including a YARP wrapper might allow the use of the iCub skin GUI instead of my custom implementation. Not sure if this would also work for the motor GUI.

We might want to consider keeping it simple tho and potentially have YARP only optional to keep the entry barrier as low as possible. What do you think?

[pattacini]

That's a good option indeed 👍🏻

[AntonioViscomi]

Hi @2103simon,
thank you for sharing your work with everyone!

Recently, I had the opportunity to generate a simulation model of iCub directly starting from the CAD data.
I also had the pleasure of speaking with @fedeceola, who shared his experience in producing the model referenced in your original post (actually, I integrated the hand model developed by Federico into the final MJCF I obtained).

Through our discussions, we realized that we had both observed the model “exploding” at simulation startup. After some investigation and backtracking, I was able to identify and resolve the underlying issue. As a result, we now have a simulation model that is directly derived from the CAD, without requiring meshes to be split to avoid collisions.

It is worth noting that this model has not yet been validated against a physical iCub, which would certainly be an interesting. That said, I would be happy to help if needed!

[2103simon]

That sounds great! I think it should be straight forward to include the new stl files and re-arrange the taxel to match to the new bodies (as the taxel positions are derived from CAD-I expect them to fit even better on model derived from CAD too).

Do you have the files uploaded somewhere? If you share the xml file with me, I can start from there and re-work the taxel fitting followed by the creation of the new xml file including the tactile sensors!

[AntonioViscomi]

Hi @2103simon
At the moment these files are stored in private repository, but we can think about proceeding together in some way 😄

[2103simon]

Potentially while moving it to robotology?

[AntonioViscomi]

cc'ing @pattacini and @Nicogene because they might know

[pattacini]

I reckon this can be easily discussed in a dedicated in person meetup.

[fedeceola]

Adding here the link to the repo with my original implementation.

https://github.com/hsp-iit/rl-icub-dexterous-manipulation

Besides the implementation of the MuJoCo simulator, in the repo you can find the code for training RL policies as described in the G-PAYN and RESPRECT papers. The code is meant to be YARP-free (robotogy-superbuild is installed to use the iKin IK solver, which turned out to be the best option for iCub), but it also provides an integration with YARP for training RL policies on the real robot.

[pattacini]

robotogy-superbuild is installed to use the iKin IK solver, which turned out to be the best option for iCub

This comment made my day 😃
It’s rewarding to see it’s still the 'go-to' choice for the iCub's kinematics...