GSoC: Create Python bindings for MoveIt 2

[peterdavidfagan]

Hi there,

I am currently working on creating Python bindings for MoveIt 2 as part of my GSoC project.

The goals of my project are to:

1. Redesign and implement a Python API for MoveIt 2
2. Create a tutorial demonstrating the usage of this API for learning-based approaches
3. Implement feature requests that come from the MoveIt community

I will be adding further details to this issue page and my project page over time. Feel free to comment below or link feature requests that you feel may be relevant to this project.

Have a great rest of the day.

[AndrejOrsula]

Hello @peterdavidfagan,

Great to have someone working on this!

I also wanted to interface MoveIt 2 via Python for RL in the past, so I made pymoveit2 (previously under ign_moveit2_examples). It does not provide proper Python bindings, but rather a basic Python interface via ROS 2 middleware interface. Maybe there is something of relevance, so I decided to share it here.

---

Feel free to comment below or link feature requests that you feel may be relevant to this project.

If it is within the scope, here are some features that I would appreciate:

• A wrapper around MoveIt Servo (example).
• A simplified interface for grippers (support grippers with GripperCommand and JointTrajectoryController interfaces).
• The ability to reset the controllers after resetting/randomising robot joint state in paused/playing simulation. I found some approaches, but they are quite "hacky".
• Fully typed function arguments and return values for linting and auto-completion.
• Support for "overloaded" function arguments of different data types. E.g. the ability to pass either Iterable (List/numpy.ndarray/...) or geometry_msgs.msg.Point while setting the position goal. I am not sure how Type Adaption added to humble for rclcpp could fit into this context with rclpy.

[peterdavidfagan]

Hello @peterdavidfagan,

Great to have someone working on this!

I also wanted to interface MoveIt 2 via Python for RL in the past, so I made pymoveit2 (previously under ign_moveit2_examples). It does not provide proper Python bindings, but rather a basic Python interface via ROS 2 middleware interface. Maybe there is something of relevance, so I decided to share it here.

Feel free to comment below or link feature requests that you feel may be relevant to this project.

If it is within the scope, here are some features that I would appreciate:

• A wrapper around MoveIt Servo (example).
• A simplified interface for grippers (support grippers with GripperCommand and JointTrajectoryController interfaces).
• The ability to reset the controllers after resetting/randomising robot joint state in paused/playing simulation. I found some approaches, but they are quite "hacky".
• Fully typed function arguments and return values for linting and auto-completion.
• Support for "overloaded" function arguments of different data types. E.g. the ability to pass either Iterable (List/numpy.ndarray/...) or Pose while setting the pose goal. I am not sure how Type Adaption added to humble for rclcpp could fit into this context with rclpy.

Thanks @AndrejOrsula,

The Python library you already created looks great, it is certainly relevant as developing user friendly interfaces is an important part of this project.

I appreciate the feature requests, I will take these into account and work towards including these features. Looking forward to discussing further, I'll share design details here and on Discord while progressing with development. It would be great to receive your feedback.

[tlpss]

Hi @peterdavidfagan

Very glad to see someone is going to work on this!

I have also created a "python bridge" for some features of Moveit2 that I wanted to use to control our UR robots, but the lack of an easy-to-use and well documented python interface to Moveit2 has been the main bottleneck for our research group to start using Moveit2 in our research, as it creates a barrier for researchers that are not experienced with ROS.

Things that we would be looking for include:

• standard aspects such as access to the planning interface and scene interface (the Move Group Interface basically).

• the ability to manually define waypoints and execute them through Moveit for IK and collision checks. (We often want to specify the end-effector trajectory exactly as our research group works quite a lot on deformable object manipulation, for which not only the pick and place pose are important). We have been using the Cartesian Interpolator in the Move Group Interface to achieve this but I have seen some discussions on removing this as it easily gets stuck. We would however still really like to be able to specify exact trajectories for the EEF to follow.

• The wrapper around Moveit Servo, as suggested by @AndrejOrsula. And more generally, the ability to specify goal poses, waypoints,... in a more pythonic format (numpy arrar/list) instead of having to create a Twist message. (Numpy arrays are typically what you want to use for Data-driven action command interfaces).

• Best practice examples on how to use the python interface of Moveit2 for "learning-based" approaches. Basically: how to start when you want to use Moveit2 for training a robot to perform a particular task. Lots of simulators are starting to include ROS interfaces (Unity, ISAAC,...) which has the potential to use the same ROS stack in a simulator and for real robots, but there are no well-documented examples on how to get started with this afaik. For researchers without ROS experience, this would be essential. Some (simulator-independent) Moveit2-Gym interface template would be of great value. An example of how to use moveit servo if you want fine-grained EEF control for the policy as wel as an example of using Moveit for interactively learning grasp poses (and then executing them with Moveit) would be very interesting.

[peterdavidfagan]

Hi @tlpss,

Thank you for these points. I have taken note of these and will look to incorporate the outlined functionalities into the Python API that gets created. It would be great to have you test the initial library that results from this project.

Currently we are deciding what to bind and how the API should be structured. At the moment it looks like the functionality of the moveit_cpp library will be binded. The overall API design is still not finalized. I hope to provide demos using Isaac Sim so maybe this will be helpful for your third point.

[tlpss]

It would be great to have you test the initial library that results from this project.

Feel free to ping me, I would be interested to take a look.

Best of luck with the project!

[peterdavidfagan]

Hi everyone,

The development of a library that is called moveit_py has begun; initially this library will consist of binding functionalities of moveit_cpp and creating useful python interfaces using these functionalities. As additional functionalities are added to moveit_cpp these will also be considered for moveit_py. For those who are interested in tracking the development work for this library the moveit_py branch on my fork should remain up to date.

I also have another repository moveit_py_example which will demonstrate examples of using the library. There's only one example there for the moment but I will be adding many more in the coming weeks (including examples which leverage isaac-sim).

Finally I have started a project to track development work which can be found here. I'll be adding tasks to this soon.

Other than this update, I may share some of the interfaces in this thread to get your feedback on these so expect further updates in the coming weeks. Have a great rest of the day.

[marc-wittwer]

Awesome. @peterdavidfagan I will follow your progress 🙌

The python interface will lower the entry barrier for our research group as well.

We will mainly use:

• Setting pose goals
• Setting collision objects into the planning scene
• Setting orientation constraints of the end effector while following a trajectory (e.g. moving a cup with liquid inside)

Best of luck with the project also from my side!

[peterdavidfagan]

Thanks @marc-wittwer, I am really looking forward to the first release of the library. I'll be sure to take into account the use cases you mentioned above. It is great to have so much interest in the library already thank you for sharing the above.

[larrylu0426]

Hi everyone, I'm working on a project about Panda's arm, and it depends on Python3 and ROS2/Moveit2/Ubuntu20.04. So I refactor the planning interface module with pybind11 on the foxy branch. Here's my repo link: https://github.com/larrylu0426/moveit2. I keep all C++ interfaces and adapt the original Python interfaces. I have applied it in my project. So you can use it and give me many reviews :).

MoveIt Command is ready to refactor.

[peterdavidfagan]

Hi @larrylu0426,

The above looks great, it sounds like you are interested in the same topics as this GSoC project.

You can view the current status of the development here and a basic example of its usage here. I am starting to work on bindings for the planning_scene_interface as part of the package linked above. The bindings you created with pybind11 are similar to the ones I am developing, except the bindings I am creating lie in a separate package and there are other minor difference (e.g. how ros messages are being handled).

If you are up for it we could hop on a call and discuss these bindings? I believe it may make sense to incorporate some of your code into the package I am creating as part of my GSoC project.