Add our MoveIt 2 roadmap, as discussed at MoveIt Workshop

[davetcoleman]

I keep getting questions about this, so it needs to be on website

[mamoll]

Milestone 1: replace "ament" with "colcon"?
Milestone 2: it seems odd to put a number on how fast the global planner can run. This will be very much problem specific. Maybe call it "semi-local" to distinguish from local, but clarify that it is not a truly global planner which is just not possible in a realtime setting.

Determinism: why single out BIT*? I'd perhaps rename this whole category to "Improved Path Quality." While OMPL may not be the best fit for some common settings, it is still useful as a fallback for hard problems, where it tends to be more efficient than alternatives. Also, completeness/correctness are sometimes hard to guarantee with other planning approaches. OMPL can do Cartesian planning, but is not deterministic. One of the suggestions from the workshop was to allow multiple planner plugins in the same move group. If this could be combined with running multiple planners from different plugins in parallel, then I think you could significantly improve path quality on most common problems:

• In easy cases, you can get by with an IK-based approach
• In slight harder cases, most of the time you can probably get by with something like TrajOpt
• In hard cases, OMPL can provide completeness and (with enough time) optimality.
  You don't have to pick a "winner"; run them all in parallel and whoever produces a solution with acceptable quality first gets selected.

Maybe change the Cartesian Planner bullet and subbullet to:
"Fully-featured Cartesian planning:
Interface for specifying commonly used trajectory constraints"
There is no need to trash Descartes before we have a working implementation. It would be good to separate the formulation of constraints from the planner. Given the constraints, there are many ways to solve it. Perhaps someone can implement a greedy, fast (but incomplete) IK-based approach. A version of TrajOpt with the right cost function is another option. One nice thing of using OMPL for this is that costs and constraints are defined as separate concepts. You can have a cost map defined for constrained planning (e.g., based on manipulability) and do asymptotically optimal constrained planning without any change to, say, RRT*.

Under Machine Learning, I'd not say nn-based motion planning. It just isn't practical on its own. However, it may be very useful to learn local motion primitives, so maybe focus on that:

• Use deep learning to generate dynamic motion primitives for contact-rich interactions
• Use machine learning for grasp synthesis
• Visual servoing in challenging conditions (occlusion, variable lighting)

[mamoll]

For completeness, here are some notes from discussions at the roadmapping session of the MoveIt workshop (thanks in part to @mikeferguson and @marip8):

Core Planning

• The main concerns related to integration between planning and control:
  • Planning with dynamics limits, torque limits, or end-effector payload
  • Real-time collision checking/avoidance & dealing with moving objects while respecting joint limits
  • Closed-loop plan execution (maybe synthesize a controller along with a path/trajectory?)
• Better support for Cartesian planning and constrained planning more generally.
• Determinism and predictable behavior are also high on the wish list.
  • Personally, I think multiple parallel planning pipelines (as described in the previous comment) is the way to go.
• Planning as a service to allow planners to be written in any language (this probably belongs in the next group).
• More flexibility is needed in how planning is performed:
  • Per-planner planner request adapters
  • Allow multiple planner plugins in the same group
  • Multi-robot cooperative planning. This can mean many different things:
    • Centralized planning for multiple robots
    • Decentralized planning and using scheduling to interleave motions as much as possible
    • Reactive planning
    • Etc.
• It would be nice to get search-based planning working again

API / High-level interface

• Concerns
  • How to provide appropriate Python support
  • Improve modularity with more plugin interfaces to core components/functionality
  • Improve the layering of abstraction from core components to higher level interfaces
• Action items
  ​ - Look into PyBind 11 for Python bindings
  ​ - Look into replacing plugins with ROS2 component nodes
  • Generate high-level architecture diagram illustrating how the components of MoveIt are structured and interact with each other
  • Deprecate the moveit_commander and use a more "stand-alone" Python package with a high-level API to call move_group services
  • Provide Python function for directly planning/moving to a Cartesian position (possibly already exists?)
    ​

Applications / Industry / Specialties

• Industry Priority 1 - Broadly described as better environment modeling/integration
  • Support for dynamically updated robot model (for instances, supporting tool changers, especially when the end effectors are actuated)
  • Support for multiple robots in a scene (and actually being able to plan with multiple robots at the same time)
  • Support for external IO integration (ties back to the tool changer aspect, first class support for integration with typical industrial IO)
• Industry Priority 2 - Broadly described as "more moveit task constructor features"
  • Support for synthesizing a sequence of tasks (large portions of this probably already exist in MTC - but aren't widely understood yet)
  • Exposing motion primitives through action interfaces (for instance, action interface to have MTC execute a particular task. At the far end of this, having MTC basically replace move_group so that we have truly dynamic/programmable/extensible ROS interface to MoveIt)
  • Process planning and simulation (Gijs can certainly expand on what this means -- I didn't capture all of that)
• ROS2 Priority 1 - Environment Representation / API
  • Look into unifying environment representations between MoveIt, ROS2 Navigation and Autoware (e.g. we have Octomap+links, Navigation has cost maps, etc - they all do similar things, lots of code being duplicated)
  • More sensors supported in planning scene, instead of just pointclouds
  • Add scene graph support to represent the relationship between objects
• Community Priority 1 - Add support for Convex Collision checking, to support things like TrajOpt
• Visual Impact 1 - Add support for mobile manipulation
  • Add planning support for diff drive robots
  • Probably need to fix bugs related to assumptions of fixed frame
  • Consider other/expanded environment representations (e.g. base_link-centered 3d-costmap instead of world-frame-centered Octomap)

@mlautman may have some additional notes from the roadmapping session.

[v4hn]

Thank you so much @mamoll @mikeferguson @marip8 for the additional notes!

As I sadly did not attend the workshop, here is some feedback from my side, too.

I highlighted everything directly related to this pull-request in bold.

Milestone 2

The following two things already exist and work in current MoveIt. It does not make sense to me to put them on the roadmap without more specific things we want to improve or rewrite there:

• Octomap updates via topics. zero-copy (if feasible) and improved runtime for map updates would of course still make sense. MoveIt's octomap usage is clearly not the fastest out there.
• Preempt motion if new collision is detected. isRemainingPathValid does exactly that and is implicitly used for every single trajectory we execute and for every change in the planning scene the system observes.

Lifecycle Management of MoveIt nodes

Supporting a simple STARTING_UP, RUNNING, SHUTDOWN transition for move_group is simple enough, but is this what the point is supposed to mean?
Usually PickNik/industry argues in favor of not using standalone nodes for MoveIt.
If people are supposed to use classes in their own node, I don't see what we can/should do to support lifecycle management of their nodes.

Replace pluginlib with components

In the last maintainer meeting where we discussed this point, it was very fuzzy what this could look like.
Also, there was no agreement at all among the maintainers whether this would be a good idea, because it would make tracing bugs much harder if no backtraces in clear call-hierarchies are possible anymore.
Were there more insights / some consensus on possible approaches discussed during the workshop?

Industry Priority 1

Support for dynamically updated robot model (for instances, supporting tool changers, especially when the end effectors are actuated)

This is not just a matter of MoveIt, but the general question on how this is should be implemented across ROS2. Still, MoveIt could pioneer a new "standard" approach for this (and extend other packages to support it).

Support for multiple robots in a scene (and actually being able to plan with multiple robots at the same time)

This is already possible right now by having a big urdf with all robots and many people did it in the past. What exactly is new/requested here?
If multiple separate URDF models in one scene are targeted, I would second this request, but this also leads to problems such as

• how/where do you define the geometric relation between the models?
• how/where are joint states for all models reported and merged?
• Should MoveIt support any number of CurrentStateMonitors? Should we generalize all interfaces to support any number of robots?

Industry Priority 2 - Broadly described as "more moveit task constructor features"

Of course, it is quite frustrating to me to see this item here without having been around.
I guess I will talk to @rhaschke about these things over the next two weeks? 😁

More sensors supported in planning scene, instead of just pointclouds

What does this mean? Sensors are not at all supported in the PlanningScene, geometric measurements are. What alternative measurements are relevant to people?

Add scene graph support to represent the relationship between objects

This would make for a great "Milestone 2/3" entry, @davetcoleman ! It's an often-discussed shortcoming of MoveIt as of today and it's clearly achievable.

Add support for Convex Collision checking, to support things like TrajOpt

@davetcoleman This, too, would make a wonderful MoveIt2 milestone entry. This request still entails a way to describe meshes as convex, which is not currently available in robot descriptions.

Visual Impact 1 - Add support for mobile manipulation

As I'm probably one of the very few people who used this in recent years, here's my experience with the current state:

• The virtual joint can be used to define a single holonomic joint for driving or flying.
• The whole concept of "workspace" in MoveIt currently limits the positional range of these (otherwise unlimited) joints.
• With a holonomic robot base, e.g., the PR2, planning arm&base motions together works. The results are trajectories that, for example, move the robot around while grasping something.
• The trajectories can, however, not easily be executed as there are usually no base controllers that can receive a MultiDOF trajectory and actuate it. moveit_simple_controller_manager does not support any ROS interface to forward such a trajectory either and needs to be extended/replaced.
• especially with such base motions, motion planning should typically be biased to prefer moving arms over the entire base. This is not trivially available in the pipeline at the moment, although OMPL probably has some support for it (e.g., optimizing for minimum effort)

Add planning support for diff drive robots

This boils down to implementing a new joint type DifferentialJoint and support it throughout the pipeline.

[mamoll]

I would like to add my vote to adding support for scene graphs. This would it make it much easier to integrate MoveIt with task planners or combined task & motion planners. It would be pretty cool if complex tasks could be expressed via, say, PDDL and a feasible motion plan could automatically be generated,

Also, as a side note, I get the impression that some people think convex collision checking (similar to what is used in TrajOpt) is the same as continuous collision detection. It is not. It is a heuristic way to approximate swept volume. If I have 2 configurations of a 1-dof revolute link rotated 180 degrees, then the convex hull of the two configurations is a very bad approximation of the swept volume. So convex collision checking is not eliminating the need to choose collision checking resolution appropriately.

[davetcoleman]

I'm a bit annoyed by the long winded responses from me trying to share a simple roadmap. And I'm afraid I don't have time to spend hours debating the subtleties of mostly un-funded, theoretical work we may or may not do in the future. The goal of this post was to quickly provide a vision for other potential funding sources. The content is based on:

• The plan presented at a MoveIt maintainer meeting several months ago
• The plan further debated as a developer team within PickNik
• The plan that has been approved in our ROSin grant
• Survey data from this year

I'll try to incorporate some feedback, but really just need to get something shared online for potential contributors:

Milestone 1: replace "ament" with "colcon"?

Both are used in ROS 2 - Dirk made it very confusing. Ament is the "Build Type"
https://index.ros.org/doc/ros2/Tutorials/Colcon-Tutorial/

Milestone 2: it seems odd to put a number on how fast the global planner can run. This will be very much problem specific. Maybe call it "semi-local" to distinguish from local, but clarify that it is not a truly global planner which is just not possible in a realtime setting.

Its just a rough approximate goal that is very very tangible / understandable. The global planner doesn't have to be realtime. Not providing a concrete target number makes the milestone way more ambiguous. This is meant to inspire us.

Determinism: why single out BIT*?

It was a passing suggestion of @henningkayser

I'd perhaps rename this whole category to "Improved Path Quality." While OMPL may not be the best fit for some common settings, it is still useful as a fallback for hard problems, where it tends to be more efficient than alternatives. Also, completeness/correctness are sometimes hard to guarantee with other planning approaches. OMPL can do Cartesian planning, but is not deterministic. One of the suggestions from the workshop was to allow multiple planner plugins in the same move group. If this could be combined with running multiple planners from different plugins in parallel, then I think you could significantly improve path quality on most common problems:

In easy cases, you can get by with an IK-based approach
In slight harder cases, most of the time you can probably get by with something like TrajOpt
In hard cases, OMPL can provide completeness and (with enough time) optimality.
You don't have to pick a "winner"; run them all in parallel and whoever produces a solution with acceptable quality first gets selected.

Added run in parallel

Maybe change the Cartesian Planner bullet and subbullet to:
"Fully-featured Cartesian planning:
Interface for specifying commonly used trajectory constraints"

We need more than an interface though...

There is no need to trash Descartes before we have a working implementation.

We do have many working implementations with Descartes and have it somewhat integrated into MoveIt, but in the wrong way. Reworded to "Similar to Descartes but fully integrated as MoveIt planner"

It would be good to separate the formulation of constraints from the planner. Given the constraints, there are many ways to solve it. Perhaps someone can implement a greedy, fast (but incomplete) IK-based approach. A version of TrajOpt with the right cost function is another option. One nice thing of using OMPL for this is that costs and constraints are defined as separate concepts. You can have a cost map defined for constrained planning (e.g., based on manipulability) and do asymptotically optimal constrained planning without any change to, say, RRT*.

Discussions like these are outside the scope of a high level roadmap, especially for milestones that we have no idea when we'll be able to get to them.

Under Machine Learning, I'd not say nn-based motion planning. It just isn't practical on its own. However, it may be very useful to learn local motion primitives, so maybe focus on that:

Its the last milestone because its the lowest priority, but its an important buzzword to acknowledge its something we're thinking about and that it will likely, someday, be the dominate approach IMHO.

Use deep learning to generate dynamic motion primitives for contact-rich interactions
Use machine learning for grasp synthesis

Added.

Visual servoing in challenging conditions (occlusion, variable lighting)

What does this have to do with ML?

[davetcoleman]

@mamoll here is the source spreadsheet with the roadmap. Please leave any further comments directly in there for easier collaboration. Again though, we need this published for MoveIt 2 funding I'm very actively working on right now post-ROSCon discussions. Please be quick.

@v4hn I also added you as a review of the spreadsheet:
https://docs.google.com/spreadsheets/d/1lyLJBkPqtwdDkBAJqNf5oTO0D1cjGrO6SVM-R0yf9cY/edit#gid=1137912102

@mamoll When you are satisfied, could you please re-add screenshots of that document into this PR, and add your consolidated / summarized notes from the MoveIt Workshop to the bottom of this new webpage? I appreciate you taking the lead on this PR.

[v4hn]

Also, as a side note, I get the impression that some people think convex collision checking (similar to what is used in TrajOpt) is the same as continuous collision detection. It is not.

Thank you for pointing out this overlap, it actually made me aware of a different one.
I was talking about supporting convex meshes as a separate collision body type.
This has nothing to do with your understanding of convex collision checking, which is a heuristic for checking collisions.
Both aspects were discussed in the past, so I'm unsure which one was meant in your original notes.

[v4hn]

@mamoll I added comments to the spreadsheet, please have a look.
The final layout seems to be on you now :)