Design docs for manipulation #226
Conversation
Original draft for the manipulation feature implementation. TODO: no architecture diagram was done yet. Signed-off-by: Jose Luis Rivero <jrivero@osrfoundation.org>
| @@ -0,0 +1 @@ | |||
| (TODO) | |||
There was a problem hiding this comment.
This can be added in another PR
There was a problem hiding this comment.
Looks like a duplicate of architecture.md
There was a problem hiding this comment.
sorry for that. Removed in 6b5e82d
|
@j-rivero I remember we spoke about the architecture. Are you working on it or should I go ahead? |
|
@j-rivero Is this PR going to be ready for review? |
I think that is mostly ready with the exception of the architecture. I can spend some time trying to create but we can probably go faster if someone with more experience in O3DE and the Gem code base create the initial draft. |
Signed-off-by: Jose Luis Rivero <jrivero@osrfoundation.org>
There was a problem hiding this comment.
These design docs are a great starting point for Manipulation in O3DE!
After internal discussion at Huawei, we came up with a bunch of minor editorial suggestions and a more fundamental question about the intended scope of the manipulation feature.
While the current architectural analysis/proposal is preliminary, it appears to be quite focused on open-loop manipulators / robotic arms (as is URDF & MoveIt). In terms of industrial applications, we note that many commercially employed manipulators (logistics, pick & place, farming, ...) are parallel robots (e.g. delta robots), which are closed-loop manipulators. This may be due to their comparatively low cost / high speed / high accuracy.
While the existing popular toolset for manipulation around MoveIt is a great starting point for the open source community and outreach, e.g. to academic partners, the requirements of many industrial users may not be met.
Thus, the integration, simulation, and control of parallel robots should probably also be considered in the architectural design of the Manipulation-related features, especially given that the first design goal raised is the highly customizable support of a variety of robots/manipulators.
| ## Design considerations | ||
|
|
||
| ### Implementation in O3DE | ||
|
|
||
| ### Use of MoveIt 2 | ||
|
|
||
| [MoveIt 2](https://moveit.picknik.ai/humble/index.html) is the main motion | ||
| planner framework used by ROS 2. There are many ways of interacting with it. In | ||
| this design, the `ROS2ManipulationRequester` component should act as the | ||
| interface for accepting inputs in the form of request for manipulation and call | ||
| MoveIt 2 ([see the code needed to implement a Pose for the end | ||
| effector](https://moveit.picknik.ai/humble/doc/tutorials/your_first_project/your_first_project.html)) | ||
| or any other motion planner. | ||
|
|
||
| The `ROS2ManipulationRequester` component should also take care of controlling | ||
| the status of the request and provide internal O3DE Gem information on the | ||
| status of the execution. | ||
|
|
||
| ### Use of controllers: ros2_control and native O3DE | ||
|
|
||
| The integration of the [ros2_control](https://github.com/ros-controls/ros2_control) into the | ||
| architecture have some difficulties: | ||
|
|
||
| - controllers are typically not modular in a sense that they handle everything | ||
| (parameters, communication and computations). | ||
| - we would like to handle some of these elements separately for more | ||
| flexibility and better UX. | ||
| - some of their design is not as flexible (likely since they have been only | ||
| used with Gazebo so far). | ||
| - for example, we would like to handle parameters in the Editor, without | ||
| running publishers/subscribers already. | ||
|
|
||
| The benefits are important though. It is valuable to reuse implementations | ||
| which are tried and tested, and benefit from the joint effort of the community. | ||
| There should be the option of using native O3DE controllers if there is a good | ||
| reason to do so like big performance improvements or much better integration in | ||
| the UI. |
There was a problem hiding this comment.
Similar to above, I am strongly in favor of migrating this section to architecture.md
| ## Design considerations | |
| ### Implementation in O3DE | |
| ### Use of MoveIt 2 | |
| [MoveIt 2](https://moveit.picknik.ai/humble/index.html) is the main motion | |
| planner framework used by ROS 2. There are many ways of interacting with it. In | |
| this design, the `ROS2ManipulationRequester` component should act as the | |
| interface for accepting inputs in the form of request for manipulation and call | |
| MoveIt 2 ([see the code needed to implement a Pose for the end | |
| effector](https://moveit.picknik.ai/humble/doc/tutorials/your_first_project/your_first_project.html)) | |
| or any other motion planner. | |
| The `ROS2ManipulationRequester` component should also take care of controlling | |
| the status of the request and provide internal O3DE Gem information on the | |
| status of the execution. | |
| ### Use of controllers: ros2_control and native O3DE | |
| The integration of the [ros2_control](https://github.com/ros-controls/ros2_control) into the | |
| architecture have some difficulties: | |
| - controllers are typically not modular in a sense that they handle everything | |
| (parameters, communication and computations). | |
| - we would like to handle some of these elements separately for more | |
| flexibility and better UX. | |
| - some of their design is not as flexible (likely since they have been only | |
| used with Gazebo so far). | |
| - for example, we would like to handle parameters in the Editor, without | |
| running publishers/subscribers already. | |
| The benefits are important though. It is valuable to reuse implementations | |
| which are tried and tested, and benefit from the joint effort of the community. | |
| There should be the option of using native O3DE controllers if there is a good | |
| reason to do so like big performance improvements or much better integration in | |
| the UI. |
| @@ -0,0 +1 @@ | |||
| (TODO) | |||
There was a problem hiding this comment.
Looks like a duplicate of architecture.md
| @@ -0,0 +1 @@ | |||
| (TODO) | |||
There was a problem hiding this comment.
Content migrated from analysis.md
| (TODO) | |
| ## Design considerations | |
| ### Implementation in O3DE | |
| ### Use of MoveIt 2 | |
| [MoveIt 2](https://moveit.picknik.ai/humble/index.html) is the main motion | |
| planner framework used by ROS 2. There are many ways of interacting with it. In | |
| this design, the `ROS2ManipulationRequester` component should act as the | |
| interface for accepting inputs in the form of request for manipulation and call | |
| MoveIt 2 ([see the code needed to implement a Pose for the end | |
| effector](https://moveit.picknik.ai/humble/doc/tutorials/your_first_project/your_first_project.html)) | |
| or any other motion planner. | |
| The `ROS2ManipulationRequester` component should also take care of controlling | |
| the status of the request and provide internal O3DE Gem information on the | |
| status of the execution. | |
| ### Use of controllers: ros2_control and native O3DE | |
| The integration of the [ros2_control](https://github.com/ros-controls/ros2_control) into the | |
| architecture have some difficulties: | |
| - controllers are typically not modular in a sense that they handle everything | |
| (parameters, communication and computations). | |
| - we would like to handle some of these elements separately for more | |
| flexibility and better UX. | |
| - some of their design is not as flexible (likely since they have been only | |
| used with Gazebo so far). | |
| - for example, we would like to handle parameters in the Editor, without | |
| running publishers/subscribers already. | |
| The benefits are important though. It is valuable to reuse implementations | |
| which are tried and tested, and benefit from the joint effort of the community. | |
| There should be the option of using native O3DE controllers if there is a good | |
| reason to do so like big performance improvements or much better integration in | |
| the UI. | |
| ## Architecture Proposal | |
| Diagram of input/outpus and data flows using ROS 2: | |
|  | |
| To implement these set of actions, the O3DE simulator should have at least: | |
| * The capacity of publish information about the state of the joints (**RO2JointStatePub** component): | |
| Include the needed information from O3DE simulation related to the joints of | |
| the simulated robot in `sensor_msgs/JointState.msg` and publish it in the `/joint_states` | |
| topic. This is required to interact with ROS 2 packages (such as MoveIt) | |
| See [gazebo_ros_pkgs plugin for reference](https://github.com/ros-simulation/gazebo_ros_pkgs/blob/galactic/gazebo_plugins/include/gazebo_plugins/gazebo_ros_joint_state_publisher.hpp#L44-L73) | |
| * Receive orders to move the different joints (**ROS2PoseTrajectory** component): | |
| Set the trajectory of points to be followed by joints in the O3DE simulation using as input | |
| `trajectory_msgs/msg/JointTrajectory.msg` | |
| See [gazebo_ros_pkgs plugin for reference](https://github.com/ros-simulation/gazebo_ros_pkgs/blob/galactic/gazebo_plugins/include/gazebo_plugins/gazebo_ros_joint_pose_trajectory.hpp#L26-L48) | |
| * The ability to request picks or motion movements to external manipulation planners (**ROS2ManipulationRequester**): | |
| Request a motion plan to external planners given different types of inputs, | |
| a target pose for end effector or a joint state target. MoveIt `move_group_interface` interaction | |
| is an example of this. | |
| See [MoveIt 2 joint state target goal](https://github.com/ros-planning/moveit2/blob/main/moveit_ros/planning_interface/move_group_interface/include/moveit/move_group_interface/move_group_interface.h#L305-L320) | |
| or [MoveIt 2](https://github.com/ros-planning/moveit2/blob/main/moveit_ros/planning_interface/move_group_interface/include/moveit/move_group_interface/move_group_interface.h#L640-L554) | |
| * Provide data from the simulated sensors: not in the scope of this document | |
| * The design does not consider any grasping technique although external frameworks can provide this | ||
| capability. | ||
|
|
||
| | In the first iteration, only support for car-like 4-wheeled robots will be targeted. | |
There was a problem hiding this comment.
Left-over from the previous design-document? This does not make much sense for manipulators.
| | In the first iteration, only support for car-like 4-wheeled robots will be targeted. | |
There was a problem hiding this comment.
These design documents are very helpful to gain a wide view on the submodules which connected together perform the manipulation task.
For quick prototyping I have lightened the manipulation.svg diagram in order to leave only the essential modules. Moreover I specified the type of controller which could be used: indeed a velocity controller allows to control the kinematics of the robotic arm and the dynamics to be forgotten, so that to highlight the early implementation stage. Any feedback on this solution are appreciated.
| manipulation pipeline. Note that the use of controllers is optional | ||
| for quick prototyping so the manipulation planning could interact | ||
| directly with the simulator without using a controller. | ||
|
|
There was a problem hiding this comment.
When you mention a direct interaction between simulator and planner do you allude to a method to apply directly the angular position and velocity to joints? In other words a method which bypass the dynamics, acting only on kinematics?
There was a problem hiding this comment.
When you mention a direct interaction between simulator and planner do you allude to a method to apply directly the angular position and velocity to joints? In other words a method which bypass the dynamics, acting only on kinematics?
Yes, I was thinking on used directly the values of position/velocity/acceleration to inject them in the simulators to quickly test things not involving the control of dynamics or not being critical for the goal of these test.
Another possibility is to mention/change that there are fake controllers available (see ros2_control Fake Components) that are probably a better approach to bypass a controller than just skip it.
| * The ability to request picks or motion movements to external manipulation planners (**ROS2ManipulationRequester**): | ||
| Request a motion plan to external planners given different types of inputs, | ||
| a target pose for end effector or a joint state target. MoveIt `move_group_interface` interaction | ||
| is an example of this. |
There was a problem hiding this comment.
Why not to leave this task to an external ROS2 package? What are the benefits of having an internal interface which orchestrates the manipulation jobs?
There was a problem hiding this comment.
Why not to leave this task to an external ROS2 package?
The diagram could not be very clear in this area, but note that I tried to represent this by "ROS interactions" coming from SMACK/BehaviourTress used as inputs for MoveIt:
What are the benefits of having an internal interface which orchestrates the manipulation jobs?
It opens the door to use other orchestration implementations not in ROS or the use of very simple interactions done directly from O3DE.
Co-authored-by: Lars Gleim <lgleim@users.noreply.github.com>
There was a problem hiding this comment.
These design docs are a great starting point for Manipulation in O3DE!
After internal discussion at Huawei, we came up with a bunch of minor editorial suggestions and a more fundamental question about the intended scope of the manipulation feature.
Really appreciated Lars. Thanks so much.
While the current architectural analysis/proposal is preliminary, it appears to be quite focused on open-loop manipulators / robotic arms (as is URDF & MoveIt).
You are right on this since I was mainly thinking on how to use it with MoveIt.
In terms of industrial applications, we note that many commercially employed manipulators (logistics, pick & place, farming, ...) are parallel robots (e.g. delta robots), which are closed-loop manipulators. This may be due to their comparatively low cost / high speed / high accuracy.
While the existing popular toolset for manipulation around MoveIt is a great starting point for the open source community and outreach, e.g. to academic partners, the requirements of many industrial users may not be met.
Thus, the integration, simulation, and control of parallel robots should probably also be considered in the architectural design of the Manipulation-related features, especially given that the first design goal raised is the highly customizable support of a variety of robots/manipulators.
Fully agree. Please feel free to submit the changes or comment on anything inside the proposal that you want in, I think that is going to be great for the design to count with people with experience on industrial and/or closed-loop manipulators.
| manipulation pipeline. Note that the use of controllers is optional | ||
| for quick prototyping so the manipulation planning could interact | ||
| directly with the simulator without using a controller. | ||
|
|
There was a problem hiding this comment.
When you mention a direct interaction between simulator and planner do you allude to a method to apply directly the angular position and velocity to joints? In other words a method which bypass the dynamics, acting only on kinematics?
Yes, I was thinking on used directly the values of position/velocity/acceleration to inject them in the simulators to quickly test things not involving the control of dynamics or not being critical for the goal of these test.
Another possibility is to mention/change that there are fake controllers available (see ros2_control Fake Components) that are probably a better approach to bypass a controller than just skip it.
| * The ability to request picks or motion movements to external manipulation planners (**ROS2ManipulationRequester**): | ||
| Request a motion plan to external planners given different types of inputs, | ||
| a target pose for end effector or a joint state target. MoveIt `move_group_interface` interaction | ||
| is an example of this. |
There was a problem hiding this comment.
Why not to leave this task to an external ROS2 package?
The diagram could not be very clear in this area, but note that I tried to represent this by "ROS interactions" coming from SMACK/BehaviourTress used as inputs for MoveIt:
What are the benefits of having an internal interface which orchestrates the manipulation jobs?
It opens the door to use other orchestration implementations not in ROS or the use of very simple interactions done directly from O3DE.
| Diagram of input/outpus and data flows using ROS 2: | ||
|
|
||
|  | ||
|
|
||
| To implement these set of actions, the O3DE simulator should have at least: | ||
|
|
||
| * The capacity of publish information about the state of the joints (**RO2JointStatePub** component): | ||
| Include the needed information from O3DE simulation related to the joints of | ||
| the simulated robot in `sensor_msgs/JointState.msg` and publish it in the `/joint_states` | ||
| topic. This is required to interact with ROS 2 packages (such as MoveIt) | ||
| See [gazebo_ros_pkgs plugin for reference](https://github.com/ros-simulation/gazebo_ros_pkgs/blob/galactic/gazebo_plugins/include/gazebo_plugins/gazebo_ros_joint_state_publisher.hpp#L44-L73) | ||
|
|
||
| * Receive orders to move the different joints (**ROS2PoseTrajectory** component): | ||
| Set the trajectory of points to be followed by joints in the O3DE simulation using as input | ||
| `trajectory_msgs/msg/JointTrajectory.msg` | ||
| See [gazebo_ros_pkgs plugin for reference](https://github.com/ros-simulation/gazebo_ros_pkgs/blob/galactic/gazebo_plugins/include/gazebo_plugins/gazebo_ros_joint_pose_trajectory.hpp#L26-L48) | ||
|
|
||
| * The ability to request picks or motion movements to external manipulation planners (**ROS2ManipulationRequester**): | ||
| Request a motion plan to external planners given different types of inputs, | ||
| a target pose for end effector or a joint state target. MoveIt `move_group_interface` interaction | ||
| is an example of this. | ||
| See [MoveIt 2 joint state target goal](https://github.com/ros-planning/moveit2/blob/main/moveit_ros/planning_interface/move_group_interface/include/moveit/move_group_interface/move_group_interface.h#L305-L320) | ||
| or [MoveIt 2](https://github.com/ros-planning/moveit2/blob/main/moveit_ros/planning_interface/move_group_interface/include/moveit/move_group_interface/move_group_interface.h#L640-L554) | ||
|
|
||
| * Provide data from the simulated sensors: not in the scope of this document |
There was a problem hiding this comment.
I am strongly in favor of moving this section to the architecture.md document.
I have nothing against it and can happily accept the patch. The only reason why I put it here at the begging is because I tried to follow the vehicle_dynamics docs that were the first one in the design dir.
@adamdbrw do you have any preference?
Co-authored-by: Lars Gleim <lgleim@users.noreply.github.com>
Signed-off-by: Jose Luis Rivero <jrivero@osrfoundation.org>
Thanks @danielemorra98. I like the simplification and as an easy use case to implement the system and check if the design is working in the very basic looks like a great approach to me. Let me comment it internally with the team and back here if someone sees something to be fixed or to complete it. How do you prefer to integrate it in this document? Adding to the current diagram as a variant of it or do you prefer to replace the original in this PR? |
@j-rivero We were discussing that it could make sense to create an |
Team agree that the simplification looks valid to our eyes. Good work.
While I don't disagree that in some cases many industrial use cases requirements are hard to implement with the current infrastructure we have people at Open Robotics successfully using it for industrial projects. Did you have any bad experience with respect to the use of it in your use cases?
Sounds good to me. Could you please send a PR against this one or a patch that I can apply? Thanks so much. |
|
As a request from @michalpelka, I've tried to document a whole integration between Gazebo + ros2_control + MoveIt. With this in hand, and regarding the controller side of things: maybe the long-term option is try to implement a GEM that replaces gazebo_ros2_control. This pkg implements a generic hardware_interface that is used by ros2 controllers like joint_trajectory_controller to know a) what the current joint values are and b) command joints to certain values. The ros2_control architecture which separates the controller and the hardware_interface allows for this nicely. |
Would be great to have the suggestions done by @lgleim and @danielemorra98 to the document, I can take care if they don't have time but would probably be more accurate if a PR is sent against this.
With the Gazebo example in hand, we probably want to add the integration with |
Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
We submitted a PR for incorporating the feedback in #358. I'll wait until Dec 14th for merging it, @lgleim and @danielemorra98 just in case that you have time to give it a look. |
Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
…-ros2-gem into mabelzhang/manip_roadmap
Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
Signed-off-by: Jose Luis Rivero <jrivero@osrfoundation.org>
Signed-off-by: Mabel Zhang <mabel@openrobotics.org>
Co-authored-by: Mabel Zhang <mabel.m.zhang@gmail.com>
Add svg file to the repo and reference it
Manipulation implementation roadmap
Already merged. |
A bit late but don't want to leave this unfinished. @adamdbrw could you please merge it? |
|
I have just a couple of question about the current status.
Maybe this is not the right place to comment on this and if that I apologize. |
|
Thanks, @danielemorra98 for the questions. Let me update you.
|
|
I would like to develop the bridge to connect the Gem to MoveIt with the latest PhysX API update (new Joint method) and the robotic arm found in o3de-physics-test-scene project.
Remarks:
What are your comments and/or suggestion on this? |
Thanks for the comments Daniele. I read briefly your comments and did not find any big conceptual error. I'm afraid that I've been moved to different projects and don't have time to follow up on this one, sorry. I'm pretty sure that @michalpelka or @adamdbrw can help you more on this. |
|
Could we move this to o3de-extras repos and merge in there? |
Fixes o3de/o3de-extras#151
Draft documents about the implementation of the manipulation feature for the o3de-ros2-gem. Mostly inspired by #162.
List of files:
I left the
architecture.mdto be done since I have no previous knowledge of the gem/o3de internals. Any guidance or help is very welcome. I left the PR as Draft by now but I can happy answer to any feedback on the current content.