[demo] Image Processing Pipeline

[wjwwood]

The purpose of this demo is to demonstrate how ROS 2 can be used to write efficient multi node processing pipelines without sacrificing reusability, rapid development, or introspection. The idea is to setup an image processing pipeline that consists of several nodes which are publishing and subscribing to large messages (for example rgb or yuv images at a resolution >= 1280x1024) at high rates (for example 30 or 60 Hz) and show that the pipeline can operate with minimal copying of image messages. Additionally the demo can be used to show ROS 2's ability to do the runtime mapping of nodes to system processes. Giving the developer freedom to easily try different configurations and to make use of fault isolation while developing without costing production performance hits. Finally, this demo would be a candidate for showing off the type masquerading ability of ROS 2 (this is similar to how ROS 1 allowed pub/sub of non-ROS message types). Ideally, if one node publishes a custom message type and another node in the same process subscribes to the same custom type, then no copies would be needed.

Goals:

• Demonstrate use of efficient intra-process comms.
• Demonstrate configuration time organization of nodes into process.
• Demonstrate using non-ROS types in comms (type masquerading).

---

Needed features:

Intra-process communications for Pub/Sub (services might be a stretch goal).

• prototype rclcpp/rcl/rmw/rosidl support for inprocess messages which are used to synchronize access to shared pointers, while mimicking the QoS of the DDS network link. Prototype Pub/Sub Intra-Process Comms #59
• Find a way to instrument messages to monitor their construction/destruction/copy methods. Add Instrumentation to Generated Msg Code rosidl#21 (for now we'll skip this)
• Propose intra-process design based on prototype and results of instrumentation.
• Develop tests for the intra-process comms.

Node Components (Component Nodes?)

The ability for users to write nodes which are agnostic to their execution, i.e. in its own process or sharing a process with others.

• Component style API (inherit from base class, instantiated by system) (already in rclcpp)
• [ ] Provide tooling (CMake and introspection) for creating “Component” style nodes. (partially available) moved to Provide CMake infrastructure for creating a node rclcpp#114
  • Each node generates a dynamically loadable shared library
  • Each node generates an executable
  • Each node generates any marker files or related meta data for discovery/tooling support
• [ ] Provide component “container” node which can load other nodes and run them. moved to Create a generic node container rclcpp#115
  • Provide convenience tools for loading and executing nodes in existing processes.
  • Either by a proxy (like running a nodelet in ROS 1) or a Service call.
• Provide infrastructure for discovering, introspecting, and loading nodes.
  • This requires something like class_loader from ROS 1. Port class_loader or alternative into ROS 2 #60
  • This requires something like pluginlib from ROS 1. https://github.com/ros/pluginlib/tree/ros2
  • Discovery should be based on the ament_index.

Node Component life cycle

• [ ] At least a minimal life cycle will be required to handle the loading of nodes into a node container, i.e. startup, run, shutdown.

Configuration Management

• [ ] Any features (syntactic sugar) needed in the launch needed to reasonably manage configuration of nodes and processes.
  • Perhaps some higher level interfaces to represent nodes and node containers.

Type Masquerading

Type masquerading is where you can publish and subscribe to types other than the message type of the topic as long as there is a mapping between them. Type masquerading is closely related to Intra-Process Communication, because if done properly no intermediate conversion is necessary if a publisher and subscribe in the same process are using the same masqueraded type.

• [ ] Define an interface for rclcpp similar to the “message traits” interface in ROS 1.
  • This allows a user to map a custom time to a ROS message type.
  • This interface needs to have something like conversion constructors so that the custom type can be converted to and from the ROS equivalent in memory.
• [ ] Add support for type masquerading in rclcpp where appropriate.
  • Publishers and Subscribers
  • Service Clients? and Service Servers? (probably a stretch goal for this demo)
• [ ] Avoid unnecessary conversions and copies when using with intra-process comms.
• [ ] Type masquerading should work with both intra-process and interprocess communications.

---

Other tasks:

• Setup package with demo in it.
  • Should have three or more nodes.
  • Should have large images being transported between nodes in more than one place.
  • [ ] Should use a non-ROS message type like OpenCV's image format.
  • [ ] Should provide a few example "launch files" for different runtime configurations.
  • [ ] Should have instructions on how to introspect the activity, including: dropped messages, message buffering, and message object construction, copying, and conversions.

[wjwwood]

@dirk-thomas @jacquelinekay @tfoote @esteve Can you guys look over this updated description and mention anything you think is missing or unclear?

[jacquelinekay]

Can you define the magnitude of "large messages" and "high rates" in the first paragraph of the description?

[jacquelinekay]

Other considerations, I don't know how specific you want to get:

Provide a concrete metric for when each feature will be considered done for the demo. For example, the goal for type masquerading could be "show efficient transport of an OpenCV image masquerading as a generic Image type in both intra and inter-process communication".

Give a dependency graph for this set of features (unless you think this is already adequately captured in the MindMup)

[dirk-thomas]

Looks detailed and complete to me.

[wjwwood]

Can you define the magnitude of "large messages" and "high rates" in the first paragraph of the description?

That's a good idea, I'll come up with something that is typical, e.g. 30 Hz and 1280x1024 pixel RGB images.

Provide a concrete metric for when each feature will be considered done for the demo. For example, the goal for type masquerading could be "show efficient transport of an OpenCV image masquerading as a generic Image type in both intra and inter-process communication".

I can try to go into more detail about the acceptance criteria, but I wasn't sure exactly what type I would use, so I left details like using OpenCV's Mat object or something like that out of the description. The point about type masquerading working in both intra and inter process comms is valid, I'll add that.

Give a dependency graph for this set of features (unless you think this is already adequately captured in the MindMup)

I think these features are actually (maybe surprisingly) isolated from each other in terms of dependencies. The sub bullets of the node component model stuff is fairly sequential, but ideally they can be done independently of intra-process, type masquerading, and component life cycle items.

[dirk-thomas]

I pushed the forked pluginlib which uses the forked class_loader as well as the ament index through ament_index_cpp: https://github.com/ros/pluginlib/tree/ros2

[wjwwood]

TODO for me. Summarize what's been done for the alpha 1 and move remaining things into new issues.

[wjwwood]

Ok, I've struck out anything we didn't close above, and I spun out a few issues for some of the more immediate tasks. We only managed to accomplish part of this demo:

• We got intra process working.
• We did not get loading of nodes by components done.
• We did not get Type Masquerading implemented.
• We did not get to extending launch with syntactic sugar for launching nodes remotely.

[dhood]

For anyone that comes across this ticket: see https://github.com/ros2/ros2/wiki/Intra-Process-Communication for the demo and https://github.com/ros2/demos/tree/release-alpha8/intra_process_demo for the accompanying code