From eeb093e7e05422a36924815ad121f8f72f1fd1cd Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Tue, 13 Jun 2023 13:42:51 -0700 Subject: [PATCH 01/10] Add REP-2016 for type descriptions Signed-off-by: Emerson Knapp --- rep-2016.rst | 150 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 rep-2016.rst diff --git a/rep-2016.rst b/rep-2016.rst new file mode 100644 index 00000000..b3e0d01b --- /dev/null +++ b/rep-2016.rst @@ -0,0 +1,150 @@ +REP: 2016 +Title: ROS 2 Interface Type Descriptions - Representation, Hashing, Discovery and Distribution +Author: Emerson Knapp +Status: Active +Type: Standards Track +Content-Type: text/x-rst +Created: 13-Jun-2023 +Post-History: + + +Abstract +======== + +This REP proposes a standard format for the communication of ROS 2 interface types, along with tooling to support their use. + +It provides a ROS 2 interface type that can contain these descriptions, runtime data structures, and a universally reproducible serialization method. +Alongside the description format, this REP also defines a hashing algorithm to create standardized hashes of type descriptions, to quickly detect type mismatch during discovery. + + +Motivation +========== + +ROS 1 clients provided an MD5 sum from msg files to check for matching definitions. + +Terminology +=========== + +Type Source - original text used to define a type +Type Description - a data structure representating a parsed Type Source, which removes irrelevant data such as comments, and will be equal regardless of whether the source was msg, IDL, or otherwise. +RIHS (ROS Interface Hashing Standard) - a versioned specification for producing a hash value for a Type Description + + +Specification +============= + +In order to communicate and compare types, Type Description Interfaces are defined to contain type information. +To detect and enforce type version mismatches, and communicate information about type descriptions compactly, a way to uniquely identify types is required. +This proposal uses a hash of the type's description for this purpose. + + +Type Description Interfaces +--------------------------- + +A Type Description must be produced in a stable way such that it is not changed by trivial differences that do not impact compatibility. +The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the TypeDescription object as an intermediate representation. +This way, types coming from two sources that have the same stated name and have the same information will be given the same description, even if they are defined using sources that do not have identical text. +The description should only contain information needed for programmatic reasons. +Thus the representation should include: + +This representation includes: + +- the package, namespace, and type name, for example `sensor_msgs/msg/Image` +- a list of field names and types +- a list of all recursively referenced types +- no comments + + +Type Hash +--------- + +The hash must also be able to be calculated at runtime from information received on the wire. +The hash must only be computed using fields that affect communication compatibility. +Thus the hash excludes one aspect of Type Descriptions: it omits default values. +This is because default values for fields are only used by the writer of a packet, the recipient always receives some value in the field and can read it, thus defaults cannot affect compatibility. + +This allows subscribers to validate the received TypeDescriptions against advertised hashes, and allows dynamic publishers to invent new types and advertise their hash programmatically. + +Finally, the resulting filled data structure must be represented in a platform-independent format, rather than running the hash function on the in-memory native type representation. +Different languages, architectures, or compilers will produce different in-memory representations, and the hash must be consistently calculable in different contexts. + +The resulting data structure is hashed using SHA-256, resulting in a 256-bit (32-byte) hash value which is also generally known as a "message digest". +This hash is paired with a type version hash standard version, which we will call the "ROS IDL Hashing Standard" or "RIHS", the first version of which will be ``RIHS01``. +RIHS Version 00 is reserved for "Invalid" / "unset", and the RIHS version is limited by this specification to a maximum value of 255. +RIHS hash values must have a well-defined UTF-8 string representation for human readability and for passing over string-only communication channels. +The prefix of a well-formed RIHS string will always be ``RIHSXX_``, where ``X`` is one hexadecimal digit, followed by the version-dependent string representation of the hash value. +For ``RIHS01``, the hash value is 64 hexadecimal digits representing the 256-bit message digest, leading to a known ``RIHS01`` string length of 71. + +This versioning allows the tooling to know if a hash mismatch is due to a change in this standard (how hash is computed) or due to a difference in the interface types themselves. +In the case of a change in standard, it will be unknown whether the interface types are equal or not. + +For now, the list of field names and their types are the only contributing factors, but in the future that could change, depending on which "annotations" are supported in ``.idl`` files. +The "IDL - Interface Definition and Language Mapping" design document\ [2]_ describes which features of the OMG IDL standard are supported by ROS 2. +If that is extended in the future, then this data structure may need to be updated, and if so the "ROS IDL Hashing Standard" version will also need to be incremented. +New sanitizing may be needed on the TypeDescription pre-hash procedure, in the case of these new features. + +.. TODO:: + + Re-audit the supported features from OMG IDL according to the referenced design document, including the @key annotation and how it may impact this for the reference implementation. + +Notes: + +The type version hash is not sequential and does not imply any rank among versions of the type. That is, given two version hashes of a type, there is no way to tell which is "newer". + +Because the hash contains the stated name of the type, differently-named types with otherwise identical descriptions will be mismatched as incompatible. +This matches existing ROS precedent of strongly-typed interfaces. + +The type version hash can only be used to determine if type versions are equal and if there exists a chain of transfer functions that can convert between them. +Because of this, when a change to a type is made, it may or may not be necessary to write transfer functions in both directions depending on how the interface is used. + +It may be desirable, as a user, to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. +There is no built-in way to do this manual re-versioning. +However, we suggest the following method which requires no special tooling: provide an extra field within the interface with a name ``bool versionX = true``. +To manually trigger a hash update, change by increment the name of the field, for example ``bool versionY = true``. + +The TypeDescription does not include the serialization format being used, nor does it include the version of the serialization technology. +This type version hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. +The type version hash must be considered in context, with the serialization format and version in order to determine wire compatibility. + + +Type Hash Discovery +------------------- + +Hashes are intended to be communicated such that they are available at the time of discovering a topic, before creating subscriptions. +The hash will be available in the ``rmw_topic_endpoint_info_t`` data structure from discovery. + +For DDS implementations of the RMW API, it is recommended but not required to use the USER_DATA QoS policy to send this information. + +This discovery-time hash availability allows for validation of type mismatch before ever requesting a subscription. + +It can also give subscription-side tooling the opportunity to obtain an appropriate type description for the given hash, if it is not available. +That point brings us to the final feature of this REP. + +Type Description Distribution +----------------------------- + +``type_description_interfaces`` defines a service ``GetTypeDescription``, that will be provided as a builtin service on nodes. +The service must be optional, but it will be a detail decided by client libraries whether it is enabled or disabled by default. + + +References +========== + +.. http://wiki.ros.org/Topics +.. REP 2011 Evolving message types +.. REP 20XX Dynamic pubsub (name TBD) + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 8b0ebff13c90a150e5bc650e33081747c86ecb2e Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Thu, 15 Jun 2023 15:13:21 -0700 Subject: [PATCH 02/10] Checking in rep-2011 for now while editing Signed-off-by: Emerson Knapp --- rep-2011.rst | 1428 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1428 insertions(+) create mode 100644 rep-2011.rst diff --git a/rep-2011.rst b/rep-2011.rst new file mode 100644 index 00000000..770797a8 --- /dev/null +++ b/rep-2011.rst @@ -0,0 +1,1428 @@ +REP: 2011 +Title: Evolving Message Types, and Other ROS Interface Types, Over Time +Author: William Woodall , Brandon Ong , You Liang Tan , Kyle Marcey +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 30-Nov-2021 +Post-History: + + +Abstract +======== + +This REP proposes patterns and approaches for evolving message, service, and action ROS interface types over time. + +The proposed patterns use the existing default serialization technology, i.e. CDR, in combination with new tooling to detect when types have changes and new tooling to convert between versions. +However, it should be possible to use features provided by different, perhaps future, serialization technologies in addition to the approaches proposed in this REP. + +Specifically, this REP proposes that we provide tooling to convert from old versions of types to newer versions of types on demand, using user-defined transfer functions. +This approach is a good way to achieve backwards compatibility in messages over long periods of time and in a variety of scenarios, e.g. over the wire, converting bag files, or in specialized tools. + +This approach does not rely on specific features of the serialization technology and instead relies on the ability to communicate the full type descriptions on the wire, beyond their name and version, and use that description to introspect the values of it at runtime using reflection. + +This approach can be used in conjunction with serialization technology specific features like optional fields, inheritance, etc., but it works even with the simplest serialization technologies. +This is true so long as we have the ability to introspect the messages at runtime without prior knowledge of the type description, which is a feature we also need for generic introspection tools like ``rosbag2`` and ``ros2 topic echo``. + +Also covered in this REP are recommended changes to the middleware API in ROS 2, as well as additional infrastructure, to support runtime introspection of messages. +It is very similar to, though slightly narrower and more generic than, the DDS-XTypes specification\ [1]_, and an interface similar to DDS-XTypes will likely be adopted in the ROS 2 middleware API, without explicitly relying on it. + +Alternatives are also discussed in more detail. + +Motivation +========== + +Evolving types over time is a necessary part of developing a new system using ROS 2, as well as a necessary part of evolving ROS 2 itself over time. +As needs change, types often need to change as well to accommodate new use cases by adding or removing fields, changing the type of fields, or splitting/combining messages across different topics. + +To understand this need, it's useful to consider some examples, like this message definition: + +.. code:: + + # Temperature.msg + uint64 timestamp + int32 temperature + +There are many potential issues with this message type, but it isn't uncommon for messages like this to be created in the process of taking an idea from a prototype into a product, and for the sake of this example, let's say that after using this message for a while the developers wanted to change the temperature field's type from int32 to float64. + +There are a few ways the developers could approach this, e.g. if the serialization technology allowed for optional fields, they may add a new optional field like this (pseudo code): + +.. code:: + + # Temperature.msg + uint64 timestamp + int32 temperature + optional float64 temperature_float + +Updated code could use it like this (pseudo code): + +.. code:: + + void on_temperature(const Temperature & temp) { + // ... + double actual_temp = 0.0; + if (temp.temperature_float_is_set()) { + actual_temp = temp.temperature_float(); + } else { + actual_temp = static_cast(temp.temperature()); + } + } + +This is not uncommon to see in projects, and it has the advantage that old data, whether it is from a program running on an older machine or is from old recorded data, can be interpreted by newer code without additional machinery or features beyond the optional field type. +Older publishing code can also use the new definition without being updated to use the new ``temperature_float`` field, which is very forgiving for developers in a way. +However, as projects grow and the number of incidents like the one described in the example above increase, this kind of approach can generate a lot of complexity to sending and receiving code. + +You can imagine other serialization features, like inheritance or implicit type conversion, could be used to address this desired change, but in each case it relies on a feature of the serialization technology which may or may not be available in all middleware implementations for ROS 2. + +Each of these approaches also requires a more sophisticated message API for the users than what is currently provided by ROS 2. +At the moment the user can access all fields of the message directly, i.e. "member based access" vs "method based access", and that would need to change in some way to use some of these features. + +A different approach would be to simply update the type as follows: + +.. code:: + + # Temperature.msg + uint64 timestamp + float64 temperature + +And also update any code publishing or subscribing to this type at the same time. +This is typically what happens right now in ROS 2, and also what happened historically in ROS 1. +This approach is simple and requires no additional serialization features, but obviously doesn't do anything on its own to help developers evolve their system while maintaining support for already deployed code and utilizing existing recorded data. + +However, this REP proposes that with additional tooling these cases can be handled without the code of the application knowing about it. +Consider again the above example, but this time the update to the message was done without backwards compatibility in the message definition. +This means that publishing old recorded data, for example, will not work with new code that subscribes to the topic, because the new subscription expects the new version of the message. +For the purpose of this example, let's call the original message type ``Temperature`` and the one using ``float64`` we'll call ``Temperature'``. +So, if you have rosbag2 publishing ``Temperature`` messages and a program consuming ``Temperature'`` messages they will not communicate, unless you have an intermediate program doing the translation. + +.. code:: + + ┌─────────┐ publishes ┌──────────────┐ publishes ┌─────┐ + │ rosbag2 ├───────────────►│transfer func.├──────────────►│ App │ + └─────────┘ Temperature └──────────────┘ Temperature' └─────┘ + +The "transfer function" can be user-defined, or for simple changes, like changing the field type to a compatible type, it can be done automatically. +We already do something like this for the "ROS 1 to ROS 2 bridge" in order to handle changes between message types in ROS 1 and ROS 2, and something like this was also done for rosbags in ROS 1 with the "bag migration rules" feature. + +.. TODO:: + + cite the ros1_bridge rules and the rosbag migration rules + +The transfer functions require the ability to have a single application which can interact with both the old and the new versions of a message at the same time. +Making this possible requires several new technical features for ROS 2, and some new infrastructure and tooling +However, by keeping the conversion logic contained in these transfer functions, it has the advantage of keeping both the publishing and subscribing code simple. +That is to say, it keeps both the publishing and subscribing code agnostic to the fact that there are other versions of the message, and it keeps the message type from being cluttered with vestigial fields, e.g. having both a ``temperature`` and ``temperature_float`` in the same message. + +As stated before, problems created by changing these ROS interfaces can usually be solved by more than one way, either using some feature like optional fields or by just breaking compatibility directly. +However, the strategy used usually depends on the features that the serialization technology being used offers. +ROS 2 has special considerations on this topic because it can support different serialization technologies, and though CDR is the default and most common right now, others could be used in the future. +Therefore, it is neither desirable to depend on features of a specific technology, nor is it desirable suggest patterns that rely on features that only some serialization technologies provide. +In either case, that would tie ROS 2 to specific serialization technologies, and that should be avoided if possible. + +That being said, this proposal will require some specific features from the middleware and serialization technology, but the goal is to choose approaches which give ROS 2 the broadest support across middleware implementations, ideally while not limiting users from using specific features of the underlying technology when that suits them. + +With those examples and design constraints as motivation, this REP makes a proposal on how to handle evolving message types in the following Specification section, as well as rationales and considered alternatives in the Rationale section and its sub-sections. + +Terminology +=========== + +TODO + + +Specification +============= + +The proposal is to provide tooling to help users: + +- identify when messages have changed +- configure their system to convert between versions of messages on demand +- write the code needed to convert between types when the conversion is not trivial + +In order to do this, this proposal describes how ROS 2 can be changed to support these tools by: + +- enforcing type compatibility by version + + - by providing type version hashes, and + - making it possible to see what versions of types are being used by other endpoints, and + - warning users when type enforcement is preventing two endpoints from communicating + +- providing access to the complete type description of types being used + + - and making it possible to access the type description from nodes remotely + +- making it possible to publish and subscribe to topics using just the type description + + - even when the type was not available at compile time + - and introspecting the values from a serialized message using just the type description + +This Specification section covers the conceptual overview in more detail, then describes each of the technical changes needed in ROS 2, and then finishes by describing the new tooling that will help users in the aforementioned ways. + +Conceptual Overview +------------------- + +Users will be able to calculate the "type version hash" for an interface (e.g. a message, service, or action) using the ``ros2 interface hash `` command. +This hash is also used by ROS 2 to determine if the same type name but different type versions are being used on the same topic, so that a warning may be logged that the endpoints that do not match may not communicate. + +.. note:: + + An exception to this rule is that if the underlying middleware has more sophisticated rules for matching types, for example the type has been extended with an optional field, then they may still match + In that case, ROS 2 will defer to the middleware and not produce warnings when the type version hashes do not match. + Instead, ROS 2 will rely on the middleware to notify it when two endpoints do not match based on their types not being compatible, so that a warning can be produced. + +When a mismatch is detected, the user can use user-defined or automatically generated generic "transfer functions" to convert between versions of the type until it is in the type version they wish to send or receive. +They can use a tool that will look at a catalogue of available transfer functions to find a single transfer function, or a set of transfer functions, to get from the current type version to the desired type version. + +.. code:: + + ┌───────────────────────┐ + ┌───────────────┐ │ Implicit Conversion │ ┌───────────────┐ + │Message@current├────►│ by ├───►│Message@desired│ + └───────────────┘ │ Generic Transfer Func.│ └───────────────┘ + └───────────────────────┘ + + or + + ┌───────────────────────┐ + ┌───────────────┐ │ Implicit Conversion │ ┌────────────────────┐ + │Message@current├────►│ by ├───►│Message@intermediate│ + └───────────────┘ │ Generic Transfer Func.│ └────────────────────┘ + └───────────────────────┘ + + or + + ┌───────────────────────┐ + ┌───────────────┐ │ User-defined transfer │ ┌───────────────┐ + │Message@current├────►│ function from current ├───...───►│Message@desired│ + └───────────────┘ │to desired/intermediate│ ▲ └───────────────┘ + └───────────────────────┘ │ + │ + possibly other transfer functions + +The tool will start with the current type version and see if it can be automatically converted to the desired type version, or if it is accepted as an input to any user-defined transfer functions or if it can be automatically converted into one of the input type versions for the transfer functions. +It will continue to do this until it reaches the desired type version or it fails to find a path from the current to the desired type version. + +.. code:: + + ┌──────────────────────┐ /topic ┌─────────────────────────┐ + │Publisher├────────X────────►│Subscription│ + └──────────────────────┘ └─────────────────────────┘ + │ │ │ + remap publisher │ │ │ and add transfer function + ▼ ▼ ▼ + ┌──────────────────────┐ ┌─────────────────────────┐ + │Publisher│ │Subscription│ + └─┬────────────────────┘ └─────────────────────────┘ + │ ▲ + │ ┌─────────────────────────────────┐ │ + ✓ /topic/ABC │ Transfer Functions for ABC->XYZ │ /topic ✓ + │ │ │ │ + │ ┌──────────┴──────────────┐ ┌──────────────┴───────┐ │ + └─►│Subscription│ │Publisher├──┘ + └──────────┬──────────────┘ └──────────────┬───────┘ + │ │ + └─────────────────────────────────┘ + +Once the set of necessary transfer functions has been identified, the ROS graph can be changed to have one side of the topic be remapped onto a new topic name which indicates it is of a different version that what is desired, and then the transfer function can be run as a component node which subscribes to one version of the message, performs the conversion using the chain of transfer functions, and then publishes the other version of the message. +Tools will assist the user in making these remappings and running the necessary component nodes with the appropriate configurations, either from their launch file or from the command line. + +.. TODO:: + + discuss the implications for large messages and the possibility of having the transfer functions be colocated with either the publisher or subscription more directly than with component nodes and remapping. + +Once the mismatched messages are flowing through the transfer functions, communication should be possible and neither the publishing side nor the subscribing side have any specific knowledge of the conversions taking place or that any conversions are necessary. + +.. TODO:: + + Extend conceptual overview to describe how this will work with Services and Actions. + Services, since they are not sensitive to the many-to-many (many publisher) issue, unlike Topics, and because they do not have as many QoS settings that apply to them, they can probably have transfer functions that are plugins, rather than separate component nodes that repeat the service call, like the ros1_bridge. + Actions will be a combination of topics and services, but will have other considerations in the tooling. + +In order to support this vision, three missing features will need to be added into ROS 2 (which were also mentioned in the introduction): + +- enforcing type compatibility by version +- providing access to the complete type description of types being used +- making it possible to publish and subscribe to topics using just the type description + +These features are described in the following sections. + +Type Version Enforcement +------------------------ + +In order to detect and enforce type version mismatches, as well as communicate information about type descriptions compactly, a way to uniquely identify versions is required. +This proposal uses a hash of the type's description for this purpose. + + +Type Version Hash +^^^^^^^^^^^^^^^^^ + +The hash must be calculated in a stable way such that it is not changed by trivial differences that do not impact serialization compatibility. +The hash must also be able to be calculated at runtime from information received on the wire. +This allows subscribers to validate the received TypeDescriptions against advertised hashes, and allows dynamic publishers to invent new types and advertise their hash programmatically. +The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the TypeDescription object as an intermediate representation. +This way, types coming from two sources that have the same stated name and are wire compatible will be given the same hash, even if they are defined using source text that is not exactly equal. +The hash must only be computed using fields that affect communication compatibility. Thus the representation should include: + +This representation includes: + +- the package, namespace, and type name, for example `sensor_msgs/msg/Image` +- a list of field names and types +- a list of all recursively referenced types +- no field default values +- no comments + +Finally, the resulting filled data structure must be represented in a platform-independent format, rather than running the hash function on the in-memory native type representation. +Different languages, architectures, or compilers will produce different in-memory representations, and the hash must be consistently calculable in different contexts. + +The resulting data structure is hashed using SHA-256, resulting in a 256-bit (32-byte) hash value which is also generally known as a "message digest". +This hash is paired with a type version hash standard version, which we will call the "ROS IDL Hashing Standard" or "RIHS", the first version of which will be ``RIHS01``. +RIHS Version 00 is reserved for "Invalid" / "unset", and the RIHS version is limited by this specification to a maximum value of 255. +RIHS hash values must have a well-defined UTF-8 string representation for human readability and for passing over string-only communication channels. +The prefix of a well-formed RIHS string will always be ``RIHSXX_``, where ``X`` is one hexadecimal digit, followed by the version-dependent string representation of the hash value. +For ``RIHS01``, the hash value is 64 hexadecimal digits representing the 256-bit message digest, leading to a known ``RIHS01`` string length of 71. + +This versioning allows the tooling to know if a hash mismatch is due to a change in this standard (how hash is computed) or due to a difference in the interface types themselves. +In the case of a change in standard, it will be unknown whether the interface types are equal or not. + +For now, the list of field names and their types are the only contributing factors, but in the future that could change, depending on which "annotations" are supported in ``.idl`` files. +The "IDL - Interface Definition and Language Mapping" design document\ [2]_ describes which features of the OMG IDL standard are supported by ROS 2. +If that is extended in the future, then this data structure may need to be updated, and if so the "ROS IDL Hashing Standard" version will also need to be incremented. +New sanitizing may be needed on the TypeDescription pre-hash procedure, in the case of these new features. + +.. TODO:: + + Re-audit the supported features from OMG IDL according to the referenced design document, including the @key annotation and how it may impact this for the reference implementation. + +Notes: + +The type version hash is not sequential and does not imply any rank among versions of the type. That is, given two version hashes of a type, there is no way to tell which is "newer". + +Because the hash contains the stated name of the type, differently-named types with otherwise identical descriptions will be mismatched as incompatible. +This matches existing ROS precedent of strongly-typed interfaces. + +The type version hash can only be used to determine if type versions are equal and if there exists a chain of transfer functions that can convert between them. +Because of this, when a change to a type is made, it may or may not be necessary to write transfer functions in both directions depending on how the interface is used. + +It may be desirable, as a user, to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. +There is no built-in way to do this manual re-versioning. +However, we suggest the following method which requires no special tooling: provide an extra field within the interface with a name ``bool versionX = true``. +To manually trigger a hash update, change by increment the name of the field, for example ``bool versionY = true``. + +The TypeDescription does not include the serialization format being used, nor does it include the version of the serialization technology. +This type version hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. +The type version hash must be considered in context, with the serialization format and version in order to determine wire compatibility. + +Enforcing Type Version +^^^^^^^^^^^^^^^^^^^^^^ + +The type version hash may be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate. +Again, whether or not it is used will depend on the underlying middleware and how it determines if types are compatible between endpoints. +Simpler middlewares will not do anything other than check that the type names match, in which case the version hash will likely be used. +However, in more sophisticated middlewares type compatibility can be determined using more complex rules and by looking at the type descriptions themselves, and in those cases the type version hash may not be used to determine matching. + +When creating a publisher or subscription, the caller normally provides: + +- a topic name, +- QoS settings, and +- a topic type + +Where the topic type is represented as a string and is automatically deduced based on the type given to the function that creates the entity. +The type may be passed as a template parameter in C++ or as an argument to the function in Python. + +For example, creating a publisher for the C++ type ``std_msgs::msg::String`` using ``rclcpp`` may result in a topic type like the string ``std_msgs/msg/String``. + +All of the above items are used by the middleware to determine if two endpoints should communicate or not, and this REP proposes that the type version be added to this list of provided information. + +Nothing needs to change from the user's perspective, as the type version can be extracted automatically based on the topic type given, either at the ``rcl`` layer or in the ``rmw`` implementation itself. +That is to say, how users create things like publishers and subscription should not need to change, no matter which programming language is being used. + +However, the type version hash would become something that the ``rmw`` implementation is provided and aware of in the course of creating a publisher or subscription. +The decision of whether or not to use that information to enforce type compatibility would be left to the middleware, rather than implementing it as logic in ``rcl`` or other packages above the ``rmw`` API. + +The method for implementing the detection and enforcement of type version mismatches is left up to the middleware. +Some middlewares will have tools to handle this without this new type version hash and others will implement something like what would be possible in the ``rcl`` and above layers using the type version hash. +By keeping this a detail of the ``rmw`` implementation, we allow the ``rmw`` implementations to make optimizations where they can. + +Recommended Strategy for Enforcing that Type Versions Match +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If the middleware has a feature to handle type compatibility already, as is the case with DDS-XTypes which is discussed later, then that can be used to enforce type safety, and then the type version hash would only be used for debugging and for storing in recordings. + +However, if the middleware lacks this kind of feature, then the recommended strategy for accomplishing this in the ``rmw`` implementation is to simply concatenate the type name and the type version hash with double underscores and then use that as the type name given to the underlying middleware. +For example, a type name using this approach may look like this: + +.. code:: + + sensor_msgs/msg/Image__RIHS01_XXXXXXXXXXXXXXXXXXXX + +This has the benefit of "just working" for most middlewares which at least match based on the name of the type, and it is simple, requiring no further custom hooks into the middleware's discovery or matchmaking process. + +However, one downside with this approach is that it makes interoperation between ROS 2 and the "native" middleware more difficult, as appending the version hash to the type name is just "one more thing" that you have to contend with when trying to connect non-ROS endpoints to a ROS graph. + +Alternative Strategy for Enforcing that Type Versions Match +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sometimes the recommended strategy would interfere with features of the middleware that allow for more complex type compatibility rules, or otherwise interferes with the function of the underlying middleware. +In these cases, it is appropriate to not take the recommended strategy and delegate the matching process and notification entirely to the middleware. + +However, in order to increase compatibility between rmw implementations, it is recommended to fulfill the recommended approach whenever possible, even if the type name is not used in determining if two endpoints will match, i.e. in the case that the underlying middleware does something more sophisticated to determine type compatibility but ignores the type name itself. + +This recommendation is particularly useful in the case where a DDS based ROS 2 endpoint is talking to another DDS-XTypes based ROS 2 endpoint. +The DDS-XTypes based endpoint then has a chance to "gracefully degrade" to interoperate with the basic DDS based ROS 2 endpoint. +This would not be the case if the DDS-XTypes based ROS 2 endpoint did not include the type version hash in the type name, as is suggested with the recommended strategy. + +.. TODO:: + + We need to confirm whether or not having a different type name prevents DDS-XTypes from working properly. + We've done some experiments, but we need to summarize the results and confirm the recommendation in the REP specifically geared towards DDS. + +Notifying the ROS Client Library +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One potential downside to delegating type matching to the rmw implementation is that detecting the mismatch is more complicated. +If ROS 2 is to provide users a warning that two endpoints will not communicate due to their types not matching, it requires there to be a way for the middleware to notify the ROS layer when a topic is not matched due to the type incompatibility. +As some of the following sections describe, it might be that the rules by which the middleware decides on type compatibility are unknown to ROS 2, and so the middleware has to indicate when matches are and are not made. + +If the middleware just uses the type name to determine compatibility, then the rmw implementation can just check the type version hash, and if they do not match between endpoints then the rmw implementation can notify ROS 2, and a warning can be produced. + +Either way, to facilitate this notice, the ``rmw_event_type_t`` shall be extended to include a new event called ``RMW_EVENT_OFFERED_TYPE_INCOMPATIBLE``. +Related functions and structures will also be updated so that the event can be associated with specific endpoints. + +.. TODO:: + + It's not clear how we will do this just now, since existing "QoS events" lack a way to communicate this information, I (wjwwood) think. + +Accessing the Type Version Hash +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For debugging and introspection, the type version hash will be accessible via the ROS graph API, by extending the ``rmw_topic_endpoint_info_t`` struct, and related types and functions, to include the type version hash, ``topic_type_version_hash``, as a string. +It should be along side the ``topic_type`` string in that struct, but the ``topic_type`` field should not include the concatenated type version hash, even if the recommended approach is used. +Instead the type name and version hash should be separated and placed in the fields separately. + +This information should be transmitted as part of the discovery process. + +This field can be optionally an empty string, in order to support interaction with older versions of ROS where this feature was not yet implemented, but it should be provided if at all possible. + +Notes for Implementing the Recommended Strategy with DDS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The DDS standard provides an ``on_inconsistent_topic()`` method on the ``ParticipantListener`` class as a callback function which is called when an ``INCONSISTENT_TOPIC`` event is detected. +This event occurs when the topic type does not match between endpoints, and can be used for this purpose, but at the time of writing (July 2022), this feature is not supported across all of the DDS vendors that can be used by ROS 2. + +Sending of the type version hash during discovery should be done using the ``USER_DATA`` QoS setting, even if the type version hash is not used for determining type compatibility, and then provided to the user-space code through the ``rmw_topic_endpoint_info_t`` struct. + +Interactions with DDS-XTypes or Similar Implicit Middleware Features +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When using DDS-Xtypes type compatibility is determined through sophisticated and configurable rules, allowing for things like extensible types, optional fields, implicit conversions, and even inheritance. +Which of these features is supported for use with ROS is out of scope with this REP, but if any of them are in use, then it may be possible for two endpoints to match and communicate even if their ROS 2 type version hashes do not match. + +In this situation the middleware is responsible for communicating to the rmw layer when an endpoint will not be matched due to type incompatibility. +The ``INCONSISTENT_TOPIC`` event in DDS applies for DDS-XTypes as well, and should be useful in fulfilling this requirement. + +Type Description Distribution +----------------------------- + +For some use cases the type version hash is insufficient and instead the full type description is required. + +One of those use cases, which is also described in this REP, is "Run-Time Interface Reflection", which is the ability to introspect the contents of a message at runtime when the description for that message, or that version of that message, was unavailable at compile time. +In this use case the type description is used to interpret the serialized data dynamically. + +Another use case, which is not covered in this REP, is using the type description in tooling to either display the type description to the user or to include it in recordings. + +In either case, where the type description comes from doesn't really matter, and so, for example, it could be looked up on the local filesystem or read from a rosbag file. +However, in practice, the correct type description may not be found locally, especially in cases where you have different versions of messages in the same system, e.g.: + +- because it's on another computer, or +- because it is from a different distribution of ROS, or +- because it was built in a different workspace, or +- because the application has not been restarted since recompiling a change to the type being used + +In any case, it is useful to have a mechanism to convey the type descriptions from the source of the data to other nodes, which we describe here as "type description distribution". + +Furthermore, this feature should be agnostic to the underlying middleware and serialization library, as two endpoints may not have the same rmw implementation, or the data may have been serialized to a different format in the case of playback of a recording. + +Sending the Type Description +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Type descriptions will be provided by a ROS Service called ``~/_get_type_description``, which will be offered by each node. +There will be a single ROS Service per node, regardless of the number of publishers or subscriptions on that node. + +This service may be optional, for example being enabled or disabled when creating the ROS Node. + +.. TODO:: + + How can we detect when a remote node is not offering this service? + It's difficult to differentiate between "the Service has not been created yet, but will be" and "the Service will never be created". + Should we use a ROS Parameter to indicate this? + But then what if remote access to Parameters (another optional Service) is disabled? + Perhaps we need a "services offered" list which is part of the Node metadata, which is sent for each node in the rmw implementation, but that's out of scope for this REP. + +A service request to this ROS Service will comprise of the type name and the type version hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described in previous sections. +The ROS Service server will respond with the type description and any necessary metadata needed to do Run-Time Interface Reflection. +This service is not expected to be called frequently, and is likely to only occur when new topic or service endpoints are created, and even then, only if the endpoint type hashes do not match. + +.. TODO:: + + Should each endpoint be asked about their type/version pair or should we assume that the type/version pair guarantees a unique type description and therefore reuse past queries? + (wjwwood) I am leaning towards assuming the type name and type version hash combo as being a unique identifier. + +Type Description Contents and Format +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The response sent by the ROS Service server will contain a combination of the original ``idl`` or ``msg`` file's content, as well as any necessary information to serialize and deserialize the raw message buffers sent on the topic. +The response will contain a version of the description that contains comments from the original type description, as those might be relevant to interpreting the semantic meaning of the message fields. + +Additionally, the response could include the serialization library used, its version, or any other helpful information from the original producer of the data. + +.. TODO:: + + What happens if the message consumer doesn't have access to the serialization library stated in the meta-type? + (wjwwood) It depends on what you're doing with the response. If you are going to subscribe to a remote endpoint, then I think we just refuse to create the subscription, as communication will not work, and there's a basic assumption that if you're going to communicate with an endpoint then you are using the same or compatible rmw implementations, which includes the serialization technology. The purpose of this section and ROS Service is to provide the information, not to ensure communication can happen. + +The ROS 2 message that defines the type description must be able to describe any message type, including itself, and since it is describing the message format, it should work independently from any serialization technologies used. +This "meta-type description" message would then be used to communicate the structure of the type as part of the "get type description" service response. +The final form of these interfaces should be found in the reference implementation, but such a Service interface might look like this: + +.. code:: + + string type_name + string type_version_hash + --- + # True if the type description information is available and populated in the response + bool successful + # Empty if 'successful' was true, otherwise contains details on why it failed + string failure_reason + + # The idl or msg file name + string type_description_raw_file_name + # The idl or msg file, with comments and whitespace + # The file extension and/or the contents can be used to determine the format + string type_description_raw + # The parsed type description which can be used programmatically + TypeDescription type_description + + # Key-value pairs of extra information. + string[] extra_information_keys + string[] extra_information_values + +.. TODO:: + + (wjwwood) I propose we use key-value string pairs for the extra information, but I am hoping for discussion on this point. + This type is sensitive to changes, since changes to it will be hard to roll out, and may need manual versioning to handle. + We could also consider bounding the strings and sequences of strings, so the message could be "bounded", which is nice for safety and embedded systems. + +Again, the final form of these interfaces should be referenced from the reference implementation, but the ``TypeDescription`` message type might look something like this: + +.. code:: + + IndividualTypeDescription type_description + IndividualTypeDescription[] referenced_type_descriptions + +And the ``IndividualTypeDescription`` type: + +.. code:: + + string type_name + Field[] fields + +And the ``Field`` type: + +.. code:: + + FIELD_TYPE_NESTED_TYPE = 0 + FIELD_TYPE_INT = 1 + FIELD_TYPE_DOUBLE = 2 + # ... and so on + FIELD_TYPE_INT_ARRAY = ... + FIELD_TYPE_INT_BOUNDED_SEQUENCE = ... + FIELD_TYPE_INT_SEQUENCE = ... + # ... and so on + + string field_name + uint8_t field_type + uint64_t field_array_size # Only for Arrays and Bounded Sequences + string nested_type_name # Only for FIELD_TYPE_NESTED_TYPE + +These examples of the interfaces just give an idea of the structure but perhaps do not yet consider some other complications like field annotations or other as yet unconsidered features we want to support. + +.. TODO:: + + (wjwwood) Add text about how to handle Service types, which are formed as a Request and a Response part, each of which is kind of like a Message. + We could just treat the Request and Response separately, or we could extend this scheme to include explicit support for Services. + Since Actions are composed of Topics and Services, it is less so impacted, but we could similarly consider officially supporting them in this scheme. + +Versioning the ``TypeDescription`` Message Type +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given that the type description message interface has to be generic enough to support anything described in the ROS interfaces, there will be a need to add or remove fields over time in the type description message itself. +This should be done in such a way that the fields are tick-tocked and deprecated properly, possibly by having explicitly named versions of this interface, e.g. ``TypeDescriptionV1`` and ``TypeDescriptionV2`` and so on. + +Implementation in the ``rcl`` Layer +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The implementation of the type description distribution feature will be made in the ``rcl`` layer as opposed to the ``rmw`` layer to take advantage of the abstraction away from the middleware and to allow for compatibility with the client libraries. + +A hook will be added to ``rcl_node_init()`` to initialize the type description distribution service with the appropriate ``rcl_service_XXX()`` functions. +This hook should also keep a map of published and subscribed types which will be populated on each initialization of a publisher or subscription in the respective ``rcl_publisher_init()`` and ``rcl_subscription_init()`` function calls. +The passed ``rosidl_message_type_support_t`` in the init call can be used to obtain the relevant information, alongside any new methods added to support type version hashing. + +There will be an option to opt-out of creating this service, and a way to start and stop the service after node creation as well. + +.. TODO:: + + (wjwwood) A thought just occurred to me, which is that we could maybe have "lazy" Service Servers, which watch for any Service Clients to come up on their Service name and when they see it, they could start a Service Server. + The benefit of this is that when we're not using this feature (or other features like Parameter get/set Services), then we don't waste resources creating them, but if you know the node name and the corresponding Service name that it should have a server on, you could cause it to be activated by trying to use it. + The problem with this would be that you cannot browse the Service servers because they wouldn't advertise until requested, but for "well know service names" that might not be such a problem. + I guess the other problem would be there might be a slight delay the first time you go to use the service, but that might be a worthy trade-off too. + +Run-Time Interface Reflection +----------------------------- + +Run-Time Interface Reflection allows access to the data in the fields of a serialized message buffer when given: + +- the serialized message as a ``rmw_serialized_message_t``, basically just an array of bytes as a ``rcutils_uint8_array_t``, +- the message's type description, e.g. received from the aforementioned "type description distribution" or from a bag file, and +- the serialization format, name and version, which was used to create the serialized message, e.g. ``XCDR2`` for ``Extended CDR encoding version 2`` + +From these inputs, we should be able to access the fields of the message from the serialized message buffer using some programmatic interface, which allows you to: + +- list the field names +- list the field types +- access fields by name or index + +.. code:: + + message_buffer ─┐ ┌───────────────────────────────┐ + │ │ │ + message_description ─┼──►│ Run-Time Interface Reflection ├───► Introspection API + │ │ │ + serialization_format ─┘ └───────────────────────────────┘ + +Given that the scope of inputs and expected outputs is so limited, this feature should ideally be implemented as a separate package, e.g. ``rcl_serialization``, that can be called independently by any downstream packages that might need Run-Time Interface Reflection, e.g. introspection tools, rosbag transport, etc. +This feature can then be combined with the ability to detect type mismatches and obtain type descriptions in the previous two sections to facilitate communication between nodes of otherwise incompatible types. + +Additionally, it is important to note that this feature is distinct from ROS 2's existing "dynamic" type support (``rosidl_typesupport_introspection_c`` and ``rosidl_typesupport_introspection_cpp``). +The ``rosidl_typesupport_introspection_c*`` generators generate code at compile time for known types that provides reflection for those types. +This new feature, Run-Time Interface Reflection, will support reflection without generated code at compile time, instead dynamically interpreting the type description to provide this reflection. + +.. TODO:: + + Determine if the generated introspection API needs to continue to exist. + It might be that we just remove it entirely in favor of the new system described here, or at least merge them, as their API (after initialization) will probably be the same. + +.. TODO:: + + (wjwwood) I'm realizing now that we probably need to separate this section into two parts, first the reflection API used by the user and the rmw implementation, and then second the rmw implementation specific part, which we could call "Run-Time Type Support"? + It would be called such because it would be something like "TypeDescription in -> ``rosidl_message_type_support_t`` out"... + This second part is needed to make truly generic subscriptions and publishers, which until now has been kind of assumed in this section about reflection. + +.. TODO:: + + This section needs to be updated to be inclusive to at least Services, and then we can also mention Actions, though they are a combination of Topics and Services and so don't need special support probably. + +Plugin Interface +^^^^^^^^^^^^^^^^ + +As Run-Time Interface Reflection is expected to work across any serialization format, the Run-Time Interface Reflection interface needs to be extensible so that the necessary serialization libraries can be loaded to process the serialized message. +Serialization format support in this case will be provided by writing plugins that wrap the serialization libraries that can then provide the Run-Time Interface Reflection feature with the needed facilities. +Therefore, when initializing a Run-Time Interface Reflection instance it will: + +- enumerate the supported serialization library plugins in the environment, +- match the given serialization format to appropriate plugins, +- select a plugin if more than one matches the criteria, and then +- dynamically load the plugin for use + +These serialization library plugins must implement an interface which provides methods to: + +- determine if the plugin can be used for a given serialization type and version, +- provide an API for reflection of a specific type, given the type's description, + + - mainly including the parsed ``TypeDescription`` instance, but also + - perhaps the original ``.idl`` / ``.msg`` text, and + - perhaps other extra information, + +- provide a ``rosidl_message_type_support_t`` instance given similar information from the previous point + +In particular, providing information beyond the ``TypeDescription``, like the ``.idl`` / ``.msg`` text and the serialization library that was used, may be necessary because there might be serialization library or type support specific steps or considerations (e.g. name mangling or ROS specific namespacing) that would not necessarily be captured in the ``.idl`` / ``.msg`` file. + +Dealing with Multiple Applicable Plugins for A Serialization Format +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +In the case where there exists multiple applicable plugins for a particular serialization format (e.g. when a user's machine has a plugin for both RTI Connext's CDR library and Fast-CDR), the plugin matching should follow this priority order: + +- a user specified override, passed to the matching function, or +- a default defined in the plugin matching function, or else +- the first suitable plugin in alphanumeric sorting order + +.. TODO:: + + (wjwwood) We should mention here that the (implicitly read-only) reflection described here is just one logical half of what we could do with the type description. + We could also provide the ability to dynamically define types and/or write to types using the reflection. + + For example, imagine a tool that subscribes to any type (that's just using the RTIR) but then modifies one of the fields by name and then re-publishes it. + That tool would need the ability to mutate an instance using reflection then serialize it to a buffer. + You couldn't just iterate with an existing buffer easily, because it would potentially need to grow the buffer. + + Also, consider a "time stamping" tool that subscribes to any message, and then on-the-fly defines a new message that has a timestamp field and a field with the original message in it, fills that out and publishes it. + That would need the ability to both create a new type from nothing (maybe no more than creating a ``TypeDescription`` instance) as well as a read-write interface to the reflection. + We don't have to support these things in this REP, but we should at least mention them and state if it is in or out of the scope of the REP. + +Tools for Evolving Types +------------------------ + +The previous sections of the specification have all be working up to supporting the new tools described in this section. + +The goal of these new tools is to help users reason about type versions of ROS interfaces, define transfer functions between two versions of a type when necessary, and put the transfer functions to use in their system to handle changes in types that have occurred. + +These tools will come in the form of new APIs, integrations with the build system, command-line tools, and integration with existing concepts like launch files. + +This section will describe a vision of what is possible with some new tools, but it should not be considered completely prescriptive. +That is to say, after this REP is accepted, tools described in this section may continue to evolve and improve, and even more tools not described here may be added to enable more things. +As always, consider the latest documentation for the various pieces of the reference implementation to get the most accurate details. + +Tools for Interacting with Type Version Hashes and Type Descriptions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ros2 command line tools, and the APIs that support them, should be updated to provide access to the type version hash where ever the type name is currently available and the type version description on-demand as well. +For example: + +- ``ros2 interface`` should be extended with a way to calculate the version hash for a type +- ``ros2 topic info`` should include the type version hash used by each endpoint +- ``ros2 topic info --verbose`` should include the type description used by each endpoint +- a new command to compare the types used by two endpoints, e.g. ``ros2 topic compare `` +- ``ros2 service ...`` commands should also be extended in this way +- ``ros2 action ...`` commands should also be extended in this way + +Again this list should not be considered prescriptive or exhaustive, but gives an idea of what should change. + +Notifying Users when Types Do Not Match +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There should be a warning to users when two endpoints (publisher/subscription or Service server/client, etc.) on the same topic (or service/action) are using different type names or versions and therefore will not communicate. +This warning should be a console logging message, but it should also be possible for the user to get a callback in their own application when this occurs. +How this should happen in the API is described in the Type Version Enforcement section, and it uses the "QoS Event" part of the API. + +The warning should include important information, like the GUID of the endpoints involved, the type name(s) and type version hash(es), and of course the topic name. +These pieces of information can then be used by the user, with the above mentioned changes to the command line tools, to investigate what is happening and why the types do not match. +The warning may even suggest how to compare the types using the previously described ``ros2 topic compare`` command-line tool. + +Tools for Determining if Two Type Versions are Convertible +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There should be a way for a user to determine if it is possible to convert between two versions of a type without writing a transfer function. +This tool would look at the two type descriptions and determine if a conversion can be done automatically with a generated transfer function. +This will help the user know if they need to write a transfer function or check if one already exists. + +This tool might look something like this: + +- ``ros2 interface convertible --from-remote-node --to-local-type `` + +This tool would query the type description from a remote node, and compare it to the type description of the second type version found locally. +If the two types can be converted without writing any code, then this tool would indicate that on the ``stdout`` and by return code. + +There could be ``--from-local-type`` and ``--to-remote-node`` options as well as others too. + +.. TODO:: + + (wjwwood) we need to come back to this section when we start working on the reference implementation, as more details will be clearer then + +Tools for Writing User-Defined Transfer Functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To help users write their own transfer functions, when conversions cannot be done automatically, we need to provide them with build system support and API support in various languages. + +The transfer functions will be defined (at least) as a C++ plugin, in the style of component nodes, rviz display plugins, and other instances. +These transfer function plugins can then be loaded and chained together as needed inside of a component node. + +.. TODO:: + + (wjwwood) we need to figure out if this is how we're gonna handle Services, or if instead they will be plugins rather than component nodes, but component nodes with remapping solves a lot of QoS related issues that arise when you think about instead making the conversions happen as a plugin on the publisher or subscription side with topics. + +At the very least we should: + +- provide a convenient way to build and install transfer functions from CMake projects +- provide an API to concisely define transfer functions in C/C++ +- document how transfer functions are installed, discovered, and run + +Documenting how this works will allow support for other build systems and programming languages can be added in the future. +It will also allow users that do not wish to use our helper functions (and the dependencies that come with them) in their projects, e.g. if they would like to use plain CMake and not depend on things like ``ament_cmake``. + +The process will look something like this: + +- compile the transfer function into a library +- put an entry into the ``ament_index`` which includes the transfer functions details, like which type and versions it converts between, and which library contains it + +Putting it into a library allows us to later load the transfer function, perhaps along side other transfer functions, into a single component node which subscribes to the input type topic and publishes to the output type topic. + +The interface for a transfer function will look like a modified Subscription callback, receiving a single ROS message (or request or response in the case of Services) as input and returning a single ROS message as output. +The input will use the Run-Time Interface Reflection API and therefore will not be a standard message structure, e.g. ``std_msgs::msg::String`` in C++, but the output may be a concrete structure if that type is available when compiling the transfer function. + +The transfer function API for C++ may look something like this: + +.. code:: + + void + my_transfer_function( + const DynamicMessage & input, + const MessageInfo & input_info, // type name/version hash, etc. + DynamicMessage & output, + const MessageInfo & output_info) + { + // ... conversion details + } + + REGISTER_TRANSFER_FUNCTION(my_transfer_function) + +Registering a transfer function in an ``ament_cmake`` project might look like this: + +.. code:: + + create_transfer_function( + src/my_transfer_function.cpp + FUNCTION_NAME my_transfer_function + FROM sensor_msgs/msg/Image RIHS01_XXXXXXXXXXXXXXXXX123 + TO sensor_msgs/msg/Image RIHS01_XXXXXXXXXXXXXXXXXabc + ) + +This CMake function would create the library target, link the necessary libraries, and install any files in the ``ament_index`` needed to make the transfer function discoverable by the tools. +This example is the for the simplest case, which we should make easy, but other cases like supporting multiple transfer functions per library, creating the target manually, or even supporting other programming languages would require more complex interfaces too. + +.. TODO:: + + (wjwwood) come back with more details of this interface as the reference implementation progresses + +Tools for Interacting with Available Transfer Functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +There should be a way to list and get information about the available transfer functions in your local workspace. +This tool would query the ``ament_index`` and list the available transfer functions found therein. + +The tool might look like this: + +- ``ros2 interface transfer_functions list`` +- ``ros2 interface transfer_functions info `` + +The tool might also have options to filter based on the type name, package name providing the type, package name providing the transfer function (not necessarily the same as the package which provides the type itself), etc. + +.. TODO:: + + (wjwwood) not clear to me if the transfer functions should have names or not. What if you have more than one transfer function for a type versions pair, e.g. more than one conversion for sensor_msgs/msg/Image from ABC to XYZ. I'm not sure why this would happen, but it is technically possible since the packages register them separately. + +Tools for Using Transfer Functions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Additionally, there should be a tool which will look at the available transfer functions, including the possible automatic conversions, and start the node that will do the conversions if there exists a set of transfer function which would convert between two different versions of a type. + +It might look like this: + +.. code:: + + ros2 interface convert_topic_types \ + --from /topic/old \ + --to /topic \ + --component-container + +This tool would find the set of transfer functions between the types on ``/topic/old`` and ``/topic``, assuming they are the same type at different versions and that there exists the necessary transfer functions, and load them into a component container as a component node. + +The tool could run a stand-alone node instance if provided a ``--stand-alone`` option instead of the ``--component-container`` option. + +It could also have options like ``--check`` which would check if it could convert the types or not, giving detailed information about why it cannot convert between them if it is not possible. +This could help users understand what transfer functions they need to write to bridge the gap. + +This tool would also potentially need to let the user decide which path to take if there exists multiple chains of transfer functions from one version to the other. +For example, if the types are ``Type@ABC`` and ``Type@XYZ``, and there are transfer functions for ``Type@ABC -> Type@DEF``, ``Type@DEF -> Type@XYZ``, ``Type@ABC -> Type@XYZ``, and possibly other paths, then it might need help from the user to know which path to take. +By default it might choose the shortest path and then if there's a tie, one arbitrarily. + +There would be similar versions of this tool for Services and Actions as well. + +Integration with Launch +^^^^^^^^^^^^^^^^^^^^^^^ + +With the previously described tool ``ros2 interface convert_topic_types``, you could simply remap topics and execute an instance of it with launch: + +.. code:: + + + + + + + + + +However, we can provide some "syntactic sugar" to make it easier, which might look like this: + +.. code:: + + + + + + + + +That syntactic sugar can do a lot of things, like: + +- remap the topic to something new, like ``/topic/XXX`` +- run the node its enclosed in, wait for the topic to come up to check the version +- run any transfer functions in a node to make the conversion from ``/topic/XXX`` to ``/topic`` + +This could also be extended to support component nodes, rather than stand alone nodes, etc. + +Integration with ros2bag +^^^^^^^^^^^^^^^^^^^^^^^^ + +Similar to the integration with launch, you could run ``ros2 bag play ...`` and the ``ros2 interface convert_topic_types ...`` separately, but we could also provide an option to rosbag itself, which might look like this: + +.. code:: + + ros2 bag play /path/to/bag --convert-to-local-type /topic + +That option would handle the conversion on the fly using the same mechanisms as the command line tool. + +Rationale +========= + +This section captures further rationales for why the specification is the way it is, and also offers summaries of alternatives considered but not selected for various parts of the specification. + +Type Version Enforcement +------------------------ + +Alternatives +^^^^^^^^^^^^ + +Use Type Hash from Middleware, e.g. from DDS-XTypes +""""""""""""""""""""""""""""""""""""""""""""""""""" + +Type hash can be obtained by the native middleware api. For example, with fastDDS, the type hash can be obtained with ``TypeIdentifier->equivalence_hash()`` during the ``on_type_discovery()`` callback. the rmw layer can choose to use the provided hash to impose the aforementioned type enforcement. + +.. TODO:: + + (wjwwood) this needs to be cleaned up + +Evolving Message Definition with Extensible Type +"""""""""""""""""""""""""""""""""""""""""""""""" + +When defining the ``.idl`` msg file, user can choose to apply annotations to the message definition (DDS XTypes spec v1.3: 7.3.1.2 Annotation Language). +Evolving message type can be achieved by leveraging optional fields and inheritance. +For example, the ``Temperature.idl`` below uses ``@optional`` and ``@extensibility`` in the message definition. + +.. code:: + + @extensibility(APPENDABLE) + struct Temperature + { + unsigned long long timestamp + long long temperature + @optional double temperature_float + }; + +Furthermore, an initial test evolving messages with FastDDS, Cyclone, and Connext middleware implementations show that ``@appendable`` and ``@optional`` are implemented in Cyclone and Connext, but not FastDDS (as of Jul 2022). + +.. TODO:: + + (wjwwood) we need to follow up on this + +Handle Detection of Version Mismatch "Above" rmw Layer +"""""""""""""""""""""""""""""""""""""""""""""""""""""" + +We can choose to utilize ``USER_DATA`` QoS to distribute the message version during discovery phase. +The message version for each participant will then be accessible across all available nodes. By getting the version hash through ``user_data`` via the ``rmw`` layer, similar type version matching can be detected. + +Prevent Communication of Mismatched Versions "Above" rmw Layer +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +TODO + +Type Description Distribution +----------------------------- + +Using a Single ROS Service per Node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The node that is publishing the data must already have access to the correct type description, at the correct version, in order to publish it, and therefore it is natural to get the data from that node. +Similarly, a subscribing node also knows what type they are wanting to receive, both in name and version, and therefore it is again natural to get that information from the subscribing node. +The type description for a given type, at a given version, could have been retrieved from other places, e.g. a centralized database, but the other alternatives considered would have had to take care to ensure that it had the right version of the message, which is not the case for the node publishing the data. + +Because the interface for getting a type description is generic, it is not necessary to have this interface on a per entity, i.e. publisher, subscription, etc, basis, but instead to offer the ROS Service on a per node basis to reduce the number of ROS Services. +Therefore, the specification dictates that the type description is distributed by single ROS Service for each individual node. + +There were also multiple alternatives for how to get this information from each node, but the use of a single ROS Service was selected because the task of requesting the type description from a node is well suited to a request-response style ROS Service. +Some of the alternatives offered other benefits, but using a ROS Service introduced the fewest dependencies, feature-wise, while accomplishing the task. + +.. TODO:: + + cite the above, https://en.wikipedia.org/wiki/Request%E2%80%93response + +Combining the Raw and Parsed Type Description in the Service Response +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The contents of the "get type description" service response should include information that supports both aforementioned use cases (i.e. tools and Run-Time Interface Reflection). +These use cases have orthogonal interests, with the former requiring human-readable descriptions, and the latter preferring machine-readable descriptions. + +Furthermore, the type description should be useful even across middlewares and serialization libraries and that makes it especially important to send at least the original inputs to the "type support pipeline" (i.e. the process of taking user-defined types and generating all supporting code). +In this case, because the "type support pipeline" is a lossy process, there is a need to ensure that enough information is sent to completely reproduce the original definition of the type, and therefore it makes sense to just send the original ``idl`` or ``msg`` file. + +At the same time, it is useful to send information with the original description that makes it easier to process data at the receiving end, as it is often not trivial to get to the "parsed" version of the type description from the original text description. + +Finally, while there could be an argument for sending a losslessly compressed version of the message file, the expected low frequency of queries to the type description service incurs a negligible overhead that heavily reduces the benefit. + +Implementing in ``rcl`` versus ``rmw`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While it is true that implementing the type description distribution on the ``rmw`` layer would allow for much lower level optimization, removing the layer of abstraction avoids having to implement this feature in each rmw implementation. + +Given that the potential gains from optimization will be small due to how infrequently the service is expected to be called, this added development overhead was determined to not be worth it. +Instead the design prefers to have a unified implementation of this feature in ``rcl`` so it is agnostic to any middleware implementations and client libraries. + +Nested ``TypeDescription`` Example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``TypeDescription`` message type shown above also supports the complete description of a type that contains other types (a nested type), up to an arbitrary level of nesting. +Consider the following example: + +.. code:: + + # A.msg + B b + C c + + # B.msg + bool b_bool + + # C.msg + D d + + # D.msg + bool d_bool + +The corresponding ``TypeDescription`` for ``A.msg`` will be as follows, with the referenced type descriptions accessible as ``IndividualTypeDescription`` types in the ``referenced_type_descriptions`` field of ``A``: + +.. code:: + + # A: TypeDescription + type_description: A_IndividualTypeDescription + referenced_type_descriptions: [B_IndividualTypeDescription, + C_IndividualTypeDescription, + D_IndividualTypeDescription] + +Note that the type description for ``A`` itself is found in the ``type_description`` field instead of the ``referenced_type_descriptions`` field. +Additionally, in the case where a type description contains no referenced types (i.e., when it has no fields, or all of its fields are primitive types), the ``referenced_type_descriptions`` array will be empty. + +.. code:: + + # A: IndividualTypeDescription + type_name: "A" + fields: [A_b_Field, A_c_Field] + + # B: IndividualTypeDescription + type_name: "B" + fields: [B_b_bool_Field] + + # C: IndividualTypeDescription + type_name: "C" + fields: [C_d_Field] + + # D: IndividualTypeDescription + type_name: "D" + fields: [D_d_bool_Field] + +With the corresponding ``Field`` fields: + +.. code:: + + # A_b_Field + field_type: 0 + field_name: "b" + nested_type_name: "B" + + # A_c_Field + field_type: 0 + field_name: "c" + nested_type_name: "C" + + # B_b_bool_Field + field_type: 9 # Suppose 9 corresponds to a boolean field + field_name: "b_bool" + nested_type_name: "" # Empty if primitive type + + # C_d_Field + field_type: 0 + field_name: "d" + nested_type_name: "D" + + # D_d_bool_Field + field_type: 9 + field_name: "d" + nested_type_name: "" + +In order to handle the type of a nested type such as ``A``, the receiver can use the ``referenced_type_descriptions`` array as a lookup table keyed by the value of ``Field.nested_type_name`` or ``IndividualTypeDescription.type_name`` (which will be identical for a given type) to obtain the type information of a referenced type. +This type handling process can also support any recursive level of nesting (e.g. while handling A, C is encountered as a nested type, C can then be looked up using the top level ``referenced_type_descriptions`` array). + +Alternatives +^^^^^^^^^^^^ + +Other Providers of Type Description +""""""""""""""""""""""""""""""""""" + +Several other candidate strategies for distributing the type descriptions were considered but ultimately discarded for one or more reasons like: causing a strong dependency on a particular middleware or a third-party technology, difficulties with resolving the message type description locally, difficulties with finding the correct entity to query, or causing network throughput issues. + +These are some of the candidates that were considered, and the reasons for their rejection: + +- Store the type description as a ROS parameter + * Causes a mass of parameter event messages being sent at once on init, worsening the network initialization problem +- Store the type description on a centralized node per machine + * Helps reduce network bandwidth, but makes it non-trivial to find the correct centralized node to query, and introduces issues of resolving the local message package, such as when nodes are started from different sourced workspaces. +- Send type description alongside discovery with middlewares + * Works very well if supported, but is only supported by some DDS implementations (which support XTypes or some other way to attach discovery metadata), but causes a strong dependency on DDS. +- Send type description using a different network protocol + * Introduces additional third-party dependencies separate from ROS and the middleware. + +Alternative Type Description Contents and Format +"""""""""""""""""""""""""""""""""""""""""""""""" + +A combination of the original ``idl`` / ``msg`` file and any other information needed for serialization and deserialization being sent allows for one to cover the weaknesses of the other. +Specifically, given that certain use-cases (e.g., ``rosbag``) might encounter situations where consumers of a message are using a different middleware or serialization scheme the message was serialized with, it becomes extremely important to send enough information to both reconstruct the type support, and also allow the message fields to be accessed in a human readable fashion to aid in the writing of transfer functions. +As such, it is not a viable option to only send one or the other. + +Additionally, the option to add a configuration option to choose what contents to receive from the service server was disregarded due to how infrequently the type description query is expected to be called. + +As for the format of the type description, using the ROS interfaces to describe the type, as opposed to an alternative format like XML, JSON, or something like the TypeObject defined by DDS-XTypes, makes it easier to embed in the ROS Service response. +It also prevents unnecessary coupling with third-party specifications that could be subject to change and reduces the formats that need to be considered on the receiving end of the ROS Service call. + +Representing Fields as An Array of Field Types +"""""""""""""""""""""""""""""""""""""""""""""" + +The use of an array of ``Field`` messages was balanced against using two arrays in the ``IndividualTypeDescription`` type to describe the field types and field names instead, e.g.: + +.. code:: + + # Rejected IndividualTypeDescription Variants + + # String variant + string type_name + string field_types[] + string field_names[] + + # uint8_t Variant + string type_name + uint8_t field_types[] + string field_names[] + +The string variant was rejected because using strings to represent primitive types wastes space, and will lead to increased bandwidth usage during the discovery and type distribution process. +The uint8_t variant was rejected because uint8_t enums are insufficiently expressive to support nested message types. + +The use of the ``Field`` type, with a ``nested_type_name`` field that defaults to an empty string mitigates the space issue while allowing for support of nested message types. +Furthermore, it allows the fields to be described in a single array, which is easier to iterate through and also reduces the chances of any errors from mismatching the array lengths. + +Using an Array to Store Referenced Types +"""""""""""""""""""""""""""""""""""""""" + +Some alternatives to using an array of type descriptions to store referenced types in a nested type were considered, including: + +- Storing the referenced types inside the individual type descriptions and accessing them by traversing the type description tree recursively instead of using a lookup table. + + - Rejected because the IDL spec does not allow for a type description to store itself, and also because it could possibly introduce duplicate, redundant type descriptions in the tree, using up unnecessary space. + +- Sending referenced types in a separate service call or message. + + - Rejected because needing to collate all of the referenced types on the receiver end introduces additional implementation complexity, and also increases network bandwidth with all the separate calls that must be made. + +Run-Time Interface Reflect +-------------------------- + +TODO + +Plugin Matching and Loading API Example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following is an example of how this plugin matching and loading interface could look like, defining new ``rcl`` interfaces; with a plugin wrapping FastCDR v1.0.24 for serialization of ``sensor_msgs/msg/LaserScan`` messages: + +.. code:: + + // Suppose LaserScanDescription reports that it uses FastCDR v1.0.24 for its serialization + rcl_runtime_introspection_description_t LaserScanDescription = node->get_type_description("/scan"); + + rcl_type_introspection_t * introspection_handle; + introspection_handle->init(); // Locate local plugins here + + // Plugin name: "fastcdr_v1_0_24" + const char * plugin_name = introspection_handle->match_plugin(LaserScanDescription->get_serialization_format()); + rcl_serialization_plugin_t * plugin = introspection_handle->load_plugin(plugin_name); + + // If we wanted to force the use of MicroCDR instead + introspection_handle->match_plugin(LaserScanDescription->get_serialization_format(), "microcdr"); + +Run-Time Interface Reflection API Example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following is an example for how the introspection API could look like. +This example will show a read-only interface. + +It should comprise several components +- a handler for the message buffer, to handle pre-processing (e.g. decompression) +- a handler for the message description, to keep track of message field names of arbitrary nesting level +- handler functions for message buffer introspection + +Also, this example uses the LaserScan message definition: https://github.com/ros2/common_interfaces/blob/foxy/sensor_msgs/msg/LaserScan.msg + +.. TODO:: + + (methylDragon) Add a reference somehow? + +First, the message buffer handler: + +.. code:: + + struct rcl_buffer_handle_t { + const void * buffer; // The buffer should not be modified + + const char * serialization_type; + rcl_serialization_plugin_t * serialization_plugin; + rcl_runtime_introspection_description_t * description; // Convenient to have + + // And some examples of whatever else might be needed to support deserialization or introspection... + void * serialization_impl; + } + +The message buffer handler should allocate new memory if necessary, or store a pointer to the message buffer otherwise in its ``buffer`` member. + +Then, functions should be written that allow for convenient traversal of the type description tree. +These functions should allow a user to get the field names and field types of the top level type, as well as from any nested types. + +.. code:: + + struct rcl_field_info_t { // Mirroring Field + const char * field_name; // This should be an absolute address (e.g. "header.seq", instead of "seq") + + uint8_t type; + const char * nested_type_name; // Populated if the type is not primitive + }; + + // Get descriptions + rcl_runtime_introspection_description_t LaserScanDescription = node->get_type_description("/scan"); + rcl_runtime_introspection_description_t HeaderDescription = node->get_referenced_description(LaserScanDescription, "Header"); + + // All top-level fields from description + rcl_field_info_t ** fields = get_field_infos(&LaserScanDescription); + + // A single field from description + rcl_field_info_t * header_field = get_field_info(&LaserScanDescription, "header"); + + // A single field from a referenced description + rcl_field_info_t * stamp_field = get_field_info(&HeaderDescription, "stamp"); + + // A nested field from top-level description + rcl_field_info_t * stamp_field = get_field_info(&LaserScanDescription, "header.stamp"); + +Finally, there should be functions to obtain the data stored in the message fields. +This could be by value or by reference, depending on what the serialization library supports, for different types. + +There minimally needs to be a family of functions to obtain data stored in a single primitive message field, no matter how deeply nested it is. +These need to be created for each primitive type. + +The rest of the type introspection machinery can then be built on top of that family of functions, in layers higher than the C API. + +.. code:: + + rcl_buffer_handle_t * scan_buffer = node->get_processed_buffer(some_raw_buffer); + + // Top-level primitive field + get_primitive_field_float32(scan_buffer, "scan_time"); + + // Nested primitive field + get_primitive_field_uint32_seq(scan_buffer, "header.seq"); + + // Nested primitive field sequence element (overloaded) + get_field_seq_length(scan_buffer, "header.seq"); // Support function + get_primitive_field_uint32(scan_buffer, "header.seq", 0); + +If we attempt to do the same by reference, the plugin might decide to allocate new memory for the pointer, or return a pointer to existing memory. + +.. code:: + + // Nested primitive field + get_primitive_field_uint32_seq_ptr(scan_buffer, "header.seq"); + + // Be sure to clean up any dangling pointers + finalize_field(some_field_data_ptr); + +The following should be error cases: + +- accessing field data as incorrect type +- accessing or introspecting incorrect/nonexistent field names + +.. TODO: (methylDragon) Are there more cases? It feels like there are... + +- the raw message buffer should outlive the ``rcl_buffer_handle_t``, since it is not guaranteed that the buffer handle will allocate new memory +- the ``rcl_buffer_handle_t`` should outlive any returned field data pointers, since it is not guaranteed that the serialization plugin will allocate new memory +- however, ``rcl_field_info_t`` objects **do not** have any lifecycle dependencies, since they are merely descriptors + +Alternatives +^^^^^^^^^^^^ + +Name of Run-Time Interface Reflection +""""""""""""""""""""""""""""""""""""" + +Other names were considered, like "Runtime Type Introspection", but this name was selected for three main reasons: + +- to avoid confusion with C++'s Run-time type information (RTTI)\ [3]_, +- to show it was more than RTTI, but instead was also reflection, + + - like how the RTTR C++ library\ [4]_ differs from normal RTTI, and + +- to show that it deals not just with any "type" but specifically ROS's Interfaces + + +Backwards Compatibility +======================= + +TODO + + +Feature Progress +================ + +Supporting Feature Development: + +- TypeDescription Message: + + - [ ] Define and place the TypeDescription.msg Message type in a package + +- Enforced Type Versioning: + + - [ ] Create library to calculate version hash from message files + - [ ] Create library to calculate version hash from service files + - [ ] Create library to calculate version hash from action files + - [ ] Create command-line tool to show the version hash of an interface + + - [ ] Add topic type version hash to the "graph" API + - [ ] Implement delivery of type version hash during discovery: + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + + - [ ] Add rmw API for notifying user when a topic endpoint was not matched due to type mismatch + + - [ ] Detect when Message versions do not match, prevent matching, notify user: + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + + - [ ] Detect when Service versions do not match, prevent matching, notify user: + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + + - [ ] Detect when Action versions do not match, prevent matching, notify user: + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + +- Type Description Distribution: + + - [ ] Define and place the GetTypeDescription.srv Service type in a package + - [ ] Implement the "get type description" service in the rcl layer + +- Run-time Interface Reflection: + + - [ ] Prototype "description of type" -> "accessing data fields from buffer" for each rmw vendor: + + - [ ] Fast-DDS + - [ ] CycloneDDS + - [ ] RTI Connext Pro + + - [ ] Prototype "description of type" -> "create datareaders/writers" for each rmw vendor: + + - [ ] Fast-DDS + - [ ] CycloneDDS + - [ ] RTI Connext Pro + + - [ ] Implement reflection API in ``rcl_serialization`` package, providing (de)serialization with just a description + - [ ] Implement single "plugin" using Fast-CDR (tentatively) + - [ ] Implement plugin system, allowing for multiple backends for (X)CDR + - [ ] Implement at least one other (X)CDR backend, perhaps based on RTI Connext Pro + + - [ ] Add "description of type" -> ``rosidl_message_type_support_t`` to ``rmw`` API + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + + - [ ] Add "description of type" -> ``rosidl_service_type_support_t`` to ``rmw`` API + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + + - [ ] Update APIs for rclcpp to support using TypeDescription instances to create entities: + + - [ ] Topics + - [ ] Services + - [ ] Actions + + - [ ] Update APIs for rclpy to support using TypeDescription instances to create entities: + + - [ ] Topics + - [ ] Services + - [ ] Actions + +Tooling Development: + +- [ ] Command-line tools for interacting with type version hashes +- [ ] Add a default warning to users when a topic has mismatched types (by type or version) + +- [ ] Command-line tool for determining if two versions of a type are convertible + +- [ ] CMake and C++ APIs for writing transfer functions, including usage documentation +- [ ] Command-line tools for interacting with user-defined transfer functions +- [ ] Command-line tool for starting a conversion node between two versions of a type + +- [ ] Integration with launch, making it easier to set up conversions in launch files + +- [ ] Integration with rosbag2 + + - [ ] Record the type version hash, TypeDescription, and other metadata into bags + - [ ] Use recorded metadata to create subscriptions without needing the type to be built locally + - [ ] Use recorded metadata to create publishers from descriptions in bag + - [ ] Provide a method to playback/update old bag files that do not have type descriptions stored + + +References +========== + +.. [1] DDS-XTYPES 1.3 + (https://www.omg.org/spec/DDS-XTypes/1.3/About-DDS-XTypes/) + +.. [2] IDL - Interface Definition and Language Mapping + (http://design.ros2.org/articles/idl_interface_definition.html) + +.. [3] Run-time type information + (https://en.wikipedia.org/wiki/Run-time_type_information) + +.. [4] RTTR (Run Time Type Reflection): An open source library, which adds reflection to C++ + (https://www.rttr.org/) + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From 52244cf52d8152eb73b52357cc2ec6fc769cec4d Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Tue, 27 Jun 2023 11:58:34 -0700 Subject: [PATCH 03/10] latest Signed-off-by: Emerson Knapp --- rep-2011.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/rep-2011.rst b/rep-2011.rst index 770797a8..05eff46b 100644 --- a/rep-2011.rst +++ b/rep-2011.rst @@ -13,13 +13,14 @@ Abstract This REP proposes patterns and approaches for evolving message, service, and action ROS interface types over time. -The proposed patterns use the existing default serialization technology, i.e. CDR, in combination with new tooling to detect when types have changes and new tooling to convert between versions. +The proposed patterns use the existing default serialization technology, i.e. CDR, in combination with new tooling to detect when types have changed and new tooling to convert between versions. However, it should be possible to use features provided by different, perhaps future, serialization technologies in addition to the approaches proposed in this REP. Specifically, this REP proposes that we provide tooling to convert from old versions of types to newer versions of types on demand, using user-defined transfer functions. This approach is a good way to achieve backwards compatibility in messages over long periods of time and in a variety of scenarios, e.g. over the wire, converting bag files, or in specialized tools. This approach does not rely on specific features of the serialization technology and instead relies on the ability to communicate the full type descriptions on the wire, beyond their name and version, and use that description to introspect the values of it at runtime using reflection. +That feature set is described in detail in REP-2016\ [#rep2016]_. This approach can be used in conjunction with serialization technology specific features like optional fields, inheritance, etc., but it works even with the simplest serialization technologies. This is true so long as we have the ability to introspect the messages at runtime without prior knowledge of the type description, which is a feature we also need for generic introspection tools like ``rosbag2`` and ``ros2 topic echo``. @@ -1291,7 +1292,7 @@ Supporting Feature Development: - TypeDescription Message: - - [ ] Define and place the TypeDescription.msg Message type in a package + - [x] Define and place the TypeDescription.msg Message type in a package - Enforced Type Versioning: @@ -1399,6 +1400,9 @@ Tooling Development: References ========== +.. [#rep2016] REP 2016: ROS 2 Interface Type Descriptions + (https://www.ros.org/reps/rep-2016.html) + .. [1] DDS-XTYPES 1.3 (https://www.omg.org/spec/DDS-XTypes/1.3/About-DDS-XTypes/) From 63d876f594972c5e8b6348f7cdaf08a4e38fa98d Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Wed, 28 Jun 2023 14:28:15 -0700 Subject: [PATCH 04/10] Working on type hash reasoning Signed-off-by: Emerson Knapp --- rep-2011.rst | 85 +++++----------------------------------------------- rep-2016.rst | 51 +++++++++++++++++++------------ 2 files changed, 40 insertions(+), 96 deletions(-) diff --git a/rep-2011.rst b/rep-2011.rst index 05eff46b..8f4a351a 100644 --- a/rep-2011.rst +++ b/rep-2011.rst @@ -141,15 +141,11 @@ The proposal is to provide tooling to help users: In order to do this, this proposal describes how ROS 2 can be changed to support these tools by: -- enforcing type compatibility by version - - - by providing type version hashes, and - - making it possible to see what versions of types are being used by other endpoints, and - - warning users when type enforcement is preventing two endpoints from communicating - -- providing access to the complete type description of types being used +- Using REP-2016 Type Description features to - - and making it possible to access the type description from nodes remotely + - detect type mismatches + - warn users when type enforcement is preventing two endpoints from communicating + - access type descriptions from nodes remotely - making it possible to publish and subscribe to topics using just the type description @@ -161,8 +157,7 @@ This Specification section covers the conceptual overview in more detail, then d Conceptual Overview ------------------- -Users will be able to calculate the "type version hash" for an interface (e.g. a message, service, or action) using the ``ros2 interface hash `` command. -This hash is also used by ROS 2 to determine if the same type name but different type versions are being used on the same topic, so that a warning may be logged that the endpoints that do not match may not communicate. +The REP-2016 Type Hash for an interface is used by ROS 2 to determine if the same type name but different type versions are being used on the same topic, so that a warning may be logged that endpoints that do not match may not communicate. .. note:: @@ -223,7 +218,7 @@ It will continue to do this until it reaches the desired type version or it fail │ │ └─────────────────────────────────┘ -Once the set of necessary transfer functions has been identified, the ROS graph can be changed to have one side of the topic be remapped onto a new topic name which indicates it is of a different version that what is desired, and then the transfer function can be run as a component node which subscribes to one version of the message, performs the conversion using the chain of transfer functions, and then publishes the other version of the message. +Once the set of necessary transfer functions has been identified, the ROS graph can be changed to have one side of the topic be remapped onto a new topic name which indicates it is of a different version than what is desired, and then the transfer function can be run as a component node which subscribes to one version of the message, performs the conversion using the chain of transfer functions, then publishes the other version of the message. Tools will assist the user in making these remappings and running the necessary component nodes with the appropriate configurations, either from their launch file or from the command line. .. TODO:: @@ -238,10 +233,9 @@ Once the mismatched messages are flowing through the transfer functions, communi Services, since they are not sensitive to the many-to-many (many publisher) issue, unlike Topics, and because they do not have as many QoS settings that apply to them, they can probably have transfer functions that are plugins, rather than separate component nodes that repeat the service call, like the ros1_bridge. Actions will be a combination of topics and services, but will have other considerations in the tooling. -In order to support this vision, three missing features will need to be added into ROS 2 (which were also mentioned in the introduction): +In order to support this vision, these features will have been added into ROS 2 (which were also mentioned in the introduction): - enforcing type compatibility by version -- providing access to the complete type description of types being used - making it possible to publish and subscribe to topics using just the type description These features are described in the following sections. @@ -249,73 +243,10 @@ These features are described in the following sections. Type Version Enforcement ------------------------ -In order to detect and enforce type version mismatches, as well as communicate information about type descriptions compactly, a way to uniquely identify versions is required. -This proposal uses a hash of the type's description for this purpose. - - -Type Version Hash -^^^^^^^^^^^^^^^^^ - -The hash must be calculated in a stable way such that it is not changed by trivial differences that do not impact serialization compatibility. -The hash must also be able to be calculated at runtime from information received on the wire. -This allows subscribers to validate the received TypeDescriptions against advertised hashes, and allows dynamic publishers to invent new types and advertise their hash programmatically. -The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the TypeDescription object as an intermediate representation. -This way, types coming from two sources that have the same stated name and are wire compatible will be given the same hash, even if they are defined using source text that is not exactly equal. -The hash must only be computed using fields that affect communication compatibility. Thus the representation should include: - -This representation includes: - -- the package, namespace, and type name, for example `sensor_msgs/msg/Image` -- a list of field names and types -- a list of all recursively referenced types -- no field default values -- no comments - -Finally, the resulting filled data structure must be represented in a platform-independent format, rather than running the hash function on the in-memory native type representation. -Different languages, architectures, or compilers will produce different in-memory representations, and the hash must be consistently calculable in different contexts. - -The resulting data structure is hashed using SHA-256, resulting in a 256-bit (32-byte) hash value which is also generally known as a "message digest". -This hash is paired with a type version hash standard version, which we will call the "ROS IDL Hashing Standard" or "RIHS", the first version of which will be ``RIHS01``. -RIHS Version 00 is reserved for "Invalid" / "unset", and the RIHS version is limited by this specification to a maximum value of 255. -RIHS hash values must have a well-defined UTF-8 string representation for human readability and for passing over string-only communication channels. -The prefix of a well-formed RIHS string will always be ``RIHSXX_``, where ``X`` is one hexadecimal digit, followed by the version-dependent string representation of the hash value. -For ``RIHS01``, the hash value is 64 hexadecimal digits representing the 256-bit message digest, leading to a known ``RIHS01`` string length of 71. - -This versioning allows the tooling to know if a hash mismatch is due to a change in this standard (how hash is computed) or due to a difference in the interface types themselves. -In the case of a change in standard, it will be unknown whether the interface types are equal or not. - -For now, the list of field names and their types are the only contributing factors, but in the future that could change, depending on which "annotations" are supported in ``.idl`` files. -The "IDL - Interface Definition and Language Mapping" design document\ [2]_ describes which features of the OMG IDL standard are supported by ROS 2. -If that is extended in the future, then this data structure may need to be updated, and if so the "ROS IDL Hashing Standard" version will also need to be incremented. -New sanitizing may be needed on the TypeDescription pre-hash procedure, in the case of these new features. - -.. TODO:: - - Re-audit the supported features from OMG IDL according to the referenced design document, including the @key annotation and how it may impact this for the reference implementation. - -Notes: - -The type version hash is not sequential and does not imply any rank among versions of the type. That is, given two version hashes of a type, there is no way to tell which is "newer". - -Because the hash contains the stated name of the type, differently-named types with otherwise identical descriptions will be mismatched as incompatible. -This matches existing ROS precedent of strongly-typed interfaces. - -The type version hash can only be used to determine if type versions are equal and if there exists a chain of transfer functions that can convert between them. -Because of this, when a change to a type is made, it may or may not be necessary to write transfer functions in both directions depending on how the interface is used. - -It may be desirable, as a user, to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. -There is no built-in way to do this manual re-versioning. -However, we suggest the following method which requires no special tooling: provide an extra field within the interface with a name ``bool versionX = true``. -To manually trigger a hash update, change by increment the name of the field, for example ``bool versionY = true``. - -The TypeDescription does not include the serialization format being used, nor does it include the version of the serialization technology. -This type version hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. -The type version hash must be considered in context, with the serialization format and version in order to determine wire compatibility. - Enforcing Type Version ^^^^^^^^^^^^^^^^^^^^^^ -The type version hash may be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate. +The Type Hash may be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate. Again, whether or not it is used will depend on the underlying middleware and how it determines if types are compatible between endpoints. Simpler middlewares will not do anything other than check that the type names match, in which case the version hash will likely be used. However, in more sophisticated middlewares type compatibility can be determined using more complex rules and by looking at the type descriptions themselves, and in those cases the type version hash may not be used to determine matching. diff --git a/rep-2016.rst b/rep-2016.rst index b3e0d01b..db617c16 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -20,34 +20,40 @@ Alongside the description format, this REP also defines a hashing algorithm to c Motivation ========== -ROS 1 clients provided an MD5 sum from msg files to check for matching definitions. +ROS 1 clients provided an MD5 sum from msg files to check for matching definitions, as well as full message definitions in connection headers. +ROS 2 before Iron did not provide any comparable capabilities. +This REP proposes a set of standards to be implemented in ROS 2 that allow users to easily + +- detect mismatches in message types between nodes +- get access to the complete type description of types being used, including from remote nodes Terminology =========== +TODO: formatting + Type Source - original text used to define a type Type Description - a data structure representating a parsed Type Source, which removes irrelevant data such as comments, and will be equal regardless of whether the source was msg, IDL, or otherwise. RIHS (ROS Interface Hashing Standard) - a versioned specification for producing a hash value for a Type Description - +type_description_interfaces - the new package defined to contain the Type Description data structures Specification ============= -In order to communicate and compare types, Type Description Interfaces are defined to contain type information. +In order to communicate and compare types, a Type Description data structure is defined to contain type information in a common format. To detect and enforce type version mismatches, and communicate information about type descriptions compactly, a way to uniquely identify types is required. -This proposal uses a hash of the type's description for this purpose. +For this purpose a hash of the Type Description, or Type Hash, is used. +The Type Hash specification, covered below, defines a common algorithm that guarantees cross-implementation consistency. -Type Description Interfaces ---------------------------- +Type Description +---------------- -A Type Description must be produced in a stable way such that it is not changed by trivial differences that do not impact compatibility. +A Type Description must be produced such that it is stable across consistent meaning, even with trivial changes to source text. The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the TypeDescription object as an intermediate representation. -This way, types coming from two sources that have the same stated name and have the same information will be given the same description, even if they are defined using sources that do not have identical text. -The description should only contain information needed for programmatic reasons. -Thus the representation should include: +This way, types coming from two sources that have the same stated name and have the same information will be given the same description, even if they are defined using sources in differing specification languages. -This representation includes: +Thus the representation includes: - the package, namespace, and type name, for example `sensor_msgs/msg/Image` - a list of field names and types @@ -97,10 +103,11 @@ This matches existing ROS precedent of strongly-typed interfaces. The type version hash can only be used to determine if type versions are equal and if there exists a chain of transfer functions that can convert between them. Because of this, when a change to a type is made, it may or may not be necessary to write transfer functions in both directions depending on how the interface is used. -It may be desirable, as a user, to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. -There is no built-in way to do this manual re-versioning. -However, we suggest the following method which requires no special tooling: provide an extra field within the interface with a name ``bool versionX = true``. -To manually trigger a hash update, change by increment the name of the field, for example ``bool versionY = true``. +.. note:: + A message provider may desire to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. + There is explicitly no built-in provision for this case. + We suggest the following method - provide an extra field within the interface with a name like ``bool versionX = true``. + To trigger a hash update, increment the name of this special field, for example to ``bool versionY = true``. The TypeDescription does not include the serialization format being used, nor does it include the version of the serialization technology. This type version hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. @@ -117,8 +124,8 @@ For DDS implementations of the RMW API, it is recommended but not required to us This discovery-time hash availability allows for validation of type mismatch before ever requesting a subscription. -It can also give subscription-side tooling the opportunity to obtain an appropriate type description for the given hash, if it is not available. -That point brings us to the final feature of this REP. +It can also give subscription-side tooling the opportunity to obtain the type description for the given hash. + Type Description Distribution ----------------------------- @@ -127,12 +134,18 @@ Type Description Distribution The service must be optional, but it will be a detail decided by client libraries whether it is enabled or disabled by default. +Supporting Tooling +================== + +- ros2 topic info -v (done - TODO more info) +- ros2 interface show (TODO implement) + References ========== .. http://wiki.ros.org/Topics -.. REP 2011 Evolving message types -.. REP 20XX Dynamic pubsub (name TBD) +.. REP 2011 Evolving message types (TODO link) +.. REP 20XX Dynamic pubsub (name TBD - TODO) Copyright ========= From fc321a093f63b24d86720a0d845884cc5821fe7a Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Mon, 3 Jul 2023 23:07:27 -0700 Subject: [PATCH 05/10] Full pass removal through 2011 complete Signed-off-by: Emerson Knapp --- rep-2011.rst | 473 +++------------------------------------------------ rep-2016.rst | 446 +++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 466 insertions(+), 453 deletions(-) diff --git a/rep-2011.rst b/rep-2011.rst index 8f4a351a..04d0a3bb 100644 --- a/rep-2011.rst +++ b/rep-2011.rst @@ -7,7 +7,6 @@ Content-Type: text/x-rst Created: 30-Nov-2021 Post-History: - Abstract ======== @@ -129,6 +128,7 @@ Terminology TODO +- "type version" Specification ============= @@ -157,12 +157,12 @@ This Specification section covers the conceptual overview in more detail, then d Conceptual Overview ------------------- -The REP-2016 Type Hash for an interface is used by ROS 2 to determine if the same type name but different type versions are being used on the same topic, so that a warning may be logged that endpoints that do not match may not communicate. +The REP-2016 Type Hash for an interface is used by ROS 2 to determine if the same type name with different type versions are being used on the same topic, so that a warning may be logged that endpoints that do not match may not communicate. .. note:: An exception to this rule is that if the underlying middleware has more sophisticated rules for matching types, for example the type has been extended with an optional field, then they may still match - In that case, ROS 2 will defer to the middleware and not produce warnings when the type version hashes do not match. + In that case, ROS 2 will defer to the middleware and not produce warnings when the type hashes do not match. Instead, ROS 2 will rely on the middleware to notify it when two endpoints do not match based on their types not being compatible, so that a warning can be produced. When a mismatch is detected, the user can use user-defined or automatically generated generic "transfer functions" to convert between versions of the type until it is in the type version they wish to send or receive. @@ -248,38 +248,38 @@ Enforcing Type Version The Type Hash may be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate. Again, whether or not it is used will depend on the underlying middleware and how it determines if types are compatible between endpoints. -Simpler middlewares will not do anything other than check that the type names match, in which case the version hash will likely be used. -However, in more sophisticated middlewares type compatibility can be determined using more complex rules and by looking at the type descriptions themselves, and in those cases the type version hash may not be used to determine matching. +Simpler middlewares may not do anything other than check that the type names match, in which case the hash will likely used as an extra comparison to determine compatibility. +However, in more sophisticated middlewares type compatibility can be determined using more complex rules and by looking at the type descriptions themselves, and in those cases the type hash may not be used to determine matching. When creating a publisher or subscription, the caller normally provides: - a topic name, - QoS settings, and -- a topic type +- a topic type name -Where the topic type is represented as a string and is automatically deduced based on the type given to the function that creates the entity. +Where the topic type name is represented as a string and is automatically deduced based on the type given to the function that creates the entity. The type may be passed as a template parameter in C++ or as an argument to the function in Python. -For example, creating a publisher for the C++ type ``std_msgs::msg::String`` using ``rclcpp`` may result in a topic type like the string ``std_msgs/msg/String``. +For example, creating a publisher for the C++ type ``std_msgs::msg::String`` using ``rclcpp`` may result in a topic type name ``"std_msgs/msg/String"``. All of the above items are used by the middleware to determine if two endpoints should communicate or not, and this REP proposes that the type version be added to this list of provided information. Nothing needs to change from the user's perspective, as the type version can be extracted automatically based on the topic type given, either at the ``rcl`` layer or in the ``rmw`` implementation itself. That is to say, how users create things like publishers and subscription should not need to change, no matter which programming language is being used. -However, the type version hash would become something that the ``rmw`` implementation is provided and aware of in the course of creating a publisher or subscription. +However, the type hash would become something that the ``rmw`` implementation is provided and aware of in the course of creating a publisher or subscription. The decision of whether or not to use that information to enforce type compatibility would be left to the middleware, rather than implementing it as logic in ``rcl`` or other packages above the ``rmw`` API. The method for implementing the detection and enforcement of type version mismatches is left up to the middleware. -Some middlewares will have tools to handle this without this new type version hash and others will implement something like what would be possible in the ``rcl`` and above layers using the type version hash. +Some middlewares will have tools to handle this without the type hash and others will implement something like what would be possible in the ``rcl`` and above layers using the type hash. By keeping this a detail of the ``rmw`` implementation, we allow the ``rmw`` implementations to make optimizations where they can. Recommended Strategy for Enforcing that Type Versions Match ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If the middleware has a feature to handle type compatibility already, as is the case with DDS-XTypes which is discussed later, then that can be used to enforce type safety, and then the type version hash would only be used for debugging and for storing in recordings. +If the middleware has a feature to handle type compatibility already, as is the case with DDS-XTypes which is discussed later, then that can be used to enforce type safety, and then the type hash would only be used for debugging and for storing in recordings. -However, if the middleware lacks this kind of feature, then the recommended strategy for accomplishing this in the ``rmw`` implementation is to simply concatenate the type name and the type version hash with double underscores and then use that as the type name given to the underlying middleware. +However, if the middleware lacks this kind of feature, then the recommended strategy for accomplishing this in the ``rmw`` implementation is to simply concatenate the type name and the type hash with double underscores and then use that as the type name given to the underlying middleware. For example, a type name using this approach may look like this: .. code:: @@ -288,7 +288,7 @@ For example, a type name using this approach may look like this: This has the benefit of "just working" for most middlewares which at least match based on the name of the type, and it is simple, requiring no further custom hooks into the middleware's discovery or matchmaking process. -However, one downside with this approach is that it makes interoperation between ROS 2 and the "native" middleware more difficult, as appending the version hash to the type name is just "one more thing" that you have to contend with when trying to connect non-ROS endpoints to a ROS graph. +However, one downside with this approach is that it makes interoperation between ROS 2 and the "native" middleware more difficult, as appending the type hash to the type name is just "one more thing" that you have to contend with when trying to connect non-ROS endpoints to a ROS graph. Alternative Strategy for Enforcing that Type Versions Match ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -300,7 +300,7 @@ However, in order to increase compatibility between rmw implementations, it is r This recommendation is particularly useful in the case where a DDS based ROS 2 endpoint is talking to another DDS-XTypes based ROS 2 endpoint. The DDS-XTypes based endpoint then has a chance to "gracefully degrade" to interoperate with the basic DDS based ROS 2 endpoint. -This would not be the case if the DDS-XTypes based ROS 2 endpoint did not include the type version hash in the type name, as is suggested with the recommended strategy. +This would not be the case if the DDS-XTypes based ROS 2 endpoint did not include the type hash in the type name, as is suggested with the recommended strategy. .. TODO:: @@ -314,206 +314,33 @@ One potential downside to delegating type matching to the rmw implementation is If ROS 2 is to provide users a warning that two endpoints will not communicate due to their types not matching, it requires there to be a way for the middleware to notify the ROS layer when a topic is not matched due to the type incompatibility. As some of the following sections describe, it might be that the rules by which the middleware decides on type compatibility are unknown to ROS 2, and so the middleware has to indicate when matches are and are not made. -If the middleware just uses the type name to determine compatibility, then the rmw implementation can just check the type version hash, and if they do not match between endpoints then the rmw implementation can notify ROS 2, and a warning can be produced. +If the middleware just uses the type name to determine compatibility, then the rmw implementation can compare type hashes, and if they do not match between endpoints then the rmw implementation can notify ROS 2, and a warning can be produced. Either way, to facilitate this notice, the ``rmw_event_type_t`` shall be extended to include a new event called ``RMW_EVENT_OFFERED_TYPE_INCOMPATIBLE``. Related functions and structures will also be updated so that the event can be associated with specific endpoints. -.. TODO:: - - It's not clear how we will do this just now, since existing "QoS events" lack a way to communicate this information, I (wjwwood) think. - -Accessing the Type Version Hash -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For debugging and introspection, the type version hash will be accessible via the ROS graph API, by extending the ``rmw_topic_endpoint_info_t`` struct, and related types and functions, to include the type version hash, ``topic_type_version_hash``, as a string. -It should be along side the ``topic_type`` string in that struct, but the ``topic_type`` field should not include the concatenated type version hash, even if the recommended approach is used. -Instead the type name and version hash should be separated and placed in the fields separately. - -This information should be transmitted as part of the discovery process. - -This field can be optionally an empty string, in order to support interaction with older versions of ROS where this feature was not yet implemented, but it should be provided if at all possible. - Notes for Implementing the Recommended Strategy with DDS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The DDS standard provides an ``on_inconsistent_topic()`` method on the ``ParticipantListener`` class as a callback function which is called when an ``INCONSISTENT_TOPIC`` event is detected. This event occurs when the topic type does not match between endpoints, and can be used for this purpose, but at the time of writing (July 2022), this feature is not supported across all of the DDS vendors that can be used by ROS 2. -Sending of the type version hash during discovery should be done using the ``USER_DATA`` QoS setting, even if the type version hash is not used for determining type compatibility, and then provided to the user-space code through the ``rmw_topic_endpoint_info_t`` struct. - Interactions with DDS-XTypes or Similar Implicit Middleware Features ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -When using DDS-Xtypes type compatibility is determined through sophisticated and configurable rules, allowing for things like extensible types, optional fields, implicit conversions, and even inheritance. -Which of these features is supported for use with ROS is out of scope with this REP, but if any of them are in use, then it may be possible for two endpoints to match and communicate even if their ROS 2 type version hashes do not match. +When using DDS-Xtypes, type compatibility is determined through sophisticated and configurable rules, allowing for things like extensible types, optional fields, implicit conversions, and even inheritance. +Which of these features is supported for use with ROS is out of scope with this REP, but if any of them are in use, then it may be possible for two endpoints to match and communicate even if their ROS 2 type hashes do not match. In this situation the middleware is responsible for communicating to the rmw layer when an endpoint will not be matched due to type incompatibility. The ``INCONSISTENT_TOPIC`` event in DDS applies for DDS-XTypes as well, and should be useful in fulfilling this requirement. -Type Description Distribution ------------------------------ - -For some use cases the type version hash is insufficient and instead the full type description is required. - -One of those use cases, which is also described in this REP, is "Run-Time Interface Reflection", which is the ability to introspect the contents of a message at runtime when the description for that message, or that version of that message, was unavailable at compile time. -In this use case the type description is used to interpret the serialized data dynamically. - -Another use case, which is not covered in this REP, is using the type description in tooling to either display the type description to the user or to include it in recordings. - -In either case, where the type description comes from doesn't really matter, and so, for example, it could be looked up on the local filesystem or read from a rosbag file. -However, in practice, the correct type description may not be found locally, especially in cases where you have different versions of messages in the same system, e.g.: - -- because it's on another computer, or -- because it is from a different distribution of ROS, or -- because it was built in a different workspace, or -- because the application has not been restarted since recompiling a change to the type being used - -In any case, it is useful to have a mechanism to convey the type descriptions from the source of the data to other nodes, which we describe here as "type description distribution". - -Furthermore, this feature should be agnostic to the underlying middleware and serialization library, as two endpoints may not have the same rmw implementation, or the data may have been serialized to a different format in the case of playback of a recording. - -Sending the Type Description -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Type descriptions will be provided by a ROS Service called ``~/_get_type_description``, which will be offered by each node. -There will be a single ROS Service per node, regardless of the number of publishers or subscriptions on that node. - -This service may be optional, for example being enabled or disabled when creating the ROS Node. - -.. TODO:: - - How can we detect when a remote node is not offering this service? - It's difficult to differentiate between "the Service has not been created yet, but will be" and "the Service will never be created". - Should we use a ROS Parameter to indicate this? - But then what if remote access to Parameters (another optional Service) is disabled? - Perhaps we need a "services offered" list which is part of the Node metadata, which is sent for each node in the rmw implementation, but that's out of scope for this REP. - -A service request to this ROS Service will comprise of the type name and the type version hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described in previous sections. -The ROS Service server will respond with the type description and any necessary metadata needed to do Run-Time Interface Reflection. -This service is not expected to be called frequently, and is likely to only occur when new topic or service endpoints are created, and even then, only if the endpoint type hashes do not match. - -.. TODO:: - - Should each endpoint be asked about their type/version pair or should we assume that the type/version pair guarantees a unique type description and therefore reuse past queries? - (wjwwood) I am leaning towards assuming the type name and type version hash combo as being a unique identifier. - -Type Description Contents and Format -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The response sent by the ROS Service server will contain a combination of the original ``idl`` or ``msg`` file's content, as well as any necessary information to serialize and deserialize the raw message buffers sent on the topic. -The response will contain a version of the description that contains comments from the original type description, as those might be relevant to interpreting the semantic meaning of the message fields. - -Additionally, the response could include the serialization library used, its version, or any other helpful information from the original producer of the data. - -.. TODO:: - - What happens if the message consumer doesn't have access to the serialization library stated in the meta-type? - (wjwwood) It depends on what you're doing with the response. If you are going to subscribe to a remote endpoint, then I think we just refuse to create the subscription, as communication will not work, and there's a basic assumption that if you're going to communicate with an endpoint then you are using the same or compatible rmw implementations, which includes the serialization technology. The purpose of this section and ROS Service is to provide the information, not to ensure communication can happen. - -The ROS 2 message that defines the type description must be able to describe any message type, including itself, and since it is describing the message format, it should work independently from any serialization technologies used. -This "meta-type description" message would then be used to communicate the structure of the type as part of the "get type description" service response. -The final form of these interfaces should be found in the reference implementation, but such a Service interface might look like this: - -.. code:: - - string type_name - string type_version_hash - --- - # True if the type description information is available and populated in the response - bool successful - # Empty if 'successful' was true, otherwise contains details on why it failed - string failure_reason - - # The idl or msg file name - string type_description_raw_file_name - # The idl or msg file, with comments and whitespace - # The file extension and/or the contents can be used to determine the format - string type_description_raw - # The parsed type description which can be used programmatically - TypeDescription type_description - - # Key-value pairs of extra information. - string[] extra_information_keys - string[] extra_information_values - -.. TODO:: - - (wjwwood) I propose we use key-value string pairs for the extra information, but I am hoping for discussion on this point. - This type is sensitive to changes, since changes to it will be hard to roll out, and may need manual versioning to handle. - We could also consider bounding the strings and sequences of strings, so the message could be "bounded", which is nice for safety and embedded systems. - -Again, the final form of these interfaces should be referenced from the reference implementation, but the ``TypeDescription`` message type might look something like this: - -.. code:: - - IndividualTypeDescription type_description - IndividualTypeDescription[] referenced_type_descriptions - -And the ``IndividualTypeDescription`` type: - -.. code:: - - string type_name - Field[] fields - -And the ``Field`` type: - -.. code:: - - FIELD_TYPE_NESTED_TYPE = 0 - FIELD_TYPE_INT = 1 - FIELD_TYPE_DOUBLE = 2 - # ... and so on - FIELD_TYPE_INT_ARRAY = ... - FIELD_TYPE_INT_BOUNDED_SEQUENCE = ... - FIELD_TYPE_INT_SEQUENCE = ... - # ... and so on - - string field_name - uint8_t field_type - uint64_t field_array_size # Only for Arrays and Bounded Sequences - string nested_type_name # Only for FIELD_TYPE_NESTED_TYPE - -These examples of the interfaces just give an idea of the structure but perhaps do not yet consider some other complications like field annotations or other as yet unconsidered features we want to support. - -.. TODO:: - - (wjwwood) Add text about how to handle Service types, which are formed as a Request and a Response part, each of which is kind of like a Message. - We could just treat the Request and Response separately, or we could extend this scheme to include explicit support for Services. - Since Actions are composed of Topics and Services, it is less so impacted, but we could similarly consider officially supporting them in this scheme. - -Versioning the ``TypeDescription`` Message Type -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Given that the type description message interface has to be generic enough to support anything described in the ROS interfaces, there will be a need to add or remove fields over time in the type description message itself. -This should be done in such a way that the fields are tick-tocked and deprecated properly, possibly by having explicitly named versions of this interface, e.g. ``TypeDescriptionV1`` and ``TypeDescriptionV2`` and so on. - -Implementation in the ``rcl`` Layer -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The implementation of the type description distribution feature will be made in the ``rcl`` layer as opposed to the ``rmw`` layer to take advantage of the abstraction away from the middleware and to allow for compatibility with the client libraries. - -A hook will be added to ``rcl_node_init()`` to initialize the type description distribution service with the appropriate ``rcl_service_XXX()`` functions. -This hook should also keep a map of published and subscribed types which will be populated on each initialization of a publisher or subscription in the respective ``rcl_publisher_init()`` and ``rcl_subscription_init()`` function calls. -The passed ``rosidl_message_type_support_t`` in the init call can be used to obtain the relevant information, alongside any new methods added to support type version hashing. - -There will be an option to opt-out of creating this service, and a way to start and stop the service after node creation as well. - -.. TODO:: - - (wjwwood) A thought just occurred to me, which is that we could maybe have "lazy" Service Servers, which watch for any Service Clients to come up on their Service name and when they see it, they could start a Service Server. - The benefit of this is that when we're not using this feature (or other features like Parameter get/set Services), then we don't waste resources creating them, but if you know the node name and the corresponding Service name that it should have a server on, you could cause it to be activated by trying to use it. - The problem with this would be that you cannot browse the Service servers because they wouldn't advertise until requested, but for "well know service names" that might not be such a problem. - I guess the other problem would be there might be a slight delay the first time you go to use the service, but that might be a worthy trade-off too. - Run-Time Interface Reflection ----------------------------- Run-Time Interface Reflection allows access to the data in the fields of a serialized message buffer when given: - the serialized message as a ``rmw_serialized_message_t``, basically just an array of bytes as a ``rcutils_uint8_array_t``, -- the message's type description, e.g. received from the aforementioned "type description distribution" or from a bag file, and +- the message's type description, e.g. received from "type description distribution" or from a bag file, and - the serialization format, name and version, which was used to create the serialized message, e.g. ``XCDR2`` for ``Extended CDR encoding version 2`` From these inputs, we should be able to access the fields of the message from the serialized message buffer using some programmatic interface, which allows you to: @@ -531,7 +358,7 @@ From these inputs, we should be able to access the fields of the message from th serialization_format ─┘ └───────────────────────────────┘ Given that the scope of inputs and expected outputs is so limited, this feature should ideally be implemented as a separate package, e.g. ``rcl_serialization``, that can be called independently by any downstream packages that might need Run-Time Interface Reflection, e.g. introspection tools, rosbag transport, etc. -This feature can then be combined with the ability to detect type mismatches and obtain type descriptions in the previous two sections to facilitate communication between nodes of otherwise incompatible types. +This feature can then be combined with the ability to detect type mismatches and obtain type descriptions to facilitate communication between nodes of otherwise incompatible types. Additionally, it is important to note that this feature is distinct from ROS 2's existing "dynamic" type support (``rosidl_typesupport_introspection_c`` and ``rosidl_typesupport_introspection_cpp``). The ``rosidl_typesupport_introspection_c*`` generators generate code at compile time for known types that provides reflection for those types. @@ -612,21 +439,6 @@ This section will describe a vision of what is possible with some new tools, but That is to say, after this REP is accepted, tools described in this section may continue to evolve and improve, and even more tools not described here may be added to enable more things. As always, consider the latest documentation for the various pieces of the reference implementation to get the most accurate details. -Tools for Interacting with Type Version Hashes and Type Descriptions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The ros2 command line tools, and the APIs that support them, should be updated to provide access to the type version hash where ever the type name is currently available and the type version description on-demand as well. -For example: - -- ``ros2 interface`` should be extended with a way to calculate the version hash for a type -- ``ros2 topic info`` should include the type version hash used by each endpoint -- ``ros2 topic info --verbose`` should include the type description used by each endpoint -- a new command to compare the types used by two endpoints, e.g. ``ros2 topic compare `` -- ``ros2 service ...`` commands should also be extended in this way -- ``ros2 action ...`` commands should also be extended in this way - -Again this list should not be considered prescriptive or exhaustive, but gives an idea of what should change. - Notifying Users when Types Do Not Match ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -634,7 +446,7 @@ There should be a warning to users when two endpoints (publisher/subscription or This warning should be a console logging message, but it should also be possible for the user to get a callback in their own application when this occurs. How this should happen in the API is described in the Type Version Enforcement section, and it uses the "QoS Event" part of the API. -The warning should include important information, like the GUID of the endpoints involved, the type name(s) and type version hash(es), and of course the topic name. +The warning should include important information, like the GUID of the endpoints involved, the type name(s) and type hash(es), and of course the topic name. These pieces of information can then be used by the user, with the above mentioned changes to the command line tools, to investigate what is happening and why the types do not match. The warning may even suggest how to compare the types using the previously described ``ros2 topic compare`` command-line tool. @@ -647,7 +459,7 @@ This will help the user know if they need to write a transfer function or check This tool might look something like this: -- ``ros2 interface convertible --from-remote-node --to-local-type `` +- ``ros2 interface convertible --from-remote-node --to-local-type `` This tool would query the type description from a remote node, and compare it to the type description of the second type version found locally. If the two types can be converted without writing any code, then this tool would indicate that on the ``stdout`` and by return code. @@ -696,7 +508,7 @@ The transfer function API for C++ may look something like this: void my_transfer_function( const DynamicMessage & input, - const MessageInfo & input_info, // type name/version hash, etc. + const MessageInfo & input_info, // type name/hash, etc. DynamicMessage & output, const MessageInfo & output_info) { @@ -732,7 +544,7 @@ This tool would query the ``ament_index`` and list the available transfer functi The tool might look like this: - ``ros2 interface transfer_functions list`` -- ``ros2 interface transfer_functions info `` +- ``ros2 interface transfer_functions info `` The tool might also have options to filter based on the type name, package name providing the type, package name providing the transfer function (not necessarily the same as the package which provides the type itself), etc. @@ -827,15 +639,6 @@ Type Version Enforcement Alternatives ^^^^^^^^^^^^ -Use Type Hash from Middleware, e.g. from DDS-XTypes -""""""""""""""""""""""""""""""""""""""""""""""""""" - -Type hash can be obtained by the native middleware api. For example, with fastDDS, the type hash can be obtained with ``TypeIdentifier->equivalence_hash()`` during the ``on_type_discovery()`` callback. the rmw layer can choose to use the provided hash to impose the aforementioned type enforcement. - -.. TODO:: - - (wjwwood) this needs to be cleaned up - Evolving Message Definition with Extensible Type """""""""""""""""""""""""""""""""""""""""""""""" @@ -862,211 +665,17 @@ Furthermore, an initial test evolving messages with FastDDS, Cyclone, and Connex Handle Detection of Version Mismatch "Above" rmw Layer """""""""""""""""""""""""""""""""""""""""""""""""""""" -We can choose to utilize ``USER_DATA`` QoS to distribute the message version during discovery phase. -The message version for each participant will then be accessible across all available nodes. By getting the version hash through ``user_data`` via the ``rmw`` layer, similar type version matching can be detected. +We can choose to utilize ``USER_DATA`` QoS to distribute the type hash during the discovery phase. +The type hash for each participant will then be accessible across all available nodes. By getting the hash through ``user_data`` via the ``rmw`` layer, similar type version matching can be detected. Prevent Communication of Mismatched Versions "Above" rmw Layer """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" TODO -Type Description Distribution +Run-Time Interface Reflection ----------------------------- -Using a Single ROS Service per Node -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The node that is publishing the data must already have access to the correct type description, at the correct version, in order to publish it, and therefore it is natural to get the data from that node. -Similarly, a subscribing node also knows what type they are wanting to receive, both in name and version, and therefore it is again natural to get that information from the subscribing node. -The type description for a given type, at a given version, could have been retrieved from other places, e.g. a centralized database, but the other alternatives considered would have had to take care to ensure that it had the right version of the message, which is not the case for the node publishing the data. - -Because the interface for getting a type description is generic, it is not necessary to have this interface on a per entity, i.e. publisher, subscription, etc, basis, but instead to offer the ROS Service on a per node basis to reduce the number of ROS Services. -Therefore, the specification dictates that the type description is distributed by single ROS Service for each individual node. - -There were also multiple alternatives for how to get this information from each node, but the use of a single ROS Service was selected because the task of requesting the type description from a node is well suited to a request-response style ROS Service. -Some of the alternatives offered other benefits, but using a ROS Service introduced the fewest dependencies, feature-wise, while accomplishing the task. - -.. TODO:: - - cite the above, https://en.wikipedia.org/wiki/Request%E2%80%93response - -Combining the Raw and Parsed Type Description in the Service Response -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The contents of the "get type description" service response should include information that supports both aforementioned use cases (i.e. tools and Run-Time Interface Reflection). -These use cases have orthogonal interests, with the former requiring human-readable descriptions, and the latter preferring machine-readable descriptions. - -Furthermore, the type description should be useful even across middlewares and serialization libraries and that makes it especially important to send at least the original inputs to the "type support pipeline" (i.e. the process of taking user-defined types and generating all supporting code). -In this case, because the "type support pipeline" is a lossy process, there is a need to ensure that enough information is sent to completely reproduce the original definition of the type, and therefore it makes sense to just send the original ``idl`` or ``msg`` file. - -At the same time, it is useful to send information with the original description that makes it easier to process data at the receiving end, as it is often not trivial to get to the "parsed" version of the type description from the original text description. - -Finally, while there could be an argument for sending a losslessly compressed version of the message file, the expected low frequency of queries to the type description service incurs a negligible overhead that heavily reduces the benefit. - -Implementing in ``rcl`` versus ``rmw`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -While it is true that implementing the type description distribution on the ``rmw`` layer would allow for much lower level optimization, removing the layer of abstraction avoids having to implement this feature in each rmw implementation. - -Given that the potential gains from optimization will be small due to how infrequently the service is expected to be called, this added development overhead was determined to not be worth it. -Instead the design prefers to have a unified implementation of this feature in ``rcl`` so it is agnostic to any middleware implementations and client libraries. - -Nested ``TypeDescription`` Example -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The ``TypeDescription`` message type shown above also supports the complete description of a type that contains other types (a nested type), up to an arbitrary level of nesting. -Consider the following example: - -.. code:: - - # A.msg - B b - C c - - # B.msg - bool b_bool - - # C.msg - D d - - # D.msg - bool d_bool - -The corresponding ``TypeDescription`` for ``A.msg`` will be as follows, with the referenced type descriptions accessible as ``IndividualTypeDescription`` types in the ``referenced_type_descriptions`` field of ``A``: - -.. code:: - - # A: TypeDescription - type_description: A_IndividualTypeDescription - referenced_type_descriptions: [B_IndividualTypeDescription, - C_IndividualTypeDescription, - D_IndividualTypeDescription] - -Note that the type description for ``A`` itself is found in the ``type_description`` field instead of the ``referenced_type_descriptions`` field. -Additionally, in the case where a type description contains no referenced types (i.e., when it has no fields, or all of its fields are primitive types), the ``referenced_type_descriptions`` array will be empty. - -.. code:: - - # A: IndividualTypeDescription - type_name: "A" - fields: [A_b_Field, A_c_Field] - - # B: IndividualTypeDescription - type_name: "B" - fields: [B_b_bool_Field] - - # C: IndividualTypeDescription - type_name: "C" - fields: [C_d_Field] - - # D: IndividualTypeDescription - type_name: "D" - fields: [D_d_bool_Field] - -With the corresponding ``Field`` fields: - -.. code:: - - # A_b_Field - field_type: 0 - field_name: "b" - nested_type_name: "B" - - # A_c_Field - field_type: 0 - field_name: "c" - nested_type_name: "C" - - # B_b_bool_Field - field_type: 9 # Suppose 9 corresponds to a boolean field - field_name: "b_bool" - nested_type_name: "" # Empty if primitive type - - # C_d_Field - field_type: 0 - field_name: "d" - nested_type_name: "D" - - # D_d_bool_Field - field_type: 9 - field_name: "d" - nested_type_name: "" - -In order to handle the type of a nested type such as ``A``, the receiver can use the ``referenced_type_descriptions`` array as a lookup table keyed by the value of ``Field.nested_type_name`` or ``IndividualTypeDescription.type_name`` (which will be identical for a given type) to obtain the type information of a referenced type. -This type handling process can also support any recursive level of nesting (e.g. while handling A, C is encountered as a nested type, C can then be looked up using the top level ``referenced_type_descriptions`` array). - -Alternatives -^^^^^^^^^^^^ - -Other Providers of Type Description -""""""""""""""""""""""""""""""""""" - -Several other candidate strategies for distributing the type descriptions were considered but ultimately discarded for one or more reasons like: causing a strong dependency on a particular middleware or a third-party technology, difficulties with resolving the message type description locally, difficulties with finding the correct entity to query, or causing network throughput issues. - -These are some of the candidates that were considered, and the reasons for their rejection: - -- Store the type description as a ROS parameter - * Causes a mass of parameter event messages being sent at once on init, worsening the network initialization problem -- Store the type description on a centralized node per machine - * Helps reduce network bandwidth, but makes it non-trivial to find the correct centralized node to query, and introduces issues of resolving the local message package, such as when nodes are started from different sourced workspaces. -- Send type description alongside discovery with middlewares - * Works very well if supported, but is only supported by some DDS implementations (which support XTypes or some other way to attach discovery metadata), but causes a strong dependency on DDS. -- Send type description using a different network protocol - * Introduces additional third-party dependencies separate from ROS and the middleware. - -Alternative Type Description Contents and Format -"""""""""""""""""""""""""""""""""""""""""""""""" - -A combination of the original ``idl`` / ``msg`` file and any other information needed for serialization and deserialization being sent allows for one to cover the weaknesses of the other. -Specifically, given that certain use-cases (e.g., ``rosbag``) might encounter situations where consumers of a message are using a different middleware or serialization scheme the message was serialized with, it becomes extremely important to send enough information to both reconstruct the type support, and also allow the message fields to be accessed in a human readable fashion to aid in the writing of transfer functions. -As such, it is not a viable option to only send one or the other. - -Additionally, the option to add a configuration option to choose what contents to receive from the service server was disregarded due to how infrequently the type description query is expected to be called. - -As for the format of the type description, using the ROS interfaces to describe the type, as opposed to an alternative format like XML, JSON, or something like the TypeObject defined by DDS-XTypes, makes it easier to embed in the ROS Service response. -It also prevents unnecessary coupling with third-party specifications that could be subject to change and reduces the formats that need to be considered on the receiving end of the ROS Service call. - -Representing Fields as An Array of Field Types -"""""""""""""""""""""""""""""""""""""""""""""" - -The use of an array of ``Field`` messages was balanced against using two arrays in the ``IndividualTypeDescription`` type to describe the field types and field names instead, e.g.: - -.. code:: - - # Rejected IndividualTypeDescription Variants - - # String variant - string type_name - string field_types[] - string field_names[] - - # uint8_t Variant - string type_name - uint8_t field_types[] - string field_names[] - -The string variant was rejected because using strings to represent primitive types wastes space, and will lead to increased bandwidth usage during the discovery and type distribution process. -The uint8_t variant was rejected because uint8_t enums are insufficiently expressive to support nested message types. - -The use of the ``Field`` type, with a ``nested_type_name`` field that defaults to an empty string mitigates the space issue while allowing for support of nested message types. -Furthermore, it allows the fields to be described in a single array, which is easier to iterate through and also reduces the chances of any errors from mismatching the array lengths. - -Using an Array to Store Referenced Types -"""""""""""""""""""""""""""""""""""""""" - -Some alternatives to using an array of type descriptions to store referenced types in a nested type were considered, including: - -- Storing the referenced types inside the individual type descriptions and accessing them by traversing the type description tree recursively instead of using a lookup table. - - - Rejected because the IDL spec does not allow for a type description to store itself, and also because it could possibly introduce duplicate, redundant type descriptions in the tree, using up unnecessary space. - -- Sending referenced types in a separate service call or message. - - - Rejected because needing to collate all of the referenced types on the receiver end introduces additional implementation complexity, and also increases network bandwidth with all the separate calls that must be made. - -Run-Time Interface Reflect --------------------------- - TODO Plugin Matching and Loading API Example @@ -1221,23 +830,7 @@ Feature Progress Supporting Feature Development: -- TypeDescription Message: - - - [x] Define and place the TypeDescription.msg Message type in a package - -- Enforced Type Versioning: - - - [ ] Create library to calculate version hash from message files - - [ ] Create library to calculate version hash from service files - - [ ] Create library to calculate version hash from action files - - [ ] Create command-line tool to show the version hash of an interface - - - [ ] Add topic type version hash to the "graph" API - - [ ] Implement delivery of type version hash during discovery: - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds +- Type Version Enforcement - [ ] Add rmw API for notifying user when a topic endpoint was not matched due to type mismatch @@ -1259,11 +852,6 @@ Supporting Feature Development: - [ ] rmw_cyclonedds_cpp - [ ] rmw_connextdds -- Type Description Distribution: - - - [ ] Define and place the GetTypeDescription.srv Service type in a package - - [ ] Implement the "get type description" service in the rcl layer - - Run-time Interface Reflection: - [ ] Prototype "description of type" -> "accessing data fields from buffer" for each rmw vendor: @@ -1309,9 +897,6 @@ Supporting Feature Development: Tooling Development: -- [ ] Command-line tools for interacting with type version hashes -- [ ] Add a default warning to users when a topic has mismatched types (by type or version) - - [ ] Command-line tool for determining if two versions of a type are convertible - [ ] CMake and C++ APIs for writing transfer functions, including usage documentation @@ -1322,10 +907,8 @@ Tooling Development: - [ ] Integration with rosbag2 - - [ ] Record the type version hash, TypeDescription, and other metadata into bags - - [ ] Use recorded metadata to create subscriptions without needing the type to be built locally + - [ ] Use recorded type description metadata to create subscriptions without needing the type to be built locally - [ ] Use recorded metadata to create publishers from descriptions in bag - - [ ] Provide a method to playback/update old bag files that do not have type descriptions stored References diff --git a/rep-2016.rst b/rep-2016.rst index db617c16..290dbd85 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -43,7 +43,7 @@ Specification In order to communicate and compare types, a Type Description data structure is defined to contain type information in a common format. To detect and enforce type version mismatches, and communicate information about type descriptions compactly, a way to uniquely identify types is required. For this purpose a hash of the Type Description, or Type Hash, is used. -The Type Hash specification, covered below, defines a common algorithm that guarantees cross-implementation consistency. +The Type Hash specification, covered below, defines a Type Description serialization and hashing algorithm that guarantees cross-implementation consistency. Type Description @@ -117,14 +117,188 @@ The type version hash must be considered in context, with the serialization form Type Hash Discovery ------------------- -Hashes are intended to be communicated such that they are available at the time of discovering a topic, before creating subscriptions. -The hash will be available in the ``rmw_topic_endpoint_info_t`` data structure from discovery. +Hashes are intended to be communicated such that they are available at the time of discovering a topic, before attempting to subscribe. +It can also give subscription-side tooling the opportunity to obtain the type description for the given hash. -For DDS implementations of the RMW API, it is recommended but not required to use the USER_DATA QoS policy to send this information. +Accessing the Type Hash +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -This discovery-time hash availability allows for validation of type mismatch before ever requesting a subscription. +For debugging and introspection, the type version hash will be accessible via the ROS graph API, by extending the ``rmw_topic_endpoint_info_t`` struct, and related types and functions, to include the type version hash, ``topic_type_hash``. +It should be alongside the ``topic_type`` string in that struct, which remains unchanged. -It can also give subscription-side tooling the opportunity to obtain the type description for the given hash. +This information will be transmitted as part of the discovery process. + +The ``topic_type_hash`` field can use RIHS version 0 (``VERSION_UNSET``), in order to support interaction with older versions of ROS where this feature was not yet implemented, but it should be provided if at all possible. + +Recommended implementation strategy for DDS-based RMW implementations +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For DDS implementations of the RMW API, it is recommended but not required to use the ``USER_DATA`` QoS policy to send the Type Hash information. + +While ``USER_DATA`` accepts arbitrary binary data, ROS 2 implementations so far use a semicolon-delimited list of key-value pairs, like the following: ``key1=value1;key2=value``. +Keys in this scheme are purely alphanumeric, so the recommended key is ``typehash``. + +.. code:: + typehash=RIHS01_XXXXXXXX; + +While this usage can be implementation-specific, consistency will allow for communication across DDS-RMW implementations. + + +Type Description Distribution +----------------------------- + +For some use cases the type version hash is insufficient and instead the full type description is required. + +One of those use cases, which is also described in this REP, is "Run-Time Interface Reflection", which is the ability to introspect the contents of a message at runtime when the description for that message, or that version of that message, was unavailable at compile time. +In this use case the type description is used to interpret the serialized data dynamically. + +Another use case, which is not covered in this REP, is using the type description in tooling to either display the type description to the user or to include it in recordings. + +In either case, where the type description comes from doesn't really matter, and so, for example, it could be looked up on the local filesystem or read from a rosbag file. +However, in practice, the correct type description may not be found locally, especially in cases where you have different versions of messages in the same system, e.g.: + +- because it's on another computer, or +- because it is from a different distribution of ROS, or +- because it was built in a different workspace, or +- because the application has not been restarted since recompiling a change to the type being used + +In any case, it is useful to have a mechanism to convey the type descriptions from the source of the data to other nodes, which we describe here as "type description distribution". + +Furthermore, this feature should be agnostic to the underlying middleware and serialization library, as two endpoints may not have the same rmw implementation, or the data may have been serialized to a different format in the case of playback of a recording. + +Sending the Type Description +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Type descriptions will be provided by a ROS Service called ``~/_get_type_description``, which will be offered by each node. +There will be a single ROS Service per node, regardless of the number of publishers or subscriptions on that node. + +This service may be optional, for example being enabled or disabled when creating the ROS Node. + +.. TODO:: + + How can we detect when a remote node is not offering this service? + It's difficult to differentiate between "the Service has not been created yet, but will be" and "the Service will never be created". + Should we use a ROS Parameter to indicate this? + But then what if remote access to Parameters (another optional Service) is disabled? + Perhaps we need a "services offered" list which is part of the Node metadata, which is sent for each node in the rmw implementation, but that's out of scope for this REP. + +A service request to this ROS Service will comprise of the type name and the type version hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described in previous sections. +The ROS Service server will respond with the type description and any necessary metadata needed to do Run-Time Interface Reflection. +This service is not expected to be called frequently, and is likely to only occur when new topic or service endpoints are created, and even then, only if the endpoint type hashes do not match. + +.. TODO:: + + Should each endpoint be asked about their type/version pair or should we assume that the type/version pair guarantees a unique type description and therefore reuse past queries? + (wjwwood) I am leaning towards assuming the type name and type version hash combo as being a unique identifier. + +Type Description Contents and Format +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The response sent by the ROS Service server will contain a combination of the original ``idl`` or ``msg`` file's content, as well as any necessary information to serialize and deserialize the raw message buffers sent on the topic. +The response will contain a version of the description that contains comments from the original type description, as those might be relevant to interpreting the semantic meaning of the message fields. + +Additionally, the response could include the serialization library used, its version, or any other helpful information from the original producer of the data. + +.. TODO:: + + What happens if the message consumer doesn't have access to the serialization library stated in the meta-type? + (wjwwood) It depends on what you're doing with the response. If you are going to subscribe to a remote endpoint, then I think we just refuse to create the subscription, as communication will not work, and there's a basic assumption that if you're going to communicate with an endpoint then you are using the same or compatible rmw implementations, which includes the serialization technology. The purpose of this section and ROS Service is to provide the information, not to ensure communication can happen. + +The ROS 2 message that defines the type description must be able to describe any message type, including itself, and since it is describing the message format, it should work independently from any serialization technologies used. +This "meta-type description" message would then be used to communicate the structure of the type as part of the "get type description" service response. +The final form of these interfaces should be found in the reference implementation, but such a Service interface might look like this: + +.. code:: + + string type_name + string type_version_hash + --- + # True if the type description information is available and populated in the response + bool successful + # Empty if 'successful' was true, otherwise contains details on why it failed + string failure_reason + + # The idl or msg file name + string type_description_raw_file_name + # The idl or msg file, with comments and whitespace + # The file extension and/or the contents can be used to determine the format + string type_description_raw + # The parsed type description which can be used programmatically + TypeDescription type_description + + # Key-value pairs of extra information. + string[] extra_information_keys + string[] extra_information_values + +.. TODO:: + + (wjwwood) I propose we use key-value string pairs for the extra information, but I am hoping for discussion on this point. + This type is sensitive to changes, since changes to it will be hard to roll out, and may need manual versioning to handle. + We could also consider bounding the strings and sequences of strings, so the message could be "bounded", which is nice for safety and embedded systems. + +Again, the final form of these interfaces should be referenced from the reference implementation, but the ``TypeDescription`` message type might look something like this: + +.. code:: + + IndividualTypeDescription type_description + IndividualTypeDescription[] referenced_type_descriptions + +And the ``IndividualTypeDescription`` type: + +.. code:: + + string type_name + Field[] fields + +And the ``Field`` type: + +.. code:: + + FIELD_TYPE_NESTED_TYPE = 0 + FIELD_TYPE_INT = 1 + FIELD_TYPE_DOUBLE = 2 + # ... and so on + FIELD_TYPE_INT_ARRAY = ... + FIELD_TYPE_INT_BOUNDED_SEQUENCE = ... + FIELD_TYPE_INT_SEQUENCE = ... + # ... and so on + + string field_name + uint8_t field_type + uint64_t field_array_size # Only for Arrays and Bounded Sequences + string nested_type_name # Only for FIELD_TYPE_NESTED_TYPE + +These examples of the interfaces just give an idea of the structure but perhaps do not yet consider some other complications like field annotations or other as yet unconsidered features we want to support. + +.. TODO:: + + (wjwwood) Add text about how to handle Service types, which are formed as a Request and a Response part, each of which is kind of like a Message. + We could just treat the Request and Response separately, or we could extend this scheme to include explicit support for Services. + Since Actions are composed of Topics and Services, it is less so impacted, but we could similarly consider officially supporting them in this scheme. + +Versioning the ``TypeDescription`` Message Type +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given that the type description message interface has to be generic enough to support anything described in the ROS interfaces, there will be a need to add or remove fields over time in the type description message itself. +This should be done in such a way that the fields are tick-tocked and deprecated properly, possibly by having explicitly named versions of this interface, e.g. ``TypeDescriptionV1`` and ``TypeDescriptionV2`` and so on. + +Implementation in the ``rcl`` Layer +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The implementation of the type description distribution feature will be made in the ``rcl`` layer as opposed to the ``rmw`` layer to take advantage of the abstraction away from the middleware and to allow for compatibility with the client libraries. + +A hook will be added to ``rcl_node_init()`` to initialize the type description distribution service with the appropriate ``rcl_service_XXX()`` functions. +This hook should also keep a map of published and subscribed types which will be populated on each initialization of a publisher or subscription in the respective ``rcl_publisher_init()`` and ``rcl_subscription_init()`` function calls. +The passed ``rosidl_message_type_support_t`` in the init call can be used to obtain the relevant information, alongside any new methods added to support type version hashing. + +There will be an option to opt-out of creating this service, and a way to start and stop the service after node creation as well. + +.. TODO:: + + (wjwwood) A thought just occurred to me, which is that we could maybe have "lazy" Service Servers, which watch for any Service Clients to come up on their Service name and when they see it, they could start a Service Server. + The benefit of this is that when we're not using this feature (or other features like Parameter get/set Services), then we don't waste resources creating them, but if you know the node name and the corresponding Service name that it should have a server on, you could cause it to be activated by trying to use it. + The problem with this would be that you cannot browse the Service servers because they wouldn't advertise until requested, but for "well know service names" that might not be such a problem. + I guess the other problem would be there might be a slight delay the first time you go to use the service, but that might be a worthy trade-off too. Type Description Distribution @@ -134,12 +308,268 @@ Type Description Distribution The service must be optional, but it will be a detail decided by client libraries whether it is enabled or disabled by default. -Supporting Tooling -================== +Tooling +======= + +The ros2 command line tools, and the APIs that support them, should be updated to provide access to the type version hash where ever the type name is currently available and the type version description on-demand as well. +For example: - ros2 topic info -v (done - TODO more info) - ros2 interface show (TODO implement) + +- ``ros2 interface`` should be extended with a way to calculate the version hash for a type +- ``ros2 topic info`` should include the type version hash used by each endpoint +- ``ros2 topic info --verbose`` should include the type description used by each endpoint +- a new command to compare the types used by two endpoints, e.g. ``ros2 topic compare `` +- ``ros2 service ...`` commands should also be extended in this way +- ``ros2 action ...`` commands should also be extended in this way + +Again this list should not be considered prescriptive or exhaustive, but gives an idea of what should change. + +Backwards Compatibility +======================= + +TODO + +Rationale +========= + +The above document lays out the specific recommendations of this REP, as reached after discussion and iteration. +This section layse out further reasoning for why certain conclusions were reached, including alternatives that were considered. + +Type Hashing +------------ +TODO - RIHS hashing algorithms (sha1, md5) +TODO - RIHS string representations (eg base64 (simplicity in conversion, human readability), pure binary (null terminators)) + +Type Description Distribution +----------------------------- + +Using a Single ROS Service per Node +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The node that is publishing the data must already have access to the correct type description, at the correct version, in order to publish it, and therefore it is natural to get the data from that node. +Similarly, a subscribing node also knows what type they are wanting to receive, both in name and version, and therefore it is again natural to get that information from the subscribing node. +The type description for a given type, at a given version, could have been retrieved from other places, e.g. a centralized database, but the other alternatives considered would have had to take care to ensure that it had the right version of the message, which is not the case for the node publishing the data. + +Because the interface for getting a type description is generic, it is not necessary to have this interface on a per entity, i.e. publisher, subscription, etc, basis, but instead to offer the ROS Service on a per node basis to reduce the number of ROS Services. +Therefore, the specification dictates that the type description is distributed by single ROS Service for each individual node. + +There were also multiple alternatives for how to get this information from each node, but the use of a single ROS Service was selected because the task of requesting the type description from a node is well suited to a request-response style ROS Service. +Some of the alternatives offered other benefits, but using a ROS Service introduced the fewest dependencies, feature-wise, while accomplishing the task. + +.. TODO:: + + cite the above, https://en.wikipedia.org/wiki/Request%E2%80%93response + +Combining the Raw and Parsed Type Description in the Service Response +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The contents of the "get type description" service response should include information that supports both aforementioned use cases (i.e. tools and Run-Time Interface Reflection). +These use cases have orthogonal interests, with the former requiring human-readable descriptions, and the latter preferring machine-readable descriptions. + +Furthermore, the type description should be useful even across middlewares and serialization libraries and that makes it especially important to send at least the original inputs to the "type support pipeline" (i.e. the process of taking user-defined types and generating all supporting code). +In this case, because the "type support pipeline" is a lossy process, there is a need to ensure that enough information is sent to completely reproduce the original definition of the type, and therefore it makes sense to just send the original ``idl`` or ``msg`` file. + +At the same time, it is useful to send information with the original description that makes it easier to process data at the receiving end, as it is often not trivial to get to the "parsed" version of the type description from the original text description. + +Finally, while there could be an argument for sending a losslessly compressed version of the message file, the expected low frequency of queries to the type description service incurs a negligible overhead that heavily reduces the benefit. + +Implementing in ``rcl`` versus ``rmw`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +While it is true that implementing the type description distribution on the ``rmw`` layer would allow for much lower level optimization, removing the layer of abstraction avoids having to implement this feature in each rmw implementation. + +Given that the potential gains from optimization will be small due to how infrequently the service is expected to be called, this added development overhead was determined to not be worth it. +Instead the design prefers to have a unified implementation of this feature in ``rcl`` so it is agnostic to any middleware implementations and client libraries. + +Nested ``TypeDescription`` Example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The ``TypeDescription`` message type shown above also supports the complete description of a type that contains other types (a nested type), up to an arbitrary level of nesting. +Consider the following example: + +.. code:: + + # A.msg + B b + C c + + # B.msg + bool b_bool + + # C.msg + D d + + # D.msg + bool d_bool + +The corresponding ``TypeDescription`` for ``A.msg`` will be as follows, with the referenced type descriptions accessible as ``IndividualTypeDescription`` types in the ``referenced_type_descriptions`` field of ``A``: + +.. code:: + + # A: TypeDescription + type_description: A_IndividualTypeDescription + referenced_type_descriptions: [B_IndividualTypeDescription, + C_IndividualTypeDescription, + D_IndividualTypeDescription] + +Note that the type description for ``A`` itself is found in the ``type_description`` field instead of the ``referenced_type_descriptions`` field. +Additionally, in the case where a type description contains no referenced types (i.e., when it has no fields, or all of its fields are primitive types), the ``referenced_type_descriptions`` array will be empty. + +.. code:: + + # A: IndividualTypeDescription + type_name: "A" + fields: [A_b_Field, A_c_Field] + + # B: IndividualTypeDescription + type_name: "B" + fields: [B_b_bool_Field] + + # C: IndividualTypeDescription + type_name: "C" + fields: [C_d_Field] + + # D: IndividualTypeDescription + type_name: "D" + fields: [D_d_bool_Field] + +With the corresponding ``Field`` fields: + +.. code:: + + # A_b_Field + field_type: 0 + field_name: "b" + nested_type_name: "B" + + # A_c_Field + field_type: 0 + field_name: "c" + nested_type_name: "C" + + # B_b_bool_Field + field_type: 9 # Suppose 9 corresponds to a boolean field + field_name: "b_bool" + nested_type_name: "" # Empty if primitive type + + # C_d_Field + field_type: 0 + field_name: "d" + nested_type_name: "D" + + # D_d_bool_Field + field_type: 9 + field_name: "d" + nested_type_name: "" + +In order to handle the type of a nested type such as ``A``, the receiver can use the ``referenced_type_descriptions`` array as a lookup table keyed by the value of ``Field.nested_type_name`` or ``IndividualTypeDescription.type_name`` (which will be identical for a given type) to obtain the type information of a referenced type. +This type handling process can also support any recursive level of nesting (e.g. while handling A, C is encountered as a nested type, C can then be looked up using the top level ``referenced_type_descriptions`` array). + +Alternatives +^^^^^^^^^^^^ + +Other Providers of Type Description +""""""""""""""""""""""""""""""""""" + +Several other candidate strategies for distributing the type descriptions were considered but ultimately discarded for one or more reasons like: causing a strong dependency on a particular middleware or a third-party technology, difficulties with resolving the message type description locally, difficulties with finding the correct entity to query, or causing network throughput issues. + +These are some of the candidates that were considered, and the reasons for their rejection: + +- Store the type description as a ROS parameter + * Causes a mass of parameter event messages being sent at once on init, worsening the network initialization problem +- Store the type description on a centralized node per machine + * Helps reduce network bandwidth, but makes it non-trivial to find the correct centralized node to query, and introduces issues of resolving the local message package, such as when nodes are started from different sourced workspaces. +- Send type description alongside discovery with middlewares + * Works very well if supported, but is only supported by some DDS implementations (which support XTypes or some other way to attach discovery metadata), but causes a strong dependency on DDS. +- Send type description using a different network protocol + * Introduces additional third-party dependencies separate from ROS and the middleware. + +Alternative Type Description Contents and Format +"""""""""""""""""""""""""""""""""""""""""""""""" + +A combination of the original ``idl`` / ``msg`` file and any other information needed for serialization and deserialization being sent allows for one to cover the weaknesses of the other. +Specifically, given that certain use-cases (e.g., ``rosbag``) might encounter situations where consumers of a message are using a different middleware or serialization scheme the message was serialized with, it becomes extremely important to send enough information to both reconstruct the type support, and also allow the message fields to be accessed in a human readable fashion to aid in the writing of transfer functions. +As such, it is not a viable option to only send one or the other. + +Additionally, the option to add a configuration option to choose what contents to receive from the service server was disregarded due to how infrequently the type description query is expected to be called. + +As for the format of the type description, using the ROS interfaces to describe the type, as opposed to an alternative format like XML, JSON, or something like the TypeObject defined by DDS-XTypes, makes it easier to embed in the ROS Service response. +It also prevents unnecessary coupling with third-party specifications that could be subject to change and reduces the formats that need to be considered on the receiving end of the ROS Service call. + +Representing Fields as An Array of Field Types +"""""""""""""""""""""""""""""""""""""""""""""" + +The use of an array of ``Field`` messages was balanced against using two arrays in the ``IndividualTypeDescription`` type to describe the field types and field names instead, e.g.: + +.. code:: + + # Rejected IndividualTypeDescription Variants + + # String variant + string type_name + string field_types[] + string field_names[] + + # uint8_t Variant + string type_name + uint8_t field_types[] + string field_names[] + +The string variant was rejected because using strings to represent primitive types wastes space, and will lead to increased bandwidth usage during the discovery and type distribution process. +The uint8_t variant was rejected because uint8_t enums are insufficiently expressive to support nested message types. + +The use of the ``Field`` type, with a ``nested_type_name`` field that defaults to an empty string mitigates the space issue while allowing for support of nested message types. +Furthermore, it allows the fields to be described in a single array, which is easier to iterate through and also reduces the chances of any errors from mismatching the array lengths. + +Using an Array to Store Referenced Types +"""""""""""""""""""""""""""""""""""""""" + +Some alternatives to using an array of type descriptions to store referenced types in a nested type were considered, including: + +- Storing the referenced types inside the individual type descriptions and accessing them by traversing the type description tree recursively instead of using a lookup table. + + - Rejected because the IDL spec does not allow for a type description to store itself, and also because it could possibly introduce duplicate, redundant type descriptions in the tree, using up unnecessary space. + +- Sending referenced types in a separate service call or message. + + - Rejected because needing to collate all of the referenced types on the receiver end introduces additional implementation complexity, and also increases network bandwidth with all the separate calls that must be made. + + +Development Progress +==================== + +TODO - remove this section for final merge + +- Type Descriptions: + + - [x] Define and place the TypeDescription.msg Message type in a package + - [ ] Create library to calculate version hash from parsed types + +- Type Description Distribution: + + - [ ] Add topic type hash to the "graph" API + - [ ] Define and place the GetTypeDescription.srv Service type in a package + - [ ] Implement the "get type description" service in the rcl layer + - [ ] Implement delivery of type version hash during discovery: + + - [ ] rmw_fastrtps_cpp + - [ ] rmw_cyclonedds_cpp + - [ ] rmw_connextdds + +- Tooling + + - [ ] Create command-line tool to show the version hash of an interface + - [ ] Command-line tools for interacting with type version hashes + - [ ] Add a default warning to users when a topic has mismatched types (by type or version) + + - [ ] Integration with rosbag2 + + - [ ] Record the Type Hash and Type Description into bags + - [ ] Provide a method to playback/update old bag files that do not have type information stored + References ========== From cb0e43dc2d4bdce92bee3b171b8ff34cddc6c5bf Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Mon, 3 Jul 2023 23:16:11 -0700 Subject: [PATCH 06/10] Remove 2011 from this branch Signed-off-by: Emerson Knapp --- rep-2011.rst | 946 --------------------------------------------------- 1 file changed, 946 deletions(-) delete mode 100644 rep-2011.rst diff --git a/rep-2011.rst b/rep-2011.rst deleted file mode 100644 index 04d0a3bb..00000000 --- a/rep-2011.rst +++ /dev/null @@ -1,946 +0,0 @@ -REP: 2011 -Title: Evolving Message Types, and Other ROS Interface Types, Over Time -Author: William Woodall , Brandon Ong , You Liang Tan , Kyle Marcey -Status: Draft -Type: Standards Track -Content-Type: text/x-rst -Created: 30-Nov-2021 -Post-History: - -Abstract -======== - -This REP proposes patterns and approaches for evolving message, service, and action ROS interface types over time. - -The proposed patterns use the existing default serialization technology, i.e. CDR, in combination with new tooling to detect when types have changed and new tooling to convert between versions. -However, it should be possible to use features provided by different, perhaps future, serialization technologies in addition to the approaches proposed in this REP. - -Specifically, this REP proposes that we provide tooling to convert from old versions of types to newer versions of types on demand, using user-defined transfer functions. -This approach is a good way to achieve backwards compatibility in messages over long periods of time and in a variety of scenarios, e.g. over the wire, converting bag files, or in specialized tools. - -This approach does not rely on specific features of the serialization technology and instead relies on the ability to communicate the full type descriptions on the wire, beyond their name and version, and use that description to introspect the values of it at runtime using reflection. -That feature set is described in detail in REP-2016\ [#rep2016]_. - -This approach can be used in conjunction with serialization technology specific features like optional fields, inheritance, etc., but it works even with the simplest serialization technologies. -This is true so long as we have the ability to introspect the messages at runtime without prior knowledge of the type description, which is a feature we also need for generic introspection tools like ``rosbag2`` and ``ros2 topic echo``. - -Also covered in this REP are recommended changes to the middleware API in ROS 2, as well as additional infrastructure, to support runtime introspection of messages. -It is very similar to, though slightly narrower and more generic than, the DDS-XTypes specification\ [1]_, and an interface similar to DDS-XTypes will likely be adopted in the ROS 2 middleware API, without explicitly relying on it. - -Alternatives are also discussed in more detail. - -Motivation -========== - -Evolving types over time is a necessary part of developing a new system using ROS 2, as well as a necessary part of evolving ROS 2 itself over time. -As needs change, types often need to change as well to accommodate new use cases by adding or removing fields, changing the type of fields, or splitting/combining messages across different topics. - -To understand this need, it's useful to consider some examples, like this message definition: - -.. code:: - - # Temperature.msg - uint64 timestamp - int32 temperature - -There are many potential issues with this message type, but it isn't uncommon for messages like this to be created in the process of taking an idea from a prototype into a product, and for the sake of this example, let's say that after using this message for a while the developers wanted to change the temperature field's type from int32 to float64. - -There are a few ways the developers could approach this, e.g. if the serialization technology allowed for optional fields, they may add a new optional field like this (pseudo code): - -.. code:: - - # Temperature.msg - uint64 timestamp - int32 temperature - optional float64 temperature_float - -Updated code could use it like this (pseudo code): - -.. code:: - - void on_temperature(const Temperature & temp) { - // ... - double actual_temp = 0.0; - if (temp.temperature_float_is_set()) { - actual_temp = temp.temperature_float(); - } else { - actual_temp = static_cast(temp.temperature()); - } - } - -This is not uncommon to see in projects, and it has the advantage that old data, whether it is from a program running on an older machine or is from old recorded data, can be interpreted by newer code without additional machinery or features beyond the optional field type. -Older publishing code can also use the new definition without being updated to use the new ``temperature_float`` field, which is very forgiving for developers in a way. -However, as projects grow and the number of incidents like the one described in the example above increase, this kind of approach can generate a lot of complexity to sending and receiving code. - -You can imagine other serialization features, like inheritance or implicit type conversion, could be used to address this desired change, but in each case it relies on a feature of the serialization technology which may or may not be available in all middleware implementations for ROS 2. - -Each of these approaches also requires a more sophisticated message API for the users than what is currently provided by ROS 2. -At the moment the user can access all fields of the message directly, i.e. "member based access" vs "method based access", and that would need to change in some way to use some of these features. - -A different approach would be to simply update the type as follows: - -.. code:: - - # Temperature.msg - uint64 timestamp - float64 temperature - -And also update any code publishing or subscribing to this type at the same time. -This is typically what happens right now in ROS 2, and also what happened historically in ROS 1. -This approach is simple and requires no additional serialization features, but obviously doesn't do anything on its own to help developers evolve their system while maintaining support for already deployed code and utilizing existing recorded data. - -However, this REP proposes that with additional tooling these cases can be handled without the code of the application knowing about it. -Consider again the above example, but this time the update to the message was done without backwards compatibility in the message definition. -This means that publishing old recorded data, for example, will not work with new code that subscribes to the topic, because the new subscription expects the new version of the message. -For the purpose of this example, let's call the original message type ``Temperature`` and the one using ``float64`` we'll call ``Temperature'``. -So, if you have rosbag2 publishing ``Temperature`` messages and a program consuming ``Temperature'`` messages they will not communicate, unless you have an intermediate program doing the translation. - -.. code:: - - ┌─────────┐ publishes ┌──────────────┐ publishes ┌─────┐ - │ rosbag2 ├───────────────►│transfer func.├──────────────►│ App │ - └─────────┘ Temperature └──────────────┘ Temperature' └─────┘ - -The "transfer function" can be user-defined, or for simple changes, like changing the field type to a compatible type, it can be done automatically. -We already do something like this for the "ROS 1 to ROS 2 bridge" in order to handle changes between message types in ROS 1 and ROS 2, and something like this was also done for rosbags in ROS 1 with the "bag migration rules" feature. - -.. TODO:: - - cite the ros1_bridge rules and the rosbag migration rules - -The transfer functions require the ability to have a single application which can interact with both the old and the new versions of a message at the same time. -Making this possible requires several new technical features for ROS 2, and some new infrastructure and tooling -However, by keeping the conversion logic contained in these transfer functions, it has the advantage of keeping both the publishing and subscribing code simple. -That is to say, it keeps both the publishing and subscribing code agnostic to the fact that there are other versions of the message, and it keeps the message type from being cluttered with vestigial fields, e.g. having both a ``temperature`` and ``temperature_float`` in the same message. - -As stated before, problems created by changing these ROS interfaces can usually be solved by more than one way, either using some feature like optional fields or by just breaking compatibility directly. -However, the strategy used usually depends on the features that the serialization technology being used offers. -ROS 2 has special considerations on this topic because it can support different serialization technologies, and though CDR is the default and most common right now, others could be used in the future. -Therefore, it is neither desirable to depend on features of a specific technology, nor is it desirable suggest patterns that rely on features that only some serialization technologies provide. -In either case, that would tie ROS 2 to specific serialization technologies, and that should be avoided if possible. - -That being said, this proposal will require some specific features from the middleware and serialization technology, but the goal is to choose approaches which give ROS 2 the broadest support across middleware implementations, ideally while not limiting users from using specific features of the underlying technology when that suits them. - -With those examples and design constraints as motivation, this REP makes a proposal on how to handle evolving message types in the following Specification section, as well as rationales and considered alternatives in the Rationale section and its sub-sections. - -Terminology -=========== - -TODO - -- "type version" - -Specification -============= - -The proposal is to provide tooling to help users: - -- identify when messages have changed -- configure their system to convert between versions of messages on demand -- write the code needed to convert between types when the conversion is not trivial - -In order to do this, this proposal describes how ROS 2 can be changed to support these tools by: - -- Using REP-2016 Type Description features to - - - detect type mismatches - - warn users when type enforcement is preventing two endpoints from communicating - - access type descriptions from nodes remotely - -- making it possible to publish and subscribe to topics using just the type description - - - even when the type was not available at compile time - - and introspecting the values from a serialized message using just the type description - -This Specification section covers the conceptual overview in more detail, then describes each of the technical changes needed in ROS 2, and then finishes by describing the new tooling that will help users in the aforementioned ways. - -Conceptual Overview -------------------- - -The REP-2016 Type Hash for an interface is used by ROS 2 to determine if the same type name with different type versions are being used on the same topic, so that a warning may be logged that endpoints that do not match may not communicate. - -.. note:: - - An exception to this rule is that if the underlying middleware has more sophisticated rules for matching types, for example the type has been extended with an optional field, then they may still match - In that case, ROS 2 will defer to the middleware and not produce warnings when the type hashes do not match. - Instead, ROS 2 will rely on the middleware to notify it when two endpoints do not match based on their types not being compatible, so that a warning can be produced. - -When a mismatch is detected, the user can use user-defined or automatically generated generic "transfer functions" to convert between versions of the type until it is in the type version they wish to send or receive. -They can use a tool that will look at a catalogue of available transfer functions to find a single transfer function, or a set of transfer functions, to get from the current type version to the desired type version. - -.. code:: - - ┌───────────────────────┐ - ┌───────────────┐ │ Implicit Conversion │ ┌───────────────┐ - │Message@current├────►│ by ├───►│Message@desired│ - └───────────────┘ │ Generic Transfer Func.│ └───────────────┘ - └───────────────────────┘ - - or - - ┌───────────────────────┐ - ┌───────────────┐ │ Implicit Conversion │ ┌────────────────────┐ - │Message@current├────►│ by ├───►│Message@intermediate│ - └───────────────┘ │ Generic Transfer Func.│ └────────────────────┘ - └───────────────────────┘ - - or - - ┌───────────────────────┐ - ┌───────────────┐ │ User-defined transfer │ ┌───────────────┐ - │Message@current├────►│ function from current ├───...───►│Message@desired│ - └───────────────┘ │to desired/intermediate│ ▲ └───────────────┘ - └───────────────────────┘ │ - │ - possibly other transfer functions - -The tool will start with the current type version and see if it can be automatically converted to the desired type version, or if it is accepted as an input to any user-defined transfer functions or if it can be automatically converted into one of the input type versions for the transfer functions. -It will continue to do this until it reaches the desired type version or it fails to find a path from the current to the desired type version. - -.. code:: - - ┌──────────────────────┐ /topic ┌─────────────────────────┐ - │Publisher├────────X────────►│Subscription│ - └──────────────────────┘ └─────────────────────────┘ - │ │ │ - remap publisher │ │ │ and add transfer function - ▼ ▼ ▼ - ┌──────────────────────┐ ┌─────────────────────────┐ - │Publisher│ │Subscription│ - └─┬────────────────────┘ └─────────────────────────┘ - │ ▲ - │ ┌─────────────────────────────────┐ │ - ✓ /topic/ABC │ Transfer Functions for ABC->XYZ │ /topic ✓ - │ │ │ │ - │ ┌──────────┴──────────────┐ ┌──────────────┴───────┐ │ - └─►│Subscription│ │Publisher├──┘ - └──────────┬──────────────┘ └──────────────┬───────┘ - │ │ - └─────────────────────────────────┘ - -Once the set of necessary transfer functions has been identified, the ROS graph can be changed to have one side of the topic be remapped onto a new topic name which indicates it is of a different version than what is desired, and then the transfer function can be run as a component node which subscribes to one version of the message, performs the conversion using the chain of transfer functions, then publishes the other version of the message. -Tools will assist the user in making these remappings and running the necessary component nodes with the appropriate configurations, either from their launch file or from the command line. - -.. TODO:: - - discuss the implications for large messages and the possibility of having the transfer functions be colocated with either the publisher or subscription more directly than with component nodes and remapping. - -Once the mismatched messages are flowing through the transfer functions, communication should be possible and neither the publishing side nor the subscribing side have any specific knowledge of the conversions taking place or that any conversions are necessary. - -.. TODO:: - - Extend conceptual overview to describe how this will work with Services and Actions. - Services, since they are not sensitive to the many-to-many (many publisher) issue, unlike Topics, and because they do not have as many QoS settings that apply to them, they can probably have transfer functions that are plugins, rather than separate component nodes that repeat the service call, like the ros1_bridge. - Actions will be a combination of topics and services, but will have other considerations in the tooling. - -In order to support this vision, these features will have been added into ROS 2 (which were also mentioned in the introduction): - -- enforcing type compatibility by version -- making it possible to publish and subscribe to topics using just the type description - -These features are described in the following sections. - -Type Version Enforcement ------------------------- - -Enforcing Type Version -^^^^^^^^^^^^^^^^^^^^^^ - -The Type Hash may be used as an additional constraint to determine if two endpoints (publishers and subscriptions) on a topic should communicate. -Again, whether or not it is used will depend on the underlying middleware and how it determines if types are compatible between endpoints. -Simpler middlewares may not do anything other than check that the type names match, in which case the hash will likely used as an extra comparison to determine compatibility. -However, in more sophisticated middlewares type compatibility can be determined using more complex rules and by looking at the type descriptions themselves, and in those cases the type hash may not be used to determine matching. - -When creating a publisher or subscription, the caller normally provides: - -- a topic name, -- QoS settings, and -- a topic type name - -Where the topic type name is represented as a string and is automatically deduced based on the type given to the function that creates the entity. -The type may be passed as a template parameter in C++ or as an argument to the function in Python. - -For example, creating a publisher for the C++ type ``std_msgs::msg::String`` using ``rclcpp`` may result in a topic type name ``"std_msgs/msg/String"``. - -All of the above items are used by the middleware to determine if two endpoints should communicate or not, and this REP proposes that the type version be added to this list of provided information. - -Nothing needs to change from the user's perspective, as the type version can be extracted automatically based on the topic type given, either at the ``rcl`` layer or in the ``rmw`` implementation itself. -That is to say, how users create things like publishers and subscription should not need to change, no matter which programming language is being used. - -However, the type hash would become something that the ``rmw`` implementation is provided and aware of in the course of creating a publisher or subscription. -The decision of whether or not to use that information to enforce type compatibility would be left to the middleware, rather than implementing it as logic in ``rcl`` or other packages above the ``rmw`` API. - -The method for implementing the detection and enforcement of type version mismatches is left up to the middleware. -Some middlewares will have tools to handle this without the type hash and others will implement something like what would be possible in the ``rcl`` and above layers using the type hash. -By keeping this a detail of the ``rmw`` implementation, we allow the ``rmw`` implementations to make optimizations where they can. - -Recommended Strategy for Enforcing that Type Versions Match -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If the middleware has a feature to handle type compatibility already, as is the case with DDS-XTypes which is discussed later, then that can be used to enforce type safety, and then the type hash would only be used for debugging and for storing in recordings. - -However, if the middleware lacks this kind of feature, then the recommended strategy for accomplishing this in the ``rmw`` implementation is to simply concatenate the type name and the type hash with double underscores and then use that as the type name given to the underlying middleware. -For example, a type name using this approach may look like this: - -.. code:: - - sensor_msgs/msg/Image__RIHS01_XXXXXXXXXXXXXXXXXXXX - -This has the benefit of "just working" for most middlewares which at least match based on the name of the type, and it is simple, requiring no further custom hooks into the middleware's discovery or matchmaking process. - -However, one downside with this approach is that it makes interoperation between ROS 2 and the "native" middleware more difficult, as appending the type hash to the type name is just "one more thing" that you have to contend with when trying to connect non-ROS endpoints to a ROS graph. - -Alternative Strategy for Enforcing that Type Versions Match -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Sometimes the recommended strategy would interfere with features of the middleware that allow for more complex type compatibility rules, or otherwise interferes with the function of the underlying middleware. -In these cases, it is appropriate to not take the recommended strategy and delegate the matching process and notification entirely to the middleware. - -However, in order to increase compatibility between rmw implementations, it is recommended to fulfill the recommended approach whenever possible, even if the type name is not used in determining if two endpoints will match, i.e. in the case that the underlying middleware does something more sophisticated to determine type compatibility but ignores the type name itself. - -This recommendation is particularly useful in the case where a DDS based ROS 2 endpoint is talking to another DDS-XTypes based ROS 2 endpoint. -The DDS-XTypes based endpoint then has a chance to "gracefully degrade" to interoperate with the basic DDS based ROS 2 endpoint. -This would not be the case if the DDS-XTypes based ROS 2 endpoint did not include the type hash in the type name, as is suggested with the recommended strategy. - -.. TODO:: - - We need to confirm whether or not having a different type name prevents DDS-XTypes from working properly. - We've done some experiments, but we need to summarize the results and confirm the recommendation in the REP specifically geared towards DDS. - -Notifying the ROS Client Library -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -One potential downside to delegating type matching to the rmw implementation is that detecting the mismatch is more complicated. -If ROS 2 is to provide users a warning that two endpoints will not communicate due to their types not matching, it requires there to be a way for the middleware to notify the ROS layer when a topic is not matched due to the type incompatibility. -As some of the following sections describe, it might be that the rules by which the middleware decides on type compatibility are unknown to ROS 2, and so the middleware has to indicate when matches are and are not made. - -If the middleware just uses the type name to determine compatibility, then the rmw implementation can compare type hashes, and if they do not match between endpoints then the rmw implementation can notify ROS 2, and a warning can be produced. - -Either way, to facilitate this notice, the ``rmw_event_type_t`` shall be extended to include a new event called ``RMW_EVENT_OFFERED_TYPE_INCOMPATIBLE``. -Related functions and structures will also be updated so that the event can be associated with specific endpoints. - -Notes for Implementing the Recommended Strategy with DDS -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The DDS standard provides an ``on_inconsistent_topic()`` method on the ``ParticipantListener`` class as a callback function which is called when an ``INCONSISTENT_TOPIC`` event is detected. -This event occurs when the topic type does not match between endpoints, and can be used for this purpose, but at the time of writing (July 2022), this feature is not supported across all of the DDS vendors that can be used by ROS 2. - -Interactions with DDS-XTypes or Similar Implicit Middleware Features -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When using DDS-Xtypes, type compatibility is determined through sophisticated and configurable rules, allowing for things like extensible types, optional fields, implicit conversions, and even inheritance. -Which of these features is supported for use with ROS is out of scope with this REP, but if any of them are in use, then it may be possible for two endpoints to match and communicate even if their ROS 2 type hashes do not match. - -In this situation the middleware is responsible for communicating to the rmw layer when an endpoint will not be matched due to type incompatibility. -The ``INCONSISTENT_TOPIC`` event in DDS applies for DDS-XTypes as well, and should be useful in fulfilling this requirement. - -Run-Time Interface Reflection ------------------------------ - -Run-Time Interface Reflection allows access to the data in the fields of a serialized message buffer when given: - -- the serialized message as a ``rmw_serialized_message_t``, basically just an array of bytes as a ``rcutils_uint8_array_t``, -- the message's type description, e.g. received from "type description distribution" or from a bag file, and -- the serialization format, name and version, which was used to create the serialized message, e.g. ``XCDR2`` for ``Extended CDR encoding version 2`` - -From these inputs, we should be able to access the fields of the message from the serialized message buffer using some programmatic interface, which allows you to: - -- list the field names -- list the field types -- access fields by name or index - -.. code:: - - message_buffer ─┐ ┌───────────────────────────────┐ - │ │ │ - message_description ─┼──►│ Run-Time Interface Reflection ├───► Introspection API - │ │ │ - serialization_format ─┘ └───────────────────────────────┘ - -Given that the scope of inputs and expected outputs is so limited, this feature should ideally be implemented as a separate package, e.g. ``rcl_serialization``, that can be called independently by any downstream packages that might need Run-Time Interface Reflection, e.g. introspection tools, rosbag transport, etc. -This feature can then be combined with the ability to detect type mismatches and obtain type descriptions to facilitate communication between nodes of otherwise incompatible types. - -Additionally, it is important to note that this feature is distinct from ROS 2's existing "dynamic" type support (``rosidl_typesupport_introspection_c`` and ``rosidl_typesupport_introspection_cpp``). -The ``rosidl_typesupport_introspection_c*`` generators generate code at compile time for known types that provides reflection for those types. -This new feature, Run-Time Interface Reflection, will support reflection without generated code at compile time, instead dynamically interpreting the type description to provide this reflection. - -.. TODO:: - - Determine if the generated introspection API needs to continue to exist. - It might be that we just remove it entirely in favor of the new system described here, or at least merge them, as their API (after initialization) will probably be the same. - -.. TODO:: - - (wjwwood) I'm realizing now that we probably need to separate this section into two parts, first the reflection API used by the user and the rmw implementation, and then second the rmw implementation specific part, which we could call "Run-Time Type Support"? - It would be called such because it would be something like "TypeDescription in -> ``rosidl_message_type_support_t`` out"... - This second part is needed to make truly generic subscriptions and publishers, which until now has been kind of assumed in this section about reflection. - -.. TODO:: - - This section needs to be updated to be inclusive to at least Services, and then we can also mention Actions, though they are a combination of Topics and Services and so don't need special support probably. - -Plugin Interface -^^^^^^^^^^^^^^^^ - -As Run-Time Interface Reflection is expected to work across any serialization format, the Run-Time Interface Reflection interface needs to be extensible so that the necessary serialization libraries can be loaded to process the serialized message. -Serialization format support in this case will be provided by writing plugins that wrap the serialization libraries that can then provide the Run-Time Interface Reflection feature with the needed facilities. -Therefore, when initializing a Run-Time Interface Reflection instance it will: - -- enumerate the supported serialization library plugins in the environment, -- match the given serialization format to appropriate plugins, -- select a plugin if more than one matches the criteria, and then -- dynamically load the plugin for use - -These serialization library plugins must implement an interface which provides methods to: - -- determine if the plugin can be used for a given serialization type and version, -- provide an API for reflection of a specific type, given the type's description, - - - mainly including the parsed ``TypeDescription`` instance, but also - - perhaps the original ``.idl`` / ``.msg`` text, and - - perhaps other extra information, - -- provide a ``rosidl_message_type_support_t`` instance given similar information from the previous point - -In particular, providing information beyond the ``TypeDescription``, like the ``.idl`` / ``.msg`` text and the serialization library that was used, may be necessary because there might be serialization library or type support specific steps or considerations (e.g. name mangling or ROS specific namespacing) that would not necessarily be captured in the ``.idl`` / ``.msg`` file. - -Dealing with Multiple Applicable Plugins for A Serialization Format -""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -In the case where there exists multiple applicable plugins for a particular serialization format (e.g. when a user's machine has a plugin for both RTI Connext's CDR library and Fast-CDR), the plugin matching should follow this priority order: - -- a user specified override, passed to the matching function, or -- a default defined in the plugin matching function, or else -- the first suitable plugin in alphanumeric sorting order - -.. TODO:: - - (wjwwood) We should mention here that the (implicitly read-only) reflection described here is just one logical half of what we could do with the type description. - We could also provide the ability to dynamically define types and/or write to types using the reflection. - - For example, imagine a tool that subscribes to any type (that's just using the RTIR) but then modifies one of the fields by name and then re-publishes it. - That tool would need the ability to mutate an instance using reflection then serialize it to a buffer. - You couldn't just iterate with an existing buffer easily, because it would potentially need to grow the buffer. - - Also, consider a "time stamping" tool that subscribes to any message, and then on-the-fly defines a new message that has a timestamp field and a field with the original message in it, fills that out and publishes it. - That would need the ability to both create a new type from nothing (maybe no more than creating a ``TypeDescription`` instance) as well as a read-write interface to the reflection. - We don't have to support these things in this REP, but we should at least mention them and state if it is in or out of the scope of the REP. - -Tools for Evolving Types ------------------------- - -The previous sections of the specification have all be working up to supporting the new tools described in this section. - -The goal of these new tools is to help users reason about type versions of ROS interfaces, define transfer functions between two versions of a type when necessary, and put the transfer functions to use in their system to handle changes in types that have occurred. - -These tools will come in the form of new APIs, integrations with the build system, command-line tools, and integration with existing concepts like launch files. - -This section will describe a vision of what is possible with some new tools, but it should not be considered completely prescriptive. -That is to say, after this REP is accepted, tools described in this section may continue to evolve and improve, and even more tools not described here may be added to enable more things. -As always, consider the latest documentation for the various pieces of the reference implementation to get the most accurate details. - -Notifying Users when Types Do Not Match -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There should be a warning to users when two endpoints (publisher/subscription or Service server/client, etc.) on the same topic (or service/action) are using different type names or versions and therefore will not communicate. -This warning should be a console logging message, but it should also be possible for the user to get a callback in their own application when this occurs. -How this should happen in the API is described in the Type Version Enforcement section, and it uses the "QoS Event" part of the API. - -The warning should include important information, like the GUID of the endpoints involved, the type name(s) and type hash(es), and of course the topic name. -These pieces of information can then be used by the user, with the above mentioned changes to the command line tools, to investigate what is happening and why the types do not match. -The warning may even suggest how to compare the types using the previously described ``ros2 topic compare`` command-line tool. - -Tools for Determining if Two Type Versions are Convertible -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There should be a way for a user to determine if it is possible to convert between two versions of a type without writing a transfer function. -This tool would look at the two type descriptions and determine if a conversion can be done automatically with a generated transfer function. -This will help the user know if they need to write a transfer function or check if one already exists. - -This tool might look something like this: - -- ``ros2 interface convertible --from-remote-node --to-local-type `` - -This tool would query the type description from a remote node, and compare it to the type description of the second type version found locally. -If the two types can be converted without writing any code, then this tool would indicate that on the ``stdout`` and by return code. - -There could be ``--from-local-type`` and ``--to-remote-node`` options as well as others too. - -.. TODO:: - - (wjwwood) we need to come back to this section when we start working on the reference implementation, as more details will be clearer then - -Tools for Writing User-Defined Transfer Functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To help users write their own transfer functions, when conversions cannot be done automatically, we need to provide them with build system support and API support in various languages. - -The transfer functions will be defined (at least) as a C++ plugin, in the style of component nodes, rviz display plugins, and other instances. -These transfer function plugins can then be loaded and chained together as needed inside of a component node. - -.. TODO:: - - (wjwwood) we need to figure out if this is how we're gonna handle Services, or if instead they will be plugins rather than component nodes, but component nodes with remapping solves a lot of QoS related issues that arise when you think about instead making the conversions happen as a plugin on the publisher or subscription side with topics. - -At the very least we should: - -- provide a convenient way to build and install transfer functions from CMake projects -- provide an API to concisely define transfer functions in C/C++ -- document how transfer functions are installed, discovered, and run - -Documenting how this works will allow support for other build systems and programming languages can be added in the future. -It will also allow users that do not wish to use our helper functions (and the dependencies that come with them) in their projects, e.g. if they would like to use plain CMake and not depend on things like ``ament_cmake``. - -The process will look something like this: - -- compile the transfer function into a library -- put an entry into the ``ament_index`` which includes the transfer functions details, like which type and versions it converts between, and which library contains it - -Putting it into a library allows us to later load the transfer function, perhaps along side other transfer functions, into a single component node which subscribes to the input type topic and publishes to the output type topic. - -The interface for a transfer function will look like a modified Subscription callback, receiving a single ROS message (or request or response in the case of Services) as input and returning a single ROS message as output. -The input will use the Run-Time Interface Reflection API and therefore will not be a standard message structure, e.g. ``std_msgs::msg::String`` in C++, but the output may be a concrete structure if that type is available when compiling the transfer function. - -The transfer function API for C++ may look something like this: - -.. code:: - - void - my_transfer_function( - const DynamicMessage & input, - const MessageInfo & input_info, // type name/hash, etc. - DynamicMessage & output, - const MessageInfo & output_info) - { - // ... conversion details - } - - REGISTER_TRANSFER_FUNCTION(my_transfer_function) - -Registering a transfer function in an ``ament_cmake`` project might look like this: - -.. code:: - - create_transfer_function( - src/my_transfer_function.cpp - FUNCTION_NAME my_transfer_function - FROM sensor_msgs/msg/Image RIHS01_XXXXXXXXXXXXXXXXX123 - TO sensor_msgs/msg/Image RIHS01_XXXXXXXXXXXXXXXXXabc - ) - -This CMake function would create the library target, link the necessary libraries, and install any files in the ``ament_index`` needed to make the transfer function discoverable by the tools. -This example is the for the simplest case, which we should make easy, but other cases like supporting multiple transfer functions per library, creating the target manually, or even supporting other programming languages would require more complex interfaces too. - -.. TODO:: - - (wjwwood) come back with more details of this interface as the reference implementation progresses - -Tools for Interacting with Available Transfer Functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There should be a way to list and get information about the available transfer functions in your local workspace. -This tool would query the ``ament_index`` and list the available transfer functions found therein. - -The tool might look like this: - -- ``ros2 interface transfer_functions list`` -- ``ros2 interface transfer_functions info `` - -The tool might also have options to filter based on the type name, package name providing the type, package name providing the transfer function (not necessarily the same as the package which provides the type itself), etc. - -.. TODO:: - - (wjwwood) not clear to me if the transfer functions should have names or not. What if you have more than one transfer function for a type versions pair, e.g. more than one conversion for sensor_msgs/msg/Image from ABC to XYZ. I'm not sure why this would happen, but it is technically possible since the packages register them separately. - -Tools for Using Transfer Functions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Additionally, there should be a tool which will look at the available transfer functions, including the possible automatic conversions, and start the node that will do the conversions if there exists a set of transfer function which would convert between two different versions of a type. - -It might look like this: - -.. code:: - - ros2 interface convert_topic_types \ - --from /topic/old \ - --to /topic \ - --component-container - -This tool would find the set of transfer functions between the types on ``/topic/old`` and ``/topic``, assuming they are the same type at different versions and that there exists the necessary transfer functions, and load them into a component container as a component node. - -The tool could run a stand-alone node instance if provided a ``--stand-alone`` option instead of the ``--component-container`` option. - -It could also have options like ``--check`` which would check if it could convert the types or not, giving detailed information about why it cannot convert between them if it is not possible. -This could help users understand what transfer functions they need to write to bridge the gap. - -This tool would also potentially need to let the user decide which path to take if there exists multiple chains of transfer functions from one version to the other. -For example, if the types are ``Type@ABC`` and ``Type@XYZ``, and there are transfer functions for ``Type@ABC -> Type@DEF``, ``Type@DEF -> Type@XYZ``, ``Type@ABC -> Type@XYZ``, and possibly other paths, then it might need help from the user to know which path to take. -By default it might choose the shortest path and then if there's a tie, one arbitrarily. - -There would be similar versions of this tool for Services and Actions as well. - -Integration with Launch -^^^^^^^^^^^^^^^^^^^^^^^ - -With the previously described tool ``ros2 interface convert_topic_types``, you could simply remap topics and execute an instance of it with launch: - -.. code:: - - - - - - - - - -However, we can provide some "syntactic sugar" to make it easier, which might look like this: - -.. code:: - - - - - - - - -That syntactic sugar can do a lot of things, like: - -- remap the topic to something new, like ``/topic/XXX`` -- run the node its enclosed in, wait for the topic to come up to check the version -- run any transfer functions in a node to make the conversion from ``/topic/XXX`` to ``/topic`` - -This could also be extended to support component nodes, rather than stand alone nodes, etc. - -Integration with ros2bag -^^^^^^^^^^^^^^^^^^^^^^^^ - -Similar to the integration with launch, you could run ``ros2 bag play ...`` and the ``ros2 interface convert_topic_types ...`` separately, but we could also provide an option to rosbag itself, which might look like this: - -.. code:: - - ros2 bag play /path/to/bag --convert-to-local-type /topic - -That option would handle the conversion on the fly using the same mechanisms as the command line tool. - -Rationale -========= - -This section captures further rationales for why the specification is the way it is, and also offers summaries of alternatives considered but not selected for various parts of the specification. - -Type Version Enforcement ------------------------- - -Alternatives -^^^^^^^^^^^^ - -Evolving Message Definition with Extensible Type -"""""""""""""""""""""""""""""""""""""""""""""""" - -When defining the ``.idl`` msg file, user can choose to apply annotations to the message definition (DDS XTypes spec v1.3: 7.3.1.2 Annotation Language). -Evolving message type can be achieved by leveraging optional fields and inheritance. -For example, the ``Temperature.idl`` below uses ``@optional`` and ``@extensibility`` in the message definition. - -.. code:: - - @extensibility(APPENDABLE) - struct Temperature - { - unsigned long long timestamp - long long temperature - @optional double temperature_float - }; - -Furthermore, an initial test evolving messages with FastDDS, Cyclone, and Connext middleware implementations show that ``@appendable`` and ``@optional`` are implemented in Cyclone and Connext, but not FastDDS (as of Jul 2022). - -.. TODO:: - - (wjwwood) we need to follow up on this - -Handle Detection of Version Mismatch "Above" rmw Layer -"""""""""""""""""""""""""""""""""""""""""""""""""""""" - -We can choose to utilize ``USER_DATA`` QoS to distribute the type hash during the discovery phase. -The type hash for each participant will then be accessible across all available nodes. By getting the hash through ``user_data`` via the ``rmw`` layer, similar type version matching can be detected. - -Prevent Communication of Mismatched Versions "Above" rmw Layer -"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" - -TODO - -Run-Time Interface Reflection ------------------------------ - -TODO - -Plugin Matching and Loading API Example -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The following is an example of how this plugin matching and loading interface could look like, defining new ``rcl`` interfaces; with a plugin wrapping FastCDR v1.0.24 for serialization of ``sensor_msgs/msg/LaserScan`` messages: - -.. code:: - - // Suppose LaserScanDescription reports that it uses FastCDR v1.0.24 for its serialization - rcl_runtime_introspection_description_t LaserScanDescription = node->get_type_description("/scan"); - - rcl_type_introspection_t * introspection_handle; - introspection_handle->init(); // Locate local plugins here - - // Plugin name: "fastcdr_v1_0_24" - const char * plugin_name = introspection_handle->match_plugin(LaserScanDescription->get_serialization_format()); - rcl_serialization_plugin_t * plugin = introspection_handle->load_plugin(plugin_name); - - // If we wanted to force the use of MicroCDR instead - introspection_handle->match_plugin(LaserScanDescription->get_serialization_format(), "microcdr"); - -Run-Time Interface Reflection API Example -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The following is an example for how the introspection API could look like. -This example will show a read-only interface. - -It should comprise several components -- a handler for the message buffer, to handle pre-processing (e.g. decompression) -- a handler for the message description, to keep track of message field names of arbitrary nesting level -- handler functions for message buffer introspection - -Also, this example uses the LaserScan message definition: https://github.com/ros2/common_interfaces/blob/foxy/sensor_msgs/msg/LaserScan.msg - -.. TODO:: - - (methylDragon) Add a reference somehow? - -First, the message buffer handler: - -.. code:: - - struct rcl_buffer_handle_t { - const void * buffer; // The buffer should not be modified - - const char * serialization_type; - rcl_serialization_plugin_t * serialization_plugin; - rcl_runtime_introspection_description_t * description; // Convenient to have - - // And some examples of whatever else might be needed to support deserialization or introspection... - void * serialization_impl; - } - -The message buffer handler should allocate new memory if necessary, or store a pointer to the message buffer otherwise in its ``buffer`` member. - -Then, functions should be written that allow for convenient traversal of the type description tree. -These functions should allow a user to get the field names and field types of the top level type, as well as from any nested types. - -.. code:: - - struct rcl_field_info_t { // Mirroring Field - const char * field_name; // This should be an absolute address (e.g. "header.seq", instead of "seq") - - uint8_t type; - const char * nested_type_name; // Populated if the type is not primitive - }; - - // Get descriptions - rcl_runtime_introspection_description_t LaserScanDescription = node->get_type_description("/scan"); - rcl_runtime_introspection_description_t HeaderDescription = node->get_referenced_description(LaserScanDescription, "Header"); - - // All top-level fields from description - rcl_field_info_t ** fields = get_field_infos(&LaserScanDescription); - - // A single field from description - rcl_field_info_t * header_field = get_field_info(&LaserScanDescription, "header"); - - // A single field from a referenced description - rcl_field_info_t * stamp_field = get_field_info(&HeaderDescription, "stamp"); - - // A nested field from top-level description - rcl_field_info_t * stamp_field = get_field_info(&LaserScanDescription, "header.stamp"); - -Finally, there should be functions to obtain the data stored in the message fields. -This could be by value or by reference, depending on what the serialization library supports, for different types. - -There minimally needs to be a family of functions to obtain data stored in a single primitive message field, no matter how deeply nested it is. -These need to be created for each primitive type. - -The rest of the type introspection machinery can then be built on top of that family of functions, in layers higher than the C API. - -.. code:: - - rcl_buffer_handle_t * scan_buffer = node->get_processed_buffer(some_raw_buffer); - - // Top-level primitive field - get_primitive_field_float32(scan_buffer, "scan_time"); - - // Nested primitive field - get_primitive_field_uint32_seq(scan_buffer, "header.seq"); - - // Nested primitive field sequence element (overloaded) - get_field_seq_length(scan_buffer, "header.seq"); // Support function - get_primitive_field_uint32(scan_buffer, "header.seq", 0); - -If we attempt to do the same by reference, the plugin might decide to allocate new memory for the pointer, or return a pointer to existing memory. - -.. code:: - - // Nested primitive field - get_primitive_field_uint32_seq_ptr(scan_buffer, "header.seq"); - - // Be sure to clean up any dangling pointers - finalize_field(some_field_data_ptr); - -The following should be error cases: - -- accessing field data as incorrect type -- accessing or introspecting incorrect/nonexistent field names - -.. TODO: (methylDragon) Are there more cases? It feels like there are... - -- the raw message buffer should outlive the ``rcl_buffer_handle_t``, since it is not guaranteed that the buffer handle will allocate new memory -- the ``rcl_buffer_handle_t`` should outlive any returned field data pointers, since it is not guaranteed that the serialization plugin will allocate new memory -- however, ``rcl_field_info_t`` objects **do not** have any lifecycle dependencies, since they are merely descriptors - -Alternatives -^^^^^^^^^^^^ - -Name of Run-Time Interface Reflection -""""""""""""""""""""""""""""""""""""" - -Other names were considered, like "Runtime Type Introspection", but this name was selected for three main reasons: - -- to avoid confusion with C++'s Run-time type information (RTTI)\ [3]_, -- to show it was more than RTTI, but instead was also reflection, - - - like how the RTTR C++ library\ [4]_ differs from normal RTTI, and - -- to show that it deals not just with any "type" but specifically ROS's Interfaces - - -Backwards Compatibility -======================= - -TODO - - -Feature Progress -================ - -Supporting Feature Development: - -- Type Version Enforcement - - - [ ] Add rmw API for notifying user when a topic endpoint was not matched due to type mismatch - - - [ ] Detect when Message versions do not match, prevent matching, notify user: - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds - - - [ ] Detect when Service versions do not match, prevent matching, notify user: - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds - - - [ ] Detect when Action versions do not match, prevent matching, notify user: - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds - -- Run-time Interface Reflection: - - - [ ] Prototype "description of type" -> "accessing data fields from buffer" for each rmw vendor: - - - [ ] Fast-DDS - - [ ] CycloneDDS - - [ ] RTI Connext Pro - - - [ ] Prototype "description of type" -> "create datareaders/writers" for each rmw vendor: - - - [ ] Fast-DDS - - [ ] CycloneDDS - - [ ] RTI Connext Pro - - - [ ] Implement reflection API in ``rcl_serialization`` package, providing (de)serialization with just a description - - [ ] Implement single "plugin" using Fast-CDR (tentatively) - - [ ] Implement plugin system, allowing for multiple backends for (X)CDR - - [ ] Implement at least one other (X)CDR backend, perhaps based on RTI Connext Pro - - - [ ] Add "description of type" -> ``rosidl_message_type_support_t`` to ``rmw`` API - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds - - - [ ] Add "description of type" -> ``rosidl_service_type_support_t`` to ``rmw`` API - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds - - - [ ] Update APIs for rclcpp to support using TypeDescription instances to create entities: - - - [ ] Topics - - [ ] Services - - [ ] Actions - - - [ ] Update APIs for rclpy to support using TypeDescription instances to create entities: - - - [ ] Topics - - [ ] Services - - [ ] Actions - -Tooling Development: - -- [ ] Command-line tool for determining if two versions of a type are convertible - -- [ ] CMake and C++ APIs for writing transfer functions, including usage documentation -- [ ] Command-line tools for interacting with user-defined transfer functions -- [ ] Command-line tool for starting a conversion node between two versions of a type - -- [ ] Integration with launch, making it easier to set up conversions in launch files - -- [ ] Integration with rosbag2 - - - [ ] Use recorded type description metadata to create subscriptions without needing the type to be built locally - - [ ] Use recorded metadata to create publishers from descriptions in bag - - -References -========== - -.. [#rep2016] REP 2016: ROS 2 Interface Type Descriptions - (https://www.ros.org/reps/rep-2016.html) - -.. [1] DDS-XTYPES 1.3 - (https://www.omg.org/spec/DDS-XTypes/1.3/About-DDS-XTypes/) - -.. [2] IDL - Interface Definition and Language Mapping - (http://design.ros2.org/articles/idl_interface_definition.html) - -.. [3] Run-time type information - (https://en.wikipedia.org/wiki/Run-time_type_information) - -.. [4] RTTR (Run Time Type Reflection): An open source library, which adds reflection to C++ - (https://www.rttr.org/) - - -Copyright -========= - -This document has been placed in the public domain. - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: From 2f84e25cb01d493f7bfe2ac9ef147c95db634d30 Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Tue, 11 Jul 2023 02:25:42 -0700 Subject: [PATCH 07/10] Full pass through the documetn Signed-off-by: Emerson Knapp --- rep-2016.rst | 502 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 324 insertions(+), 178 deletions(-) diff --git a/rep-2016.rst b/rep-2016.rst index 290dbd85..2955c3e0 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -11,39 +11,45 @@ Post-History: Abstract ======== -This REP proposes a standard format for the communication of ROS 2 interface types, along with tooling to support their use. +This REP proposes a standard format for communicating the structure of ROS 2 interface types, as well as tooling to support the use of these descriptions. -It provides a ROS 2 interface type that can contain these descriptions, runtime data structures, and a universally reproducible serialization method. -Alongside the description format, this REP also defines a hashing algorithm to create standardized hashes of type descriptions, to quickly detect type mismatch during discovery. +It provides a ROS 2 interface type that can contain descriptions of types, runtime data structures, and a universally reproducible serialization method. +Alongside the description format, a standard hashing algorithm is also defined to create compact reperentations of type descriptions, to quickly transmit and detect type mismatch during discovery. Motivation ========== ROS 1 clients provided an MD5 sum from msg files to check for matching definitions, as well as full message definitions in connection headers. -ROS 2 before Iron did not provide any comparable capabilities. -This REP proposes a set of standards to be implemented in ROS 2 that allow users to easily +This REP proposes to match the goal of those features with a set of standards to that allow users to easily - detect mismatches in message types between nodes -- get access to the complete type description of types being used, including from remote nodes +- get access to the complete description of types being used, including from remote nodes + +One part of this motivation is for correctness of existing applications. +When a message type is updated, a different communicating node may begin to receive messages it does not actually understand. +A second part of the motivation is to expand capabilities of the ROS 2 system to work with descriptions of types programmatically. +This REP will not go into detail on those supported use cases, they are discussed in more detail in REP-2011\ [#rep2011]_. + Terminology =========== -TODO: formatting +Some terms used in this document, which will be described in much more detail in the specification: + +- **ROS 2 Interface** or **Interface Type** - a ROS 2 message, service, or action, as described in \ [#interfaces]_. +- **Type Source** - the original text provided by author to define an interface type, such as a ``.msg`` or ``.idl`` file. +- **Type Description** - a data structure representating a parsed type source, which will be equal regardless of source format such as ``.msg`` or ``.idl`` if the described type is the same. +- **Type Hash** - a compact identifier for a type description that is expected to be unique +- **RIHS (ROS Interface Hashing Standard)** - a versioned specification for producing a hash value for a type description -Type Source - original text used to define a type -Type Description - a data structure representating a parsed Type Source, which removes irrelevant data such as comments, and will be equal regardless of whether the source was msg, IDL, or otherwise. -RIHS (ROS Interface Hashing Standard) - a versioned specification for producing a hash value for a Type Description -type_description_interfaces - the new package defined to contain the Type Description data structures Specification ============= -In order to communicate and compare types, a Type Description data structure is defined to contain type information in a common format. -To detect and enforce type version mismatches, and communicate information about type descriptions compactly, a way to uniquely identify types is required. -For this purpose a hash of the Type Description, or Type Hash, is used. -The Type Hash specification, covered below, defines a Type Description serialization and hashing algorithm that guarantees cross-implementation consistency. +In order to represent, communicate and compare types, a Type Description data structure is defined. +To communicate information about type descriptions compactly and detect type mismatches, we use a Type Hash. +The Type Hash specification defines a Type Description serialization and hashing algorithm that guarantees cross-implementation consistency. Type Description @@ -60,29 +66,172 @@ Thus the representation includes: - a list of all recursively referenced types - no comments +Note that the reference implementation is provided in the package ``type_description_interfaces``\ [#tdi_pkg]_. +The structure is discussed here in concept, but the implementation is the source of exact truth for ROS releases. + +The ``TypeDescription`` message type looks like: + +.. code:: + + IndividualTypeDescription type_description + IndividualTypeDescription[] referenced_type_descriptions + +And the ``IndividualTypeDescription`` type: + +.. code:: + + string type_name + Field[] fields + +And the ``Field`` type: + +.. code:: + + FIELD_TYPE_NESTED_TYPE = 0 + FIELD_TYPE_INT = 1 + FIELD_TYPE_DOUBLE = 2 + # ... and so on + FIELD_TYPE_INT_ARRAY = ... + FIELD_TYPE_INT_BOUNDED_SEQUENCE = ... + FIELD_TYPE_INT_SEQUENCE = ... + # ... and so on + + string field_name + uint8_t field_type + uint64_t field_array_size # Only for Arrays and Bounded Sequences + string nested_type_name # Only for FIELD_TYPE_NESTED_TYPE + +These interfaces perhaps do not yet consider some other complications like field annotations or other as yet unconsidered features we want to support. + + +Describing Services and Actions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +ROS 2 Services and Actions are considered "aggregate interfaces", in that they compose several "sub-interfaces" into a more complex type. +A Service is defined using two Messages - a Request and a Reponse. +An Action is defined using three Messages - a Goal, Feedback, and a Result. +These definitions from the user are then expanded into the more derived utilized by communications. +Because these full structures affect interoperability, the full description as used by ROS 2 should be represented to accurately describe the types, rather than just the inputs. +To represent these types, we consider them to simply be types with these members as fields. + +As of ROS 2 Iron, Services contain + +- a Request message as defined in input +- a Response message as defined in input +- an Event message, as defined in REP-2012 Service Introspection\ [#rep2012]_. + +Therefore the implicit intermediary output looks like: + +.. code:: + + ServiceType_Request request_message + ServiceType_Response response_message + ServiceType_Event event_message + +And Actions contain + +- a Goal message as defined in input +- a Result message as defined in input +- a Feedback message as defined in input +- a SetGoal service, derived +- a GetResult service, derived +- a FeedbackMessage, derived + +Therefore their implicit intermediary output looks like: + +.. code:: + + ActionType_Goal goal + ActionType_Result result + ActionType_Feedback feedback + ActionType_SendGoalService send_goal_service + ActionType_GetResultService get_result_service + ActionType_FeedbackMessage feedback_message + + +While this method is simple and therefore recommended, it is worth noting that it has the following drawback: +A message defined explicitly with those above field types and names would have an identical description to a Service or Action, even though they are different category of interfaces. +However, this case is fairly contrived and in practice an application will already know what interface category it is trying to interact with. + +Given this approach, the ``type_description_interfaces`` provides no special accommodation for Services and Actions, leaving the transform into a representable type to code generation. +The reference implementation for this derivation is in ``rosidl_generator_type_description``\ [#td_gen]_. + +Versioning the ``TypeDescription`` Message Type +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Given that the type description message interface has to be generic enough to support anything described in the ROS interfaces, there will be a need to add or remove fields over time in the type description message itself. +This should be done in such a way that the fields are tick-tocked and deprecated properly. +It is recommended to do this by having explicitly named versions of this interface, e.g. ``TypeDescriptionV2``, ``TypeDescriptionV3`` and so on. +The initial version is named without a version identifier. +Revisions may be inevitable, but there should be a default preference against modifying, to keep the number of these versions to a minimum. + +``TypeDescription`` Self-Describing Loop Problem +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This section covers an implementation detail rather than a goal-oriented design decision, but is worth covering as it represents a significant difficulty. + +The ``TypeDescription`` messages are defined as just another ROS interface type, using all the same tooling. +This means that its data structures are not available yet, during build time, to describe itself and other interface packages. +The final decision to support this is a checked-in mirror of the generated type structs, provided in ``rosidl_runtime_c``, for use by C (used by Python) and C++ code generation. +This decision incurs overhead and possibility for human error, if the interfaces are ever updated, as the runtime structs must be also updated with new generated code. +To support this, infrastructure will have been added to automatically perform the updates, and check that they are updated properly in PR reviews. + +A few other options were considered but ultimately all were deemed too convoluted to follow even for those familiar with the ``rosidl`` type pipeline. Type Hash --------- -The hash must also be able to be calculated at runtime from information received on the wire. -The hash must only be computed using fields that affect communication compatibility. -Thus the hash excludes one aspect of Type Descriptions: it omits default values. -This is because default values for fields are only used by the writer of a packet, the recipient always receives some value in the field and can read it, thus defaults cannot affect compatibility. +Hash Requirements +^^^^^^^^^^^^^^^^^ + +The following requirements are set for type hashes. + +Hashes for local types be available to client libraries without computation, thus must be precalculated at build time. +The hash must be able to be calculated at runtime from information received on the wire. This allows subscribers to validate the received TypeDescriptions against advertised hashes, and allows dynamic publishers to invent new types and advertise their hash programmatically. +The hash must only be computed using fields that affect communication compatibility, so that trivial changes do not change the hash +Thus the hash excludes one aspect of Type Descriptions: it omits field default values. +This is because default values for fields are only used by the writer of a packet, the recipient always receives some value in the field and can read it, thus defaults cannot affect compatibility. +It also omits all other non-programmatic contents such as comments + Finally, the resulting filled data structure must be represented in a platform-independent format, rather than running the hash function on the in-memory native type representation. Different languages, architectures, or compilers will produce different in-memory representations, and the hash must be consistently calculable in different contexts. -The resulting data structure is hashed using SHA-256, resulting in a 256-bit (32-byte) hash value which is also generally known as a "message digest". -This hash is paired with a type version hash standard version, which we will call the "ROS IDL Hashing Standard" or "RIHS", the first version of which will be ``RIHS01``. -RIHS Version 00 is reserved for "Invalid" / "unset", and the RIHS version is limited by this specification to a maximum value of 255. +Representation for Hashing +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``RIHS01`` uses a JSON-serialized representation of the Type Description contents. +This representation is not intended to be used by humans, and therefore is kept as compact as possible by omitting line breaks and therefore indentation. +JSON was chosen as a ubiquitously available format with simple implementation. + +The serialized output must be byte-for-byte identical to ensure matching hash output. +``RIHS01`` has two separate implementations that together form its specification by their agreement. +These two reference implementations can be found in: + +- ``rosidl_generator_type_description`` +- ``rcl`` + +The JSON serialization must pay attention to certain aspects that even a JSON Schema can't represent: + +- Whitespace management, including newlines and spaces between commas, colons, etc +- Sorting ``referenced_type_descriptions`` in a consistent manner (probably alphabetically by type name) + +However, to handle everything else, the reference implementation does provide a ``jsonschema`` file that other implementatations can use. + +Hashing Method +^^^^^^^^^^^^^^ + +The resulting representation is hashed using SHA-256, resulting in a 256-bit (32-byte) hash value which is also generally known as a "message digest". +This hash is paired with a hashing standard version, which we will call the "ROS IDL Hashing Standard" or "RIHS", the first version of which will be ``RIHS01``. +RIHS Version 0 is reserved to mean "invalid" or "unset", and the RIHS version is limited by this specification to a maximum value of 255. RIHS hash values must have a well-defined UTF-8 string representation for human readability and for passing over string-only communication channels. The prefix of a well-formed RIHS string will always be ``RIHSXX_``, where ``X`` is one hexadecimal digit, followed by the version-dependent string representation of the hash value. -For ``RIHS01``, the hash value is 64 hexadecimal digits representing the 256-bit message digest, leading to a known ``RIHS01`` string length of 71. +For ``RIHS01``, the hash value is 64 hexadecimal digits representing the 256-bit message digest, leading to a fixed ``RIHS01`` string length of 71. -This versioning allows the tooling to know if a hash mismatch is due to a change in this standard (how hash is computed) or due to a difference in the interface types themselves. -In the case of a change in standard, it will be unknown whether the interface types are equal or not. +This versioning allows the tooling to know if a hash mismatch is due to a change in how the hash computed or due to a difference in the information content. +In the case of a new RIHS version, meaning a change in computation method, it will be unknown whether the interface types are equal or not without full description comparison. For now, the list of field names and their types are the only contributing factors, but in the future that could change, depending on which "annotations" are supported in ``.idl`` files. The "IDL - Interface Definition and Language Mapping" design document\ [2]_ describes which features of the OMG IDL standard are supported by ROS 2. @@ -95,23 +244,21 @@ New sanitizing may be needed on the TypeDescription pre-hash procedure, in the c Notes: -The type version hash is not sequential and does not imply any rank among versions of the type. That is, given two version hashes of a type, there is no way to tell which is "newer". +The type hash is not sequential and does not imply any rank among versions of the type. +That is, given two hashes of a type, there is no way to tell which is "newer". -Because the hash contains the stated name of the type, differently-named types with otherwise identical descriptions will be mismatched as incompatible. +Because the hash contains the stated name of the type, differently-named types with otherwise identical descriptions will receive different hashes, and therefore be mismatched as incompatible. This matches existing ROS precedent of strongly-typed interfaces. -The type version hash can only be used to determine if type versions are equal and if there exists a chain of transfer functions that can convert between them. -Because of this, when a change to a type is made, it may or may not be necessary to write transfer functions in both directions depending on how the interface is used. - .. note:: A message provider may desire to change the version hash of a message even when no field types or names have changed, perhaps due to a change in semantics of existing fields. There is explicitly no built-in provision for this case. We suggest the following method - provide an extra field within the interface with a name like ``bool versionX = true``. - To trigger a hash update, increment the name of this special field, for example to ``bool versionY = true``. + To trigger a hash update, increment the name of this special versioning field, for example to ``bool versionY = true``. The TypeDescription does not include the serialization format being used, nor does it include the version of the serialization technology. -This type version hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. -The type version hash must be considered in context, with the serialization format and version in order to determine wire compatibility. +This type hash is for the *description* of the type, and is not meant to be used to determine wire compatibility by itself. +The type hash must be considered in context, with the serialization format and version in order to determine wire compatibility. Type Hash Discovery @@ -147,169 +294,147 @@ While this usage can be implementation-specific, consistency will allow for comm Type Description Distribution ----------------------------- -For some use cases the type version hash is insufficient and instead the full type description is required. +For some use cases the type hash is insufficient and instead the full type description is required. -One of those use cases, which is also described in this REP, is "Run-Time Interface Reflection", which is the ability to introspect the contents of a message at runtime when the description for that message, or that version of that message, was unavailable at compile time. +One of those use cases is "Run-Time Interface Reflection", which is the ability to introspect the contents of a message at runtime when the description for that message, or that version of that message, was unavailable at compile time. In this use case the type description is used to interpret the serialized data dynamically. +Another use case is using the type description in tooling to either display the type description to the user or to include it in recordings. +Neither of these cases are covered in detail in this REP. -Another use case, which is not covered in this REP, is using the type description in tooling to either display the type description to the user or to include it in recordings. - -In either case, where the type description comes from doesn't really matter, and so, for example, it could be looked up on the local filesystem or read from a rosbag file. -However, in practice, the correct type description may not be found locally, especially in cases where you have different versions of messages in the same system, e.g.: +In any case, where the type description comes from doesn't matter, and so, for example, it could be looked up on the local filesystem or read from a rosbag file. +However, the correct type description may not be available locally, especially in cases where you have different versions of messages in the same system, e.g.: - because it's on another computer, or - because it is from a different distribution of ROS, or - because it was built in a different workspace, or - because the application has not been restarted since recompiling a change to the type being used -In any case, it is useful to have a mechanism to convey the type descriptions from the source of the data to other nodes, which we describe here as "type description distribution". +It is useful to have a mechanism to convey the type descriptions from the source of the data to other nodes, which is described here as "type description distribution". Furthermore, this feature should be agnostic to the underlying middleware and serialization library, as two endpoints may not have the same rmw implementation, or the data may have been serialized to a different format in the case of playback of a recording. Sending the Type Description ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Type descriptions will be provided by a ROS Service called ``~/_get_type_description``, which will be offered by each node. + +Type descriptions will be provided by a ROS Service called ``~/get_type_description``, which will be offered by each node. There will be a single ROS Service per node, regardless of the number of publishers or subscriptions on that node. -This service may be optional, for example being enabled or disabled when creating the ROS Node. +The type of this service is defined in ``type_description_interfaces`` as ``GetTypeDescription``. + +The service must be optional, but it will be a detail decided by each client library whether it is enabled or disabled by default. +This REP recommends that the default client libraries ``rclcpp`` and ``rclpy`` enable the service by a boolean Parameter named ``start_type_description_service``. .. TODO:: How can we detect when a remote node is not offering this service? It's difficult to differentiate between "the Service has not been created yet, but will be" and "the Service will never be created". - Should we use a ROS Parameter to indicate this? - But then what if remote access to Parameters (another optional Service) is disabled? + In the recommended approach, it is enabled via a parameter, but what if remote access to Parameters (another optional Service) is disabled? Perhaps we need a "services offered" list which is part of the Node metadata, which is sent for each node in the rmw implementation, but that's out of scope for this REP. -A service request to this ROS Service will comprise of the type name and the type version hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described in previous sections. -The ROS Service server will respond with the type description and any necessary metadata needed to do Run-Time Interface Reflection. +A service request to this ROS Service will comprise of the type name and the type hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described previously. +The ROS Service server will respond with the type description and optionally the original raw sources used to define the type. This service is not expected to be called frequently, and is likely to only occur when new topic or service endpoints are created, and even then, only if the endpoint type hashes do not match. - -.. TODO:: - - Should each endpoint be asked about their type/version pair or should we assume that the type/version pair guarantees a unique type description and therefore reuse past queries? - (wjwwood) I am leaning towards assuming the type name and type version hash combo as being a unique identifier. +It is expected that a name and hash pair uniquely represent a type, so an end tool such as rosbag can only make a single query per type, rather than per topic or publisher, further reducing the number of likely calls. +Given the SHA-256 implementation of RIHS01, the possibility of hash collision is so low as to be practically considered 0 and disregarded. Type Description Contents and Format ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The response sent by the ROS Service server will contain a combination of the original ``idl`` or ``msg`` file's content, as well as any necessary information to serialize and deserialize the raw message buffers sent on the topic. -The response will contain a version of the description that contains comments from the original type description, as those might be relevant to interpreting the semantic meaning of the message fields. - -Additionally, the response could include the serialization library used, its version, or any other helpful information from the original producer of the data. - -.. TODO:: +The response sent by the ROS Service server wll contain the type description. +Optionally, this response will also contain the original ``idl`` or ``msg`` file contents, as those might be relevant to interpreting the semantic meaning of the message fields. - What happens if the message consumer doesn't have access to the serialization library stated in the meta-type? - (wjwwood) It depends on what you're doing with the response. If you are going to subscribe to a remote endpoint, then I think we just refuse to create the subscription, as communication will not work, and there's a basic assumption that if you're going to communicate with an endpoint then you are using the same or compatible rmw implementations, which includes the serialization technology. The purpose of this section and ROS Service is to provide the information, not to ensure communication can happen. +This service does not contain any information about the serialization format or other middleware details. +The purpose of this section and the ROS Service is to provide information about the type, not to ensure full information required for wire communication. The ROS 2 message that defines the type description must be able to describe any message type, including itself, and since it is describing the message format, it should work independently from any serialization technologies used. -This "meta-type description" message would then be used to communicate the structure of the type as part of the "get type description" service response. -The final form of these interfaces should be found in the reference implementation, but such a Service interface might look like this: +This "meta-type description" message is used to communicate the structure of the type as part of the ``GetTypeDescription`` service response. +The final form of these interfaces should be found in the reference implementation, but such a Service looks like this: .. code:: + # ROS interface type name, in PACKAGE/NAMESPACE/TYPENAME format. string type_name - string type_version_hash + + # REP-2011 RIHS hash string. + string type_hash + + # Whether to return the original idl/msg/etc. source file(s) in the response. + bool include_type_sources true --- - # True if the type description information is available and populated in the response + # True if the type description information is available and populated in the response. + # If false, all other fields except `failure_reason` are considered undefined. bool successful - # Empty if 'successful' was true, otherwise contains details on why it failed + # If `successful` is false, contains a reason for failure. + # If `successful` is true, this is left empty. string failure_reason - # The idl or msg file name - string type_description_raw_file_name - # The idl or msg file, with comments and whitespace - # The file extension and/or the contents can be used to determine the format - string type_description_raw - # The parsed type description which can be used programmatically + # The parsed type description which can be used programmatically. TypeDescription type_description - # Key-value pairs of extra information. - string[] extra_information_keys - string[] extra_information_values + # A list containing the interface definition source text of the requested type, + # plus all types it recursively depends on. + # Each source text is a copy of the original contents of the + # .msg, .srv, .action, .idl, or other file if it exists, including comments and whitespace. + # Sources can be matched with IndividualTypeDescriptions by their `type_name`. + # The `encoding` field of each entry informs how to interpret its contents. + TypeSource[] type_sources -.. TODO:: + # Key-value pairs of extra information. + KeyValue[] extra_information - (wjwwood) I propose we use key-value string pairs for the extra information, but I am hoping for discussion on this point. - This type is sensitive to changes, since changes to it will be hard to roll out, and may need manual versioning to handle. - We could also consider bounding the strings and sequences of strings, so the message could be "bounded", which is nice for safety and embedded systems. -Again, the final form of these interfaces should be referenced from the reference implementation, but the ``TypeDescription`` message type might look something like this: +Where ``TypeSource`` looks like: .. code:: - IndividualTypeDescription type_description - IndividualTypeDescription[] referenced_type_descriptions - -And the ``IndividualTypeDescription`` type: - -.. code:: + # Represents the original source of a ROS 2 interface definition. + # ROS interface type name, in PACKAGE/NAMESPACE/TYPENAME format. string type_name - Field[] fields -And the ``Field`` type: + # The type of the original source file, typically matching the file extension. + # Well-known encodings: "idl", "msg", "srv", "action", "dynamic", "implicit". + # "dynamic" specifies a type created programmatically by a user, thus having no source. + # "implicit" specifies a type created automatically as a subtype of a + # complex type (service or action) - such as the request message for a service. + # Implicit types will have no contents, the full source will be available on the parent srv/action. + string encoding -.. code:: + # Dumped contents of the interface definition source file. + # If `encoding` is "dynamic" or "implicit", this field will be empty. + string raw_file_contents - FIELD_TYPE_NESTED_TYPE = 0 - FIELD_TYPE_INT = 1 - FIELD_TYPE_DOUBLE = 2 - # ... and so on - FIELD_TYPE_INT_ARRAY = ... - FIELD_TYPE_INT_BOUNDED_SEQUENCE = ... - FIELD_TYPE_INT_SEQUENCE = ... - # ... and so on - string field_name - uint8_t field_type - uint64_t field_array_size # Only for Arrays and Bounded Sequences - string nested_type_name # Only for FIELD_TYPE_NESTED_TYPE +And ``KeyValue`` is a simple: -These examples of the interfaces just give an idea of the structure but perhaps do not yet consider some other complications like field annotations or other as yet unconsidered features we want to support. +.. code:: -.. TODO:: + # Represents an arbitrary key-value pair for application-specific information. - (wjwwood) Add text about how to handle Service types, which are formed as a Request and a Response part, each of which is kind of like a Message. - We could just treat the Request and Response separately, or we could extend this scheme to include explicit support for Services. - Since Actions are composed of Topics and Services, it is less so impacted, but we could similarly consider officially supporting them in this scheme. + string key + string value -Versioning the ``TypeDescription`` Message Type -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Given that the type description message interface has to be generic enough to support anything described in the ROS interfaces, there will be a need to add or remove fields over time in the type description message itself. -This should be done in such a way that the fields are tick-tocked and deprecated properly, possibly by having explicitly named versions of this interface, e.g. ``TypeDescriptionV1`` and ``TypeDescriptionV2`` and so on. +This ``KeyValue[] extra_information`` field is provided as a catch-all for any application-specific or extension functionality to this Service, outside the scope of this REP. + Implementation in the ``rcl`` Layer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The implementation of the type description distribution feature will be made in the ``rcl`` layer as opposed to the ``rmw`` layer to take advantage of the abstraction away from the middleware and to allow for compatibility with the client libraries. -A hook will be added to ``rcl_node_init()`` to initialize the type description distribution service with the appropriate ``rcl_service_XXX()`` functions. +An API will be provided to initialize the type description distribution service with the appropriate ``rcl_service_XXX()`` functions. This hook should also keep a map of published and subscribed types which will be populated on each initialization of a publisher or subscription in the respective ``rcl_publisher_init()`` and ``rcl_subscription_init()`` function calls. -The passed ``rosidl_message_type_support_t`` in the init call can be used to obtain the relevant information, alongside any new methods added to support type version hashing. - -There will be an option to opt-out of creating this service, and a way to start and stop the service after node creation as well. - -.. TODO:: +The passed ``rosidl_XXX_type_support_t`` structs in these ``init`` calls can be used to obtain the relevant information, alongside any new methods added to support type hashing. - (wjwwood) A thought just occurred to me, which is that we could maybe have "lazy" Service Servers, which watch for any Service Clients to come up on their Service name and when they see it, they could start a Service Server. - The benefit of this is that when we're not using this feature (or other features like Parameter get/set Services), then we don't waste resources creating them, but if you know the node name and the corresponding Service name that it should have a server on, you could cause it to be activated by trying to use it. - The problem with this would be that you cannot browse the Service servers because they wouldn't advertise until requested, but for "well know service names" that might not be such a problem. - I guess the other problem would be there might be a slight delay the first time you go to use the service, but that might be a worthy trade-off too. - - -Type Description Distribution ------------------------------ - -``type_description_interfaces`` defines a service ``GetTypeDescription``, that will be provided as a builtin service on nodes. -The service must be optional, but it will be a detail decided by client libraries whether it is enabled or disabled by default. +The service will not be started automatically, and must be initialized by client libraries according to their implementation of enabling the service. +The client library should then also call the provided API to shutdown the service on shutdown. +``rclcpp`` and ``rclpy`` will implement these calls and act as reference implementations of usage. Tooling -======= +------- The ros2 command line tools, and the APIs that support them, should be updated to provide access to the type version hash where ever the type name is currently available and the type version description on-demand as well. For example: @@ -318,19 +443,14 @@ For example: - ros2 interface show (TODO implement) -- ``ros2 interface`` should be extended with a way to calculate the version hash for a type -- ``ros2 topic info`` should include the type version hash used by each endpoint -- ``ros2 topic info --verbose`` should include the type description used by each endpoint -- a new command to compare the types used by two endpoints, e.g. ``ros2 topic compare `` +- ``ros2 interface`` should be extended with a way to print the hash for an interface type +- ``ros2 topic info --versbose`` should include the type version hash used by each endpoint +- ``ros2 node`` should be extended with a way to print exact definitions for its used types, if its type description service is available - ``ros2 service ...`` commands should also be extended in this way - ``ros2 action ...`` commands should also be extended in this way -Again this list should not be considered prescriptive or exhaustive, but gives an idea of what should change. +Again this list should not be considered prescriptive or exhaustive, but gives an idea of recommended tooling. -Backwards Compatibility -======================= - -TODO Rationale ========= @@ -340,8 +460,50 @@ This section layse out further reasoning for why certain conclusions were reache Type Hashing ------------ -TODO - RIHS hashing algorithms (sha1, md5) -TODO - RIHS string representations (eg base64 (simplicity in conversion, human readability), pure binary (null terminators)) + +RIHS Type Description Serialization +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A serialization method had to be chosen to guarantee consistent hashing across implementations and machines. + +One alternative could have been the CDR serialization provided by DDS middlewares, but this was rejected because + +- few implementations exist for diverse languages such as Python and Javascript, both of which environments are considered as users of the type hashing functionality +- build-time calculation was desired _before_ generating the code for specific RMW implementations + +YAML was rejected as less available, given that JSON is built in to the Python standard libaries and is first-class in Javascript. +The C implementation actually uses ``libyaml``, leaning on the fact that YAML is a superset of JSON, and uses an output configuration that outputs plain JSON. + +RIHS Hashing Algorithm +^^^^^^^^^^^^^^^^^^^^^^ + +MD5 and SHA1 were both considered as algorithms for the RIHS01 hashing function. + +Both of these algorithms have been deprecated in security use due to demonstrated attacks. +While type description hashing does not have security ramifications, the mere presence of those algorithms was mentioned in discussion to be a red flag in some organizations. +Therefore, to avoid any complaint, they were ruled out. +This does result in a more resource intensive hashing implementation, both for computation and for size. +The trade off was deemed worthwhile. + + +RIHS String Representation +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following alternative RIHS01 string representations were considered. + +Pure binary. +Don't use any character set, just send a string of bytes, ``RIHS01_<32 bytes of data>``. +This would have a fixed size of 7 + 32 = 39 bytes, saving space over the 71 chosen. +However, these would not be human readable and would require special handling to print and parse. +The existing ``USER_DATA`` implementations are null-terminated, and type hashes can contain zero-bytes, breaking existing string reading. + +Base64. +This would take 7 + 44 = 53 bytes, also saving space. +However, type hashes are represented in memory as bytes, so the conversion does incurr a computation cost. +This representation is also less readable. + +While it didn't make much of a difference, the simpler and more human-readable implementation of a plain hex string was chosen. +It was decided that the extra couple dozen bytes was worth the tradeoff. Type Description Distribution ----------------------------- @@ -349,9 +511,12 @@ Type Description Distribution Using a Single ROS Service per Node ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The node that is publishing the data must already have access to the correct type description, at the correct version, in order to publish it, and therefore it is natural to get the data from that node. +ROS 1 contained full type descriptions in its connection headers, but this decision was a result of the central Master node design. +With the ROS 2 fully-distributed design, it was decided that the full description would add too much network overhead to system startup, given that most cases will not need the data. + +A node that is publishing the data must already have access to the correct type description, at the correct version, in order to publish it, and therefore it is natural to get the data from that node. Similarly, a subscribing node also knows what type they are wanting to receive, both in name and version, and therefore it is again natural to get that information from the subscribing node. -The type description for a given type, at a given version, could have been retrieved from other places, e.g. a centralized database, but the other alternatives considered would have had to take care to ensure that it had the right version of the message, which is not the case for the node publishing the data. +The type description for a given type, at a given version, could have been retrieved from other places, e.g. a centralized database, but the other alternatives considered would have had to take care to ensure that it had the right version of the message, whereas the node doing the communicating definitely knows the types it is using. Because the interface for getting a type description is generic, it is not necessary to have this interface on a per entity, i.e. publisher, subscription, etc, basis, but instead to offer the ROS Service on a per node basis to reduce the number of ROS Services. Therefore, the specification dictates that the type description is distributed by single ROS Service for each individual node. @@ -359,18 +524,15 @@ Therefore, the specification dictates that the type description is distributed b There were also multiple alternatives for how to get this information from each node, but the use of a single ROS Service was selected because the task of requesting the type description from a node is well suited to a request-response style ROS Service. Some of the alternatives offered other benefits, but using a ROS Service introduced the fewest dependencies, feature-wise, while accomplishing the task. -.. TODO:: - - cite the above, https://en.wikipedia.org/wiki/Request%E2%80%93response -Combining the Raw and Parsed Type Description in the Service Response -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Combining the Parsed Type Description and Raw Sources in the Service Response +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The contents of the "get type description" service response should include information that supports both aforementioned use cases (i.e. tools and Run-Time Interface Reflection). +The contents of the ``GetTypeDescription`` service response should include information that supports mentioned use cases (i.e. developer tooling and Run-Time Interface Reflection). These use cases have orthogonal interests, with the former requiring human-readable descriptions, and the latter preferring machine-readable descriptions. Furthermore, the type description should be useful even across middlewares and serialization libraries and that makes it especially important to send at least the original inputs to the "type support pipeline" (i.e. the process of taking user-defined types and generating all supporting code). -In this case, because the "type support pipeline" is a lossy process, there is a need to ensure that enough information is sent to completely reproduce the original definition of the type, and therefore it makes sense to just send the original ``idl`` or ``msg`` file. +In this case, because the "type support pipeline" is a lossy process, there is a need to ensure that enough information is sent to completely reproduce the original definition of the type, and therefore it makes sense to send the original ``idl`` or ``msg`` file. At the same time, it is useful to send information with the original description that makes it easier to process data at the receiving end, as it is often not trivial to get to the "parsed" version of the type description from the original text description. @@ -537,45 +699,29 @@ Some alternatives to using an array of type descriptions to store referenced typ - Rejected because needing to collate all of the referenced types on the receiver end introduces additional implementation complexity, and also increases network bandwidth with all the separate calls that must be made. +Backwards Compatibility +======================= -Development Progress -==================== - -TODO - remove this section for final merge - -- Type Descriptions: - - - [x] Define and place the TypeDescription.msg Message type in a package - - [ ] Create library to calculate version hash from parsed types - -- Type Description Distribution: - - - [ ] Add topic type hash to the "graph" API - - [ ] Define and place the GetTypeDescription.srv Service type in a package - - [ ] Implement the "get type description" service in the rcl layer - - [ ] Implement delivery of type version hash during discovery: - - - [ ] rmw_fastrtps_cpp - - [ ] rmw_cyclonedds_cpp - - [ ] rmw_connextdds +Discovery +--------- -- Tooling +The recommended DDS implementation for type hash discovery adds the type hash in a new previously-unused key, therefore it will be ignored by prior ROS distributions. +This leaves discovery unaffected for backwards compatibility. - - [ ] Create command-line tool to show the version hash of an interface - - [ ] Command-line tools for interacting with type version hashes - - [ ] Add a default warning to users when a topic has mismatched types (by type or version) +References +========== - - [ ] Integration with rosbag2 +.. [#rep2011] REP 2011: Evolving Message Types (final link TBD) + (https://github.com/ros-infrastructure/rep/pull/358) - - [ ] Record the Type Hash and Type Description into bags - - [ ] Provide a method to playback/update old bag files that do not have type information stored +.. [#interfaces] ROS Interfaces + (https://docs.ros.org/en/rolling/Concepts/Basic/About-Interfaces.html?highlight=interface) -References -========== +.. [#tdi_pkg] ``type_description_interfaces`` + (https://index.ros.org/p/type_description_interfaces/#rolling) -.. http://wiki.ros.org/Topics -.. REP 2011 Evolving message types (TODO link) -.. REP 20XX Dynamic pubsub (name TBD - TODO) +.. [#td_gen] ``rosidl_generator_type_description`` + (https://index.ros.org/p/rosidl_generator_type_description/#rolling) Copyright ========= From 3fe3a5652b6ec690c32a762f4c59f9d06153d6a8 Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Tue, 11 Jul 2023 02:35:07 -0700 Subject: [PATCH 08/10] rm dangling lines Signed-off-by: Emerson Knapp --- rep-2016.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/rep-2016.rst b/rep-2016.rst index 2955c3e0..0f353175 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -439,10 +439,6 @@ Tooling The ros2 command line tools, and the APIs that support them, should be updated to provide access to the type version hash where ever the type name is currently available and the type version description on-demand as well. For example: -- ros2 topic info -v (done - TODO more info) -- ros2 interface show (TODO implement) - - - ``ros2 interface`` should be extended with a way to print the hash for an interface type - ``ros2 topic info --versbose`` should include the type version hash used by each endpoint - ``ros2 node`` should be extended with a way to print exact definitions for its used types, if its type description service is available From b8287f25b451e6a4caeff3b1c5588dd3523c66d0 Mon Sep 17 00:00:00 2001 From: Emerson Knapp Date: Tue, 22 Aug 2023 08:40:22 -0700 Subject: [PATCH 09/10] Make Draft status for CI Signed-off-by: Emerson Knapp --- rep-2016.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rep-2016.rst b/rep-2016.rst index 0f353175..8733f0c9 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -1,7 +1,7 @@ REP: 2016 Title: ROS 2 Interface Type Descriptions - Representation, Hashing, Discovery and Distribution Author: Emerson Knapp -Status: Active +Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Jun-2023 From 00bfd825b76448bf1a5a79fbff428f8c773f0b7e Mon Sep 17 00:00:00 2001 From: Emerson Knapp <537409+emersonknapp@users.noreply.github.com> Date: Thu, 21 Sep 2023 16:24:25 -0700 Subject: [PATCH 10/10] Apply typo corrections and formatting from suggestions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Mirko Ferrati Co-authored-by: Chris Lalancette Co-authored-by: Tomoya Fujita Co-authored-by: Francisco Martín Rico --- rep-2016.rst | 48 +++++++++++++++++++++++------------------------- 1 file changed, 23 insertions(+), 25 deletions(-) diff --git a/rep-2016.rst b/rep-2016.rst index 8733f0c9..93a6465d 100644 --- a/rep-2016.rst +++ b/rep-2016.rst @@ -14,7 +14,7 @@ Abstract This REP proposes a standard format for communicating the structure of ROS 2 interface types, as well as tooling to support the use of these descriptions. It provides a ROS 2 interface type that can contain descriptions of types, runtime data structures, and a universally reproducible serialization method. -Alongside the description format, a standard hashing algorithm is also defined to create compact reperentations of type descriptions, to quickly transmit and detect type mismatch during discovery. +Alongside the description format, a standard hashing algorithm is also defined to create compact representations of type descriptions, to quickly transmit and detect type mismatch during discovery. Motivation @@ -29,7 +29,7 @@ This REP proposes to match the goal of those features with a set of standards to One part of this motivation is for correctness of existing applications. When a message type is updated, a different communicating node may begin to receive messages it does not actually understand. A second part of the motivation is to expand capabilities of the ROS 2 system to work with descriptions of types programmatically. -This REP will not go into detail on those supported use cases, they are discussed in more detail in REP-2011\ [#rep2011]_. +This REP will not go into detail on those supported use cases, they are discussed in more detail in REP-2011 [#rep2011]_. Terminology @@ -37,9 +37,9 @@ Terminology Some terms used in this document, which will be described in much more detail in the specification: -- **ROS 2 Interface** or **Interface Type** - a ROS 2 message, service, or action, as described in \ [#interfaces]_. +- **ROS 2 Interface** or **Interface Type** - a ROS 2 message, service, or action, as described in [#interfaces]_. - **Type Source** - the original text provided by author to define an interface type, such as a ``.msg`` or ``.idl`` file. -- **Type Description** - a data structure representating a parsed type source, which will be equal regardless of source format such as ``.msg`` or ``.idl`` if the described type is the same. +- **Type Description** - a data structure representing a parsed type source, which will be equal regardless of source format such as ``.msg`` or ``.idl`` if the described type is the same. - **Type Hash** - a compact identifier for a type description that is expected to be unique - **RIHS (ROS Interface Hashing Standard)** - a versioned specification for producing a hash value for a type description @@ -55,8 +55,8 @@ The Type Hash specification defines a Type Description serialization and hashing Type Description ---------------- -A Type Description must be produced such that it is stable across consistent meaning, even with trivial changes to source text. -The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the TypeDescription object as an intermediate representation. +A Type Description must be produced such that it is stable across consistent meaning, even with trivial changes to Type Source. +The interface description source provided by the user, which may be a ``.msg``, ``.idl`` or other file type, is parsed into the Type Description object as an intermediate representation. This way, types coming from two sources that have the same stated name and have the same information will be given the same description, even if they are defined using sources in differing specification languages. Thus the representation includes: @@ -66,7 +66,7 @@ Thus the representation includes: - a list of all recursively referenced types - no comments -Note that the reference implementation is provided in the package ``type_description_interfaces``\ [#tdi_pkg]_. +Note that the reference implementation is provided in the package ``type_description_interfaces`` [#tdi_pkg]_. The structure is discussed here in concept, but the implementation is the source of exact truth for ROS releases. The ``TypeDescription`` message type looks like: @@ -108,7 +108,7 @@ Describing Services and Actions ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ROS 2 Services and Actions are considered "aggregate interfaces", in that they compose several "sub-interfaces" into a more complex type. -A Service is defined using two Messages - a Request and a Reponse. +A Service is defined using two Messages - a Request and a Response. An Action is defined using three Messages - a Goal, Feedback, and a Result. These definitions from the user are then expanded into the more derived utilized by communications. Because these full structures affect interoperability, the full description as used by ROS 2 should be represented to accurately describe the types, rather than just the inputs. @@ -118,7 +118,7 @@ As of ROS 2 Iron, Services contain - a Request message as defined in input - a Response message as defined in input -- an Event message, as defined in REP-2012 Service Introspection\ [#rep2012]_. +- an Event message, as defined in REP-2012 Service Introspection [#rep2012]_. Therefore the implicit intermediary output looks like: @@ -154,7 +154,7 @@ A message defined explicitly with those above field types and names would have a However, this case is fairly contrived and in practice an application will already know what interface category it is trying to interact with. Given this approach, the ``type_description_interfaces`` provides no special accommodation for Services and Actions, leaving the transform into a representable type to code generation. -The reference implementation for this derivation is in ``rosidl_generator_type_description``\ [#td_gen]_. +The reference implementation for this derivation is in ``rosidl_generator_type_description`` [#td_gen]_. Versioning the ``TypeDescription`` Message Type ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -218,7 +218,7 @@ The JSON serialization must pay attention to certain aspects that even a JSON Sc - Whitespace management, including newlines and spaces between commas, colons, etc - Sorting ``referenced_type_descriptions`` in a consistent manner (probably alphabetically by type name) -However, to handle everything else, the reference implementation does provide a ``jsonschema`` file that other implementatations can use. +However, to handle everything else, the reference implementation does provide a ``jsonschema`` file that other implementations can use. Hashing Method ^^^^^^^^^^^^^^ @@ -234,13 +234,10 @@ This versioning allows the tooling to know if a hash mismatch is due to a change In the case of a new RIHS version, meaning a change in computation method, it will be unknown whether the interface types are equal or not without full description comparison. For now, the list of field names and their types are the only contributing factors, but in the future that could change, depending on which "annotations" are supported in ``.idl`` files. -The "IDL - Interface Definition and Language Mapping" design document\ [2]_ describes which features of the OMG IDL standard are supported by ROS 2. +The "IDL - Interface Definition and Language Mapping" design document [#interface_definition]_ describes which features of the OMG IDL standard are supported by ROS 2. If that is extended in the future, then this data structure may need to be updated, and if so the "ROS IDL Hashing Standard" version will also need to be incremented. New sanitizing may be needed on the TypeDescription pre-hash procedure, in the case of these new features. -.. TODO:: - - Re-audit the supported features from OMG IDL according to the referenced design document, including the @key annotation and how it may impact this for the reference implementation. Notes: @@ -286,6 +283,7 @@ While ``USER_DATA`` accepts arbitrary binary data, ROS 2 implementations so far Keys in this scheme are purely alphanumeric, so the recommended key is ``typehash``. .. code:: + typehash=RIHS01_XXXXXXXX; While this usage can be implementation-specific, consistency will allow for communication across DDS-RMW implementations. @@ -325,12 +323,6 @@ The type of this service is defined in ``type_description_interfaces`` as ``GetT The service must be optional, but it will be a detail decided by each client library whether it is enabled or disabled by default. This REP recommends that the default client libraries ``rclcpp`` and ``rclpy`` enable the service by a boolean Parameter named ``start_type_description_service``. -.. TODO:: - - How can we detect when a remote node is not offering this service? - It's difficult to differentiate between "the Service has not been created yet, but will be" and "the Service will never be created". - In the recommended approach, it is enabled via a parameter, but what if remote access to Parameters (another optional Service) is disabled? - Perhaps we need a "services offered" list which is part of the Node metadata, which is sent for each node in the rmw implementation, but that's out of scope for this REP. A service request to this ROS Service will comprise of the type name and the type hash, which is distributed during discovery of endpoints and will be accessible through the ROS Graph API, as described previously. The ROS Service server will respond with the type description and optionally the original raw sources used to define the type. @@ -440,7 +432,7 @@ The ros2 command line tools, and the APIs that support them, should be updated t For example: - ``ros2 interface`` should be extended with a way to print the hash for an interface type -- ``ros2 topic info --versbose`` should include the type version hash used by each endpoint +- ``ros2 topic info --verbose`` should include the type version hash used by each endpoint - ``ros2 node`` should be extended with a way to print exact definitions for its used types, if its type description service is available - ``ros2 service ...`` commands should also be extended in this way - ``ros2 action ...`` commands should also be extended in this way @@ -452,7 +444,7 @@ Rationale ========= The above document lays out the specific recommendations of this REP, as reached after discussion and iteration. -This section layse out further reasoning for why certain conclusions were reached, including alternatives that were considered. +This section lays out further reasoning for why certain conclusions were reached, including alternatives that were considered. Type Hashing ------------ @@ -467,7 +459,7 @@ One alternative could have been the CDR serialization provided by DDS middleware - few implementations exist for diverse languages such as Python and Javascript, both of which environments are considered as users of the type hashing functionality - build-time calculation was desired _before_ generating the code for specific RMW implementations -YAML was rejected as less available, given that JSON is built in to the Python standard libaries and is first-class in Javascript. +YAML was rejected as less available, given that JSON is built in to the Python standard libraries and is first-class in Javascript. The C implementation actually uses ``libyaml``, leaning on the fact that YAML is a superset of JSON, and uses an output configuration that outputs plain JSON. RIHS Hashing Algorithm @@ -495,7 +487,7 @@ The existing ``USER_DATA`` implementations are null-terminated, and type hashes Base64. This would take 7 + 44 = 53 bytes, also saving space. -However, type hashes are represented in memory as bytes, so the conversion does incurr a computation cost. +However, type hashes are represented in memory as bytes, so the conversion does incur a computation cost. This representation is also less readable. While it didn't make much of a difference, the simpler and more human-readable implementation of a plain hex string was chosen. @@ -710,6 +702,9 @@ References .. [#rep2011] REP 2011: Evolving Message Types (final link TBD) (https://github.com/ros-infrastructure/rep/pull/358) +.. [#rep2012] REP 2012: Service introspection (final link TBD) + (https://github.com/ros-infrastructure/rep/pull/360) + .. [#interfaces] ROS Interfaces (https://docs.ros.org/en/rolling/Concepts/Basic/About-Interfaces.html?highlight=interface) @@ -719,6 +714,9 @@ References .. [#td_gen] ``rosidl_generator_type_description`` (https://index.ros.org/p/rosidl_generator_type_description/#rolling) +.. [#interface_definition] IDL Interface Definition + (http://design.ros2.org/articles/idl_interface_definition.html) + Copyright =========