Multiple Partitions in Custom Profile

[jwelling2]

I haven't found any extensive examples in the documentation as to how partitions are set up within a custom profile. Some information is available at the following link:
https://github.com/eclipse-cyclonedds/cyclonedds/blob/master/docs/manual/options.md#cycloneddsdomainpartitioning
But I haven't found it very useful. Can a more extensive explanation of the partitioning syntax be provided?

Specifically, we have many nodes being launched for ROS 2 and very few of them actually need to send data across our wireless network. Setting up a profile to ignore partitions corresponding to (groups of) ROS 2 topics is needed.

[eboasson]

Hi @jwelling2, with apologies for responding with a bit of a delay and TMI in compensation 😁: here is an example that maps any topic named chatter-something to a separate multicast address, maps all service requests to another and all service responses to a third. It also ignores the rosout topic:

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
  <Domain id="any">
    <Tracing>
      <Verbosity>fine</Verbosity>
      <OutputFile>cdds.log.${CYCLONEDDS_PID}</OutputFile>
    </Tracing>
    <Partitioning>
      <NetworkPartitions>
        <NetworkPartition Address="239.255.0.3" Name="data"/>
        <NetworkPartition Address="239.255.0.4" Name="requests"/>
        <NetworkPartition Address="239.255.0.5" Name="replies"/>
      </NetworkPartitions>
      <PartitionMappings>
        <PartitionMapping DCPSPartitionTopic=".*/chatter*" NetworkPartition="data"/>
        <PartitionMapping DCPSPartitionTopic=".rq/*" NetworkPartition="requests"/>
        <PartitionMapping DCPSPartitionTopic=".rr/*" NetworkPartition="replies"/>
      </PartitionMappings>
      <IgnoredPartitions>
        <IgnoredPartition DCPSPartitionTopic=".rt/rosout"/>
      </IgnoredPartitions>
    </Partitioning>
  </Domain>
</CycloneDDS>

Of course these particular choices are total nonsense, but it might serve as an example and/or a starting point. There are a few things worth noting:

• The DCPSPartitionTopic attribute is an expression (* and ? are wildcards) that needs to match the partition and topic, joined with a .. So .*/chatter* matches a topic names containing /chatter published or subscribed-to in a partition "" (empty string), which is the default partition.
• ROS 2 doesn't make use of DDS partitions and always uses this default partition. In practice, for ROS 2 (at least in its current form), that means a * or an empty string before the . has the same result.
• All ROS 2 data topics are prefixed with rt followed by the namespace (e.g. rt/turtle1/cmd_vel), all services use rq and rr instead of rt.
• If a subscription (ROS 2 terminology) a.k.a.reader (DDS terminology) matches a partition mapping, it will advertise the multicast address specified in the associated "network partition" instead of the default multicast address (if multicast is enabled). This can be very helpful if a subset of the machines is subscribed to a high bandwidth flow.
• Beware of Linux loopback interfaces: they support multicast but often claim not to (the MULTICAST flag in ifconfig); and Cyclone will be default disable multicast for everything except initial discovery if it is directly attached to a WiFi network. Both can be overridden, the first with ifconfig lo MULTICAST or by adding <Internal><AssumeMulticastCapable>lo</AssumeMulticastCapable></Internal> to the Cyclone configuration.
• If a publisher/subscription/writer/reader matches an ignored partition, it will communicate locally within the process, but it will not be advertised on the wire and not match any discovered remote entity. This doesn't currently distinguish between within-a-host and outside-the-host (that's somewhat tricky and very much not DDS-like behaviour, but whatever). I think the "ignored partitions" in their current form are too restrictive in most cases, including yours, as it will keep the data inside the process.

In your case, or at least in my interpretation of it, it all depends on whether the nodes "across the wireless network" are actually subscribed to the data. If they are subscribed but should not be receiving it — that seems unlikely — I'd say the proper solution would be to use ROS 2 namespaces.

If there is multicast involved and those "a. t. w. n." are not actually subscribed to the data, they may still be receiving the multicasts "by accident": the default multicast address happens to be the same as the one used for discovery, and so the presence of multiple subscribing processes on one end of the wireless network might cause all data to end up on the wireless network. In that case, mapping the data to a different multicast address should take care of it, provided your switches do IGMP snooping. (If the switches do not do that, you could firewall the data. Once you have a unique multicast address, that is pretty straightforward.)

Since not-so-long (for ROS 2, that means Galactic patch release 1) and still a bit rough around the edges and therefore not something I generally advertise, there is also the possibility of specifying both the loopback interface and the real network interface in the General/NetworkInterfaceAddress setting (e.g., <General><NetworkInterfaceAddress>lo,enp0s5</NetworkInterfaceAddress></General>). If you then have a multicast-capable loopback interface, it will take the possibility of a multicast on the loopback interface into consideration.

I don't know the details of your problem and hence I don't know for certain whether the above is enough to solve your problem. Clearly the current (and truly ancient) mechanism for network partitioning is a bit limited. If it doesn't suffice, I'd be happy to use your case as a source of inspiration for improving it. If you don't want to discuss the details in a public forum but are ok with sharing them over, e.g., email, that's fine with me.

Another option worth looking into is adding Eclipse Zenoh to the mix, like some at the Indy Autonomous Challenge are doing.

Finally, note that while considering your question I found a few issues. The "ignored partition" setting was broken at some point in the past, and the order of the network mappings was incorrect. #885 addresses these issues, but is not required as long as you don't require "ignored partitions".

[jwelling2]

This is great! Thanks for the information. I'll need to some time to go through all this and try it out with my use case, but I'll get back to you soon.

[jwelling2]

From your answer, it sounds like there is still some work to be done on the partitioning system, and there isn't a built-in way to disable discovery traffic from topics using Cyclone DDS. Using different multicast addresses and partition mappings didn't change the discovery traffic/heartbeats we saw, which is the problem we're dealing with currently (bandwidth constraints with many topics). We decided to try out domain ID forwarding with the ROS 2 Integration Service, and that's currently getting developed and showing promise.

We would prefer to not introduce a system like Eclipse Zenoh.

In the meantime, is there any chance of repairing the "ignored partitions"? If I'm understanding correctly, that would disable discovery traffic for topics specified to a certain partition in ROS 2.

[eboasson]

@jwelling2 with apologies for the delay,

From your answer, it sounds like there is still some work to be done on the partitioning system, and there isn't a built-in way to disable discovery traffic from topics using Cyclone DDS. Using different multicast addresses and partition mappings didn't change the discovery traffic/heartbeats we saw, which is the problem we're dealing with currently (bandwidth constraints with many topics).

I see — I had interpreted your question as concerning application data, and then those settings will work fine, but as these settings depend on the discovery data getting exchanged, they cannot help with a problem caused by the discovery data itself.

For the discovery traffic the only current option is to completely eliminate some using the "ignored partitions" feature matching topic names with patterns. There are also some ways of controlling which processes will perform discovery with which other processes without falling back to unicast discovery, but once processes have discovered each other, there'll be a full discovery of the endpoints. I don't if you'd be interested in that, but I'd be happy to explain in detail if it is (or could be) of use.

We decided to try out domain ID forwarding with the ROS 2 Integration Service, and that's currently getting developed and showing promise.

Happy to hear that.

We would prefer to not introduce a system like Eclipse Zenoh.

Any reason in particular? Both the ROS 2 integration service or Zenoh should do the trick but of course I'll try to push Cyclone's sister project 😀

In the meantime, is there any chance of repairing the "ignored partitions"? If I'm understanding correctly, that would disable discovery traffic for topics specified to a certain partition in ROS 2.

As mentioned, fixed in current master. If you have no issue doing a source build you'll be able to use it.

[thijsmie]

Answered in great deal, missing/broken features mentioned here have long since been fixed, merged and released.