adding article about topic and service names and how they relate to DDS

[wjwwood]

This is an initial draft of an article about the ROS 2 topic and service names.

Things I need to do still:

• talk about capital letter alternative
• summarize differences from ROS 1

Some points of interest or RFC:

• Stuff about URL's
• Changes in the way the ~ is interpreted
• Is a single _ a valid namespace? Like foo/_/bar?
  • It is not, because /foo/_/_bar is ambiguous. It is a special case of "tokens must not end with _".

Some other points that require deeper discussion:

• Should services be content oriented like topics or service oriented like websites?
  • Should it be rosservice://node.name/service/name/space or just rosservice:///service/name/space?

[wjwwood]

I added a section for the capital letter alternative and fixed style issues. I'm going to wait on the section about differences from ROS 1 until I get feedback on the more questionable bits like the ~ expansion differences.

[dirk-thomas]

The paragraph above refers to this as rp_ (which also collides with parameters from this table). I would propose rr_ for the response / result. That leaves q for the query, s for the native service, and p for parameters`.

[wjwwood]

Sounds good to me: 2e8628c

[dirk-thomas]

I certainly wouldn't agree with the rational. I think the increased parser complexity in potentially N implementations is pretty severe compared to the benefit of 1 extra character.

[wjwwood]

Well then we'll have to vote on it, because I disagree. We don't have any slack in the limitation on the name length, so any extra work we can do, which doesn't also restrict the user, to maximize the available characters is worth it in my opinion. This isn't a complex rule and in all likelihood would be a single line in any language where it is implemented.

[dirk-thomas]

If length is so crucial you might not want to use a three letter prefix in the first place. A single character could encode all the subsystem. For readability two would be reasonable, e.g. just rX. There is no need for the underscore to separate.

[wjwwood]

I agree, but in that case I think at least one underscore helps because it is less likely to share a common prefix with a non-ROS DDS topic, e.g. the sorting of ramy_action, radical_non_ros_topic, raweird_action versus ra_my_action, ra_weird_topic, radical_non_ros_topic. Also, startswith('rt') is more likely to overmatch more than startswith('rt_') in my opinion. However, in my opinion an additional _ doesn't help with readability or reduce the likelihood of prefix congestion considerably (it does help a little). Mostly a double underscore would help with consistency in my opinion.

Admittedly this is all pretty subjective, but I think there is a significant enough distinction to be made between rX and rX_ to justify one underscore and not enough of an advantage for rX__ over rX_ to justify a second. But we can vote on it as a group if we can't come to a consensus.

[tfoote]

I think that a separator is important. And since we already have a separator reusing it makes sense. It will greatly simplify the parsing process and visually it will also be consistent.

As much as I completely agree that the hard limit on the topic length in DDS is a big issue and we should work to minimize the overhead we use. I think that optimizing down to the character at this point is overkill. I would suggest that we reserve for ourselves several characters. Maybe 6 for the prefix. We will plan to use only 2 in our implementations. And assuming we are happy in the future we can decrease our reserved characters to the user. If we really wanted to preserve characters we'd want to use a single separator with an escape character. The requirement that there's not trailing _ and leading numbers could make most separators only single width. But the introspection and readability gets much worse.

[tfoote]

I'm not sure I agree because I can't imagine why we'd need more characters (but it's hard to prove a negative). Especially given that we can start using the second character as an op-code if we need more options and then we have around 30 unique values (a-z + 0-9 - the ones we've already reserved). That would be a hit to readability and obviousness, but it's also something I don't think we'll have to do.

Also, when considering reserving more characters, I think that we should assume that we'll never give the characters back, because I think it would be a painful process to do so. So much so that it would never be worth rolling out. It would basically be a wire level breaking change to change the prefix. I'm not arguing that we shouldn't reserve space because of this, but it's like a mail-in rebate to me: I just assume the price does not include the rebate and that I'm not going to get it back, but then if I do it's icing on top.

Indeed we shouldn't assume they'll come back , and we don't need to use them in the prefix. If we just enforce that we leave empty space at the end of the strings it's a forward compatible change. All old topics will still work. The value in this that to users if we allow them x elements in their names, it's a non breaking change to allow them x+3 elements in their names. It's a potentially crippling change if we change it to allow x-3 which is almost impossible to audit if there are dynamically generated topic names.

I agree that there are no strong use cases right now for going beyond the 32 characters. But there are some things we could consider in the future such as embedded ROS version semantics into the prefix such that you could differentiate versions of ROS running over DDS in parallel etc. Leaving ourselves some extra bits might be worth it.

If we really wanted to preserve characters we'd want to use a single separator with an escape character.
What does this mean?

For example:
If '0' is prefixed by '_' then '0' else '/'

• a_0b -> a0b
• a0_b -> a/_b
• a_00b -> a0/b
• a__0b -> a_0b
• a_____b -> a_____b
• a0000000 -> b invald empty namespaces
• a_0_0_0_0b -> a0000b

Or looking the other way.

• trailing 0: a0/b -> a_00b
• leading underscore: a/_b -> a0_b
• zero after underscore: a_0b -> a__0b
• zero before underscore: a0_b -> a_0_b

Overhead 1 character per namespace separator + 1 character per instance of 0 in the string.

This is a lower overhead than 2 characters per namespace separator assuming that there are more namespace separators than zeros. But i'm not advocating for this but saying that if we need more space.

We could pick any other character such as a capital letter which is theoretically invalid in the ROS space and decrease the number of collisions. We've been talking about allowing them since we haven't enforced it before.

The requirement that there's not trailing _ and leading numbers could make most separators only single width.
Can you give an example? I don't see how we could use those rules to implement a separator (I assume you mean the namespace delimiter).

Zero can never start a token and underscore can never end a token thus _0 can never be at the boundary. It limits where it could be but on expanding it out, it's not limited enough to avoid the need for an escape character. I think using the capitals would be a better optimization since they should appear less.

But again I think we should use the __ for clarity.

[gbiggs]

I agree with Tully that it may be worth having a few reserved characters even if they are not used now. It is easier to reduce the required prefix length than increase it.

I do not think that having rt_ instead of rt__ is too much of a chore. Although the hidden topic /_foo becomes rt__foo, which is awful, I'm assuming that users will not see it, and that the first step in processing a topic name will be lopping off the rt_ prefix leaving _foo anyway.

If the first step of parsing a DDS topic name to a tokenised ROS topic is to lop off and process the prefix, then having one underscore or two will not complicate the processing implementation.

The only "against" I can think of for a single underscore in the prefix is that there is a higher chance that it will collide with a DDS topic name, but that risk seems fairly low to me without any data on frequently-used DDS topic names.

[wjwwood]

I switched the prefix to be rX and removed the stipulation that the first namespace delimiter is dropped, so now /foo becomes rt__foo (rt + __ + foo): 70fb704

[wjwwood]

Instead of reserving characters, can we just say anything between rX and the first __ is reserved? That way parsers will be written to handle either no new characters or arbitrarily more? In general I'm still not convinced enough that we'll need them to justify preallocating characters to the cause.

[gbiggs]

The problem of saying "anything between rX and the first __" is that you run the risk of extending the prefix and suddenly invalidating some borderline topic names. If the reserved length is known in advance, then everyone knows where the limit lies.

[gbiggs]

The wording here could be confusing. '/foo' can be split into tokens, we just find that there are no additional tokens (parent names) between the root and the topic name, so it is in the root namespace.

[wjwwood]

d78250c

[jack-oquin]

proceeded -> followed

[wjwwood]

d78250c

[tkruse]

Possibly it could be useful to have a section dedicated to potential mappings between ROS1 and ROS2 names. Or to other 3rd-party middlewares.

[jack-oquin]

Why is there a repeated /foo/foo in the previous two examples?

[wjwwood]

I assume you mean the last column of the previous two lines?

The formulae is <name space> + <stuff before the ~> + <expanded ~> + <stuff after the ~>. In the case of line 158 (example 2), it's /foo + <nothing> + /foo/my_node + <nothing> which results in /foo/foo/my_node. For line 159 (example 2), it's /foo + <nothing> + /foo/my_node + /ping which results in /foo/foo/my_node/ping.

Basically the ~ expands to "namespace" + "node name" and is expanded in place. In these cases the namespace is represented twice, once as the normal namespace and once in the expanded ~. Does that make more sense?

In the following lines this doesn't happen because the input is absolute already and therefore not affected by the namespace implicitly.

[gbiggs]

I must admit that some of the examples in the table made my head hurt, too. But, when I thought about it, I decided that as painful as they are, there are probably minor use cases (such as auto-generated topic names for infrastructure and tooling) where expanding to "namespace + node name" within the node within the namespace, leading to odd-looking repitition, could be used.

[wjwwood]

I'm not necessarily in favor of this level of flexibility, but I suggested it because the current restrictions on how to use ~ in ROS 1 seemed arbitrarily constraining. Having the character is nice though because it makes remapping slightly easier.

[jack-oquin]

I see. That seems to violate the design principle of "least surprise".

The only useful ROS1 semantic I know of for the ~ prefix is at the beginning of the name. I can't think of any valid reason to expand the node's namespace and name in the middle of some string of tokens.

Unless there is a valid use case, I recommend only expanding ~ at the beginning of a name (as the first token). For compatibility, it should probably not be preceded by a /, but I would accept that too, as long as it's optional.

[brunohexsel]

I like the idea of having a subsitution token such as ${ns} instead of "~". However I worry about the following case:

# Somewhere this is defined:
export ns="some_value"

# The user now tries to manually instantiate a node passing the topic as parameter:
my_node my_parameter_with_namespace:=${ns}/some_topic
# bash would interpret it as:
my_node my_parameter_with_namespace:=some_value/some_topic
# but this could be avoided with:
my_node 'my_parameter_with_namespace:=${ns}/some_topic'
# which is a little cumbersome and not everyone knows about it.

I would go with what @wjwwood suggested with "{ns}" or "%{}" that would not be a source of very obscure and hard to debug errors.

[gbiggs]

I vote for treating ~ the same way as Bash, except for ~foo. While it would be nifty to say "the namespace of node foo", it would be ambiguous if there are many foos in different name spaces.

I also vote for {} with no other options. As William said, this is not Bash so we do not need to be exactly like them. I do have one concern, though: It might need escaping in Python strings when .format() is used, which would mean that every instance of {...} becomes {{...}}.

As potentially usable substitutions, {ns} and {node} sound like they will be frequently useful.

[wjwwood]

Sorry for letting the ball drop on this. Let me try to recap:

• Only allow ~ at the beginning of names
• ~ must be followed by /
  • uses of ~foo (which is allowed in ROS 1) can always be implicitly upgraded to ~/foo, but in ROS 2 we can additionally choose to disallow that or warn about it
• use {} as a generic substitution escape sequence
  • actual substitutions, e.g. {ns} or {node}, to be determined later

I reiterate the proposal that we use {} as the substitution syntax because of the argument that ${} would collide with expressing the substitutions within shell scripts which use the ${} already. Did @brunohexsel's argument about bash collisions sway any of the ${} proponents (@codebot or @mikaelarguedas)?

Anyone else want to weigh in with an opinion?

[mikaelarguedas]

Fine by me to use {} rather than ${} to avoid collision in shell scripts 👍

[codebot]

Thanks everyone for the excellent points; I hereby rescind my preference for ${foo} and henceforth prefer {foo}. Thanks @brunohexsel for the concise and helpful example. 👍

[wjwwood]

Thanks for the feedback everyone. I addressed a lot of the inline comments and changed the prefixing to be /foo -> rt__foo rather than /foo -> rt_foo.

---

Possibly it could be useful to have a section dedicated to potential mappings between ROS1 and ROS2 names. Or to other 3rd-party middlewares.

I plan to make a section that summarizes the differences between ROS 1 and ROS 2 after resolving the discussion around the ~, since it will likely change a lot depending on that. I'm not sure what other middleware I'd additionally compare it to. Do you have a suggestion? ZeroMQ? YARP?

[wjwwood]

Do you have a suggestion? ZeroMQ? YARP?

Well on second thought ZeroMQ isn't a great thing to compare to, since it's not an "anonymous" or "by name" pub/sub. You have to specify the transport, something like tcp://<ip>:port.

I can't find YARP's guidelines on this (maybe it's not explicitly documented), but based on their tutorials I'd guess it's really similar to ROS's. Does anyone with more experience know what YARP's port name rules are?

I also looked for LCM's, but other than seeing them drop the topic name into a regex with the equivalent of sscanf('^%s$', channel_name) (https://github.com/lcm-proj/lcm/blob/master/lcm/lcm.c#L323), I haven't seen any other validation and I can't find anything in their documentation. So if anyone has a lead on that one, I'll happily add it too.

I looked into the port naming conventions of the orocos family of projects (orocos, rock, and orca), but also came up short. If anyone can link to or recite their name rules I'll add them too.

At this point I'm thinking there's something wrong with me, maybe it's the wine 🍷. If anyone wants to contribute name guidelines from other middlewares or robot kits I'll gladly add them.

[dhood]

if we think the current expression seems too contrived, we can also refer to the name length limit as C+N+P <= 255 where C is the number of characters in the fully-qualified name and N is the number of name tokens. not essential.

[wjwwood]

It would still need to be C+(N*2)+P <= 255 right?

[dhood]

not if you let C refer to any character, as opposed to any non-/ character

[wjwwood]

Ah, that's probably simpler, do you want to reword it or would you prefer I do it?

[dhood]

I'll do it

[dirk-thomas]

What is the difference between a relative topic (without a leading slash) and a topic starting with a tilde? Do we need the special semantic of the tilde?

[wjwwood]

Put another way, if the tilde is used then the tilde may only be at the front of the name. It doesn't have anything to do with absolute versus relative names in my opinion.

[dirk-thomas]

Maybe you can clarify the difference based on an example. How are the following two topics resolved to fully qualified names in a node with the namespace /ns:

• foo
• ~/foo

[wjwwood]

There are examples just as you've described further into the document. This bullet list is meant to be a quick reference of all the rules.

[dirk-thomas]

I guess I don't see a significant benefit in have two separate concepts (the node name as well as a namespace) which together form the prefix of the nested topics.

The complexity of the table in "Private Namespace Substitution Character" could be reduced by only having a single concept. The namespace of a node is by default equal to the name of the node. And when you specify a custom namespace you are free to choose something completely different or use something which contains the node name. With that you only have to distinguish between absolute topics and relative topics (no need for ~) and the node provides the namespace to convert a relative topic into an absolute one.

[wjwwood]

I think the point is emulate the behavior of ROS 1:

http://wiki.ros.org/Names#Resolving

You can imagine that you have two nodes, one publishes on foo and the other subscribes on foo, if you put them both in the namespace bar then they should still communicate. If the node name is always included in the namespace then they wouldn't talk because one would publish to /bar/node1/foo and the other would subscribe to /bar/node2/foo.

The ~ is specifically for when you want it to be node private, but in that case it should probably still include the namespace of the node in addition to the node name.

With your proposal I do not think it is possible to have a relative name resolve to the same name but absolute, i.e. foo -> /foo, because the namespace is never empty.

[wjwwood]

Sorry my previous comment wasn't very concise. I understand your proposal as:

Remove ~ and default the namespace to <node name>

If I interpret that correctly, then I think it no longer allows for simple relative names like foo to become simple absolute names like /foo. Is that right? Maybe that use case doesn't matter?
It also makes it harder to naturally map to ROS 1 where ~ is used.

[dirk-thomas]

My proposal is to unify the two concepts. As far as I can see the expressive power should be equivalent to what is possible with the two separate concepts.

You can any of the following:

• not set a custom ns: the name of the node is being used (e.g. node_name)
• you can set a custom ns: that name space is being used, that ns can be:
  • empty
  • any name (not related to the node name): e.g. foo or bar/baz
  • something nested the node name: e.g. node_name/foo (to make this easier to specify without retyping the node name the namespace of a node could use a substitution like {nodeName}/bar

[wjwwood]

I wasn't considering a difference between not set and empty for the namespace. I think what you've proposed is fine if the "empty" namespace is used, but I don't think it significantly simplifies the rules because now you have to know that an unset namespace implicitly implies node name. With the ~, you may not know right off what it does, but it is more explicit that something is happening and you can look it up. To me it is less surprising when foo becomes /foo as opposed to /node_name/foo.

I'm the last person to choose an alternative solely for ROS 1 compatibility, but in this case I think the benefit of getting rid of ~ doesn't outweigh the problems of converting to and from ROS 1 topic names easily. Perhaps that is simpler than I am imagining, can you come up with a set of rules that could be followed to go from one to the other?

[dirk-thomas]

With this proposal all topics are being prefixed before they are being passed to DDS. I think it would be valuable to allow users to e.g. subscribe to "native" (non ROS created) DDS topics using the ROS api. E.g. if someone wrote a DDS participant (without using ROS) which publishes on the DDS topic "image" there should be a way to subscribe to that topic using ROS (maybe requiring special flags).

[wjwwood]

@dirk-thomas that use case is covered in the document.

[dirk-thomas]

Are you referring to this line? https://github.com/ros2/design/pull/90/files#diff-11cdcb7792e2ef4c9f19a62673484497R367

Can you please reference the line you have in mind. I couldn't find it myself. Thanks.

[wjwwood]

https://github.com/ros2/design/pull/90/files#diff-11cdcb7792e2ef4c9f19a62673484497R378

[dirk-thomas]

If this rational is valid for the proposed approach should this line be moved to a different section? It looks to me like it was only relevant for the alternative approach where it is mentioned.

[wjwwood]

We can mention it in one of the sections other than the alternatives. I saw it as a prescription on how to address a specific application of the rules, so I didn't mention it directly. Which section do you think it belongs in, the "ROS Specific Name Prefix" section maybe?

[dirk-thomas]

Maybe mention the use case to be able to subscribe to any DDS topics name at the end of section "ROS to DDS Name Conversion Examples"? Something like:

In order to be able to interoperate with "native" topic names the API should provide a way to skip the ROS specific prefixing.

[wjwwood]

I added a section about the native topic names: d787ac2

[dirk-thomas]

Great, the new section is perfect.

[tkruse]

Regarding names I would suggest formalizing the rules, e.g. using EBNF. Here is a working prototype:
https://gist.github.com/tkruse/e07d9c5af6d3caad945f4715b130d407 (had to cheat for double underscores, multiple substitutions in same segment missing, trailing underscore in path segment not failing yet (I got bored) )

I assume the benefits are obvious to you.

[wjwwood]

An EBNF would be useful, but I won't have time to pick up where you left off right now. Does anyone else have the interest and time to finish it?

[wjwwood]

I'm going to go ahead and merge this since it is in a pretty decent place. I've ticketed the task to finish the EBNF: #96

Also, there is a chance that this will need to be changed once we address the topic of Service name addressing, as mentioned in the original post of this pr, but we can handle that when it comes up.

[wjwwood]

For anyone who was following this pr, but not the repository, I've proposed some changes to how substitutions are parsed: #127