RFC REP-155: Conventions, Topics, Interfaces for Perception in HRI

[severin-lemaignan]

Following announcement on discourse, I open a PR for a new REP 155, aiming at standardizing perception for human-robot interaction in ROS.

The REP is relatively large by ROS standards. It is based on the IROS paper 'ROS for Human-Robot Interaction' (non-paywalled version) with several significant updates based on experience learned during initial implementations.

PDF printouts of the REP:

• 13-Jul-2022: REP-155-20220713.pdf
• 25-Jan-2022, commit a1d50f3: rep-0155-a1d50f374fa2.pdf
• original proposal: REP-155-20220124.pdf

Feedback welcome!

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/new-rep-proposal-human-robot-interaction-in-ros-ros4hri/23776/6

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-news-for-the-week-of-1-21-2022/23940/1

[severin-lemaignan]

Suggestions by Andres Ramirez-Duque received by email on the 24-Jan-2022:

Add Head pose (geometry_msgs/Pose Message) also as a subtopic under faces: it can be useful in addition to the TF Face frame

In your idea, what would be the frame of reference for this pose?
Also if head pose duplicates the face TF frame, is it really needed? if it is mostly for convenience, it might be better to add it to the libhri API instead (libhri would then take care of subscribing to the right TF topics and expose a head pose)

Add Gaze vector as geometry_msgs/Pose Message or TF

Thanks for pointing that out! Initially, a 'gaze' TF frame was also specified, but for some reason, it seems it disappeared. I'll add it back. My idea was to expose a gaze_<face id> frame, collocated with the face_<face id> frame, but oriented toward the person's gaze.

If you think a pose is better, can you quickly explain why?

I'm curious because almost no message includes a timestamp

Yes. In earlier version, more messages were timestamped. But I eventually removed the headers, as I was unsure what the timestamp would add. Initially, the purpose of the timestamp was to be able to match eg a face detection to its source image (image with the same timestamp), but it is not a great (!) way of tracking the source, so I finally decided to remove them. Happy to reconsider, though!

[andres-ramirez-duque]

Thanks for posting my message,

Suggestions by Andres Ramirez-Duque received by email on the 24-Jan-2022:

Add Head pose (geometry_msgs/Pose Message) also as a subtopic under faces: it can be useful in addition to the TF Face frame

In your idea, what would be the frame of reference for this pose? Also if head pose duplicates the face TF frame, is it really needed? if it is mostly for convenience, it might be better to add it to the libhri API instead (libhri would then take care of subscribing to the right TF topics and expose a head pose)

The frame of reference could be /world or /local_sensor_frame and could be useful in applications that use multiple fields of view (onboard and global) or for applications that use multiple sensors (e.x. IMU) and to apply fusion algorithms before publishing the TF

Add Gaze vector as geometry_msgs/Pose Message or TF

Thanks for pointing that out! Initially, a 'gaze' TF frame was also specified, but for some reason, it seems it disappeared. I'll add it back. My idea was to expose a gaze_<face id> frame, collocated with the face_<face id> frame, but oriented toward the person's gaze.

Totally agree

If you think a pose is better, can you quickly explain why?

For the same reason as above, to apply fusion before publishing the TF

I'm curious because almost no message includes a timestamp

Yes. In earlier version, more messages were timestamped. But I eventually removed the headers, as I was unsure what the timestamp would add. Initially, the purpose of the timestamp was to be able to match eg a face detection to its source image (image with the same timestamp), but it is not a great (!) way of tracking the source, so I finally decided to remove them. Happy to reconsider, though!

A possible reason would be to monitor the quality of communication between the main topics (e.x. /humans/persons) when working on multiple workstations

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-news-for-the-week-of-january-23rd-2022/24073/1

[cst0]

I think it's worth erring on the side of 'make use of headers': time plays an important part of interaction, and it's better to have the ability to leverage that (even if some applications won't use it).

[severin-lemaignan]

I think it's worth erring on the side of 'make use of headers': time plays an important part of interaction, and it's better to have the ability to leverage that (even if some applications won't use it).

Yes... I guess I'm a bit reluctant to create a bunch of new messages that are simply standard messages + headers. For instance, I initially had a RegionOfInterestStamped defined as:

Header header
sensor_msgs/RegionOfInterest roi

Any thoughts, @wjwwood @tfoote or others?

But happy to add headers to the new types. Would you recommend suffixing them all with Stamped? (eg Skeleton2DStamped) or it is accepted/more readable to drop the Stamped suffix?

[wjwwood]

I don't have a strong preference, but my general guidance is this:

• only include a Header if you need both frame_id and a timestamp (this is the case most of the time, i.e. you need when AND where most of the time), otherwise consider if you actually need the timestamp in the message vs the time you received it
• only use the Stamped suffix if there is a use for the non-stamped counterpart (or you think there could be in the future)

I'm sorry I don't have time to dig into your proposal in more detail right now, but hopefully my recommendations help a bit.

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-2022-04-21/25293/1

[wjwwood]

One general comment, we prefer one sentence per line of text, and usually avoid manually wrapping lines by length, though that is allowed. Either way, new sentences should start on a new line. See:

http://docs.ros.org/en/rolling/The-ROS2-Project/Contributing/Code-Style-Language-Versions.html#markdown-restructured-text-docblocks

Though that doc is for ROS 2 specifically, the reasons behind the requested style still stand.

[wjwwood]

Just some random comments while reading.

[wjwwood]

I wonder if this should be a subtopic or instead should be part of one of the other messages.

[severin-lemaignan]

what other messages do you have in mind? (the rationale for using a subtopic here was to ensure that any node dealing with persons would transparently see 'anonymous' persons as well. But I guess this could be modeled differently...)

[severin-lemaignan]

One general comment, we prefer one sentence per line of text, and usually avoid manually wrapping lines by length, though that is allowed. Either way, new sentences should start on a new line. See:

http://docs.ros.org/en/rolling/The-ROS2-Project/Contributing/Code-Style-Language-Versions.html#markdown-restructured-text-docblocks

Though that doc is for ROS 2 specifically, the reasons behind the requested style still stand.

thanks, formatting fixed.

[clalancette]

I've left a number of fairly minor comments inline.

Overall, this is a well-written document. However, the concern with it as a REP is that it seems to describe a particular implementation of an HRI system. It's unclear whether this is a good generic representation of HRI systems in general, and we (Open Robotics) aren't really in a position to make that determination.

To move this forward, our recommendation is to socialize this REP proposal with others in the HRI community. We know you've started this with https://discourse.ros.org/t/new-rep-proposal-human-robot-interaction-in-ros-ros4hri/23776/6 and this REP, but there hasn't been a lot of feedback to either. Encourage others in the HRI community to comment on, approve, or otherwise interact with this REP so we can gauge its wider applicability. If we get enough engagement from the wider HRI community, then we can move forward here (including eventually calling for a vote on it).

[clalancette]

Suggested change

	.. Note:: The reverse operation (spliting a person into two) can be realised
	.. Note:: The reverse operation (splitting a person into two) can be realised

[guyhoffman]

Hi - I am not very familiar with the ROS standardization process, but I went over the document linked here: REP-155-20220713.pdf and I do not see what @clalancette sees, namely that this is a particular HRI implementation. These seem to be useful and extremely generic conventions for any person-interactive robotic system.

On a more general note, I think that having conventions and data structures that HRI ROS folks agree on would greatly enhance adoption of standard libraries and tools that could save HRI researchers time and accelerate research.

[cst0]

In fairness to clancette and the OSRF folks, it is a particular HRI implementation (here)... the question is "is this a general enough toolset that should be made standard across all of ROS, or is it a standalone tool"? Generally standards like this would be more "here's what a message should look like", this standard is focused on what functionality will be available to a system: it's more of an attempt at standardizing an architecture.

Personally, while I do feel that it describes a specific implementation, I also feel that that implementation prioritizes being general and is therefore useful for a wide array of use-cases.

[guyhoffman]

As I said, I might not be looking in the right place, but the REP PDF linked above only (mostly?) describes what the messages should be, and not how they are implemented. Does the REP also include a specific reference to the implementation you linked to?

[severin-lemaignan]

@cst0 as per ROS process, REP must have a reference implementation -- that is what you found under https://github.com/ros4hri/

As pointed by @guyhoffman , the REP itself is design to be agnostic of a particular implementation -- of course, one might always object that a specification always shape implementations to some extend...

In any case, I'd be happy to try to address specific issue with the spec that over-constrain potential implementations.

[severin-lemaignan]

(as a side note, I'm already aware of a 2nd implementation of the spec that only reuse one or two nodes from the reference implementation)

[clalancette]

This is specific to ROS 1, so I'd change this to be a little more generic.

[audrow]

Some small changes from a partial review:

[audrow]

Do we want this to be global (e.g., starting with leading /)?

[severin-lemaignan]

I'd say yes -- not sure what would be the use-case for 'different' humans in different parts of the system, but happy to discuss that point/change it

[cst0]

I've been sort-of lurking on this thread and keeping an eye on how it goes (since I'm in the HRI research space). @clalancette made the point that it would be good to get more members of the HRI community involved: I have some thoughts I can share. My apologies in advance for a long comment.

The short version of my positive notes would be that:

• I like having a standard for this. There are a bunch of human-perception tools out there and I've had to rewrite wrappers for too many.
• Related to that point, this RFC (correctly) notes that there are a collection of projects that have aimed to do this that have not been widely adopted and have not been recently maintained. A push for more standard tooling is useful, too.
• To the question of generality: I think this is adaptable to a wide array of use-cases. I have some implementation concerns, but I think this addresses some of the more critical use-cases in the HRI space and is reasonably general for them.

My implementation concerns:

• The lack of timestamps in many of the messages might be a problem. People are very sensitive to timing between utterances, motion, etc in much smaller windows than most people would guess (milliseconds), and so I feel that the time at which a social event took place is more valuable than the time at which a node received that information. But that's pretty minor, implementation-wise.
• Providing a unique subtopic per-human is something I'm not a huge fan of. This one is a bit bigger.

I think I see why there's a unique subtopic per person: as the REP observes there needs to be a unique identifier per person. Each identifier is a likely a unique node, which again makes sense to me (different data sources, different goals). So those two things combined force you to split the identification results to unique topics which are grouped by person; a "Human.msg" that encapsulates all that data isn't ideal because that would require some centralized aggregator node. That's not terrible, but it's not ideal and would restrict more distributed systems, and so subtopics it is.

There's a few problems the multi-subtopic approach presents, in my opinion. First, when is it acceptable to introduce or remove a new topic? Identification processes will have some uncertainty, so it's not uncommon for a recognizer to "flicker". Surely we don't want to bring up and take down a topic with every image message we mis-identify someone with, but then we don't want to leave them up indefinitely either. With each human, the RFC discusses as many as 9 persons subtopics, 4 audio subtopics, 6 bodies subtopics, and 8 faces subtopics, with room for more if more identifiers are presented. So are we potentially having implementations where we're dynamically constructing 25+ new topics per person? I don't love that in settings where I know there are only a handful of people being recognized, and I predict it being prohibitively difficult to interact with in (for example) the mall setting where a single time step may have 30+ people. Each of those is another publisher and subscriber to spin up, and will also present developer-side difficulties in introspecting on node/graph behavior. Developer-side difficulty presents a fairly major hurdle to adoption in my view.

The good news is, I think both these concerns could be resolved by introducing a "human header" message and grouping topics into streams of per-person messages. As a result, I think the current approach could be adapted to have a topic per identifier, where the topic is responsible for publishing a stream of observations that are "tagged" by person.

For example, the current approach of

/humans/bodies/alice/skeleton2d # containing points of alice's skeleton
/humans/bodies/bob/skeleton2d   # containing points of bob's skeleton
/humans/faces/alice/roi         # containing alice's face's ROI
/humans/faces/alice/expression  # containing alice's expression
/humans/faces/bob/roi           # containing bob's face's ROI
/humans/faces/bob/expression    # containing bob's expression
...

could become:

/humans/bodies/skeletons2d   # a msg of points of alice's skeleton, then a msg of bob's
/humans/faces/rois           # a msg of alice's face's ROI, then a msg of bob's
/humans/faces/expressions    # a msg of alice's expression, then a msg of bob's

Per @wjwwood's earlier advice on the Header message, frame_id's aren't necessary for these messages (and the seq probably isn't either). But, relating a unit of information to a person and a time is valuable, and so a message encapsulating this can be put on each message. Other parts of the RFC relate people to frame_ids and so that information becomes redundant.

If an identifier can't provide information about what is happening due to time, uncertainty, or it being out of view, it doesn't provide it. An identifier can also be multiple nodes from multiple sources publishing on the one topic. It's then up to the subscriber to do what they want with that observation (or that a Bob hasn't been seen for a while). This approach does have the problem that a node might only care about Alice, and now it's getting a firehose of data about Alice, Bob, and whoever else... But we do see this come up with TF (where I might only care about map->base_link but to get that I receive all of /tf), and it seems to be OK there.

Anyway, those are my main thoughts. My apologies again for a long comment, but hopefully that's helpful.

[severin-lemaignan]

Thanks @cst0 for the insightful comment. I like your suggestion of a 'human header' and I will definitely give it an in-depth look.

Some additional comments below (and a first general terminology point: the REP calls identifier the unique IDs associated to a person/voice/face/body/etc -- each of these IDs correspond to namespace with its own subtopics. Those subtopics could be called attributes of the identifiers, but not identifiers themselves).

I think I see why there's a unique subtopic per person: as the REP observes there needs to be a unique identifier per person. Each identifier is a likely a unique node, which again makes sense to me (different data sources, different goals). So those two things combined force you to split the identification results to unique topics which are grouped by person; a "Human.msg" that encapsulates all that data isn't ideal because that would require some centralized aggregator node. That's not terrible, but it's not ideal and would restrict more distributed systems, and so subtopics it is.

Yes, this summarizes well the rationale for identifiers' namespaces with subtopics.

There's a few problems the multi-subtopic approach presents, in my opinion. First, when is it acceptable to introduce or remove a new topic?

This is a question that is in general highly dependent on the application domain: for instance, if your application is tracking people in a mall, you might only care about the tf frame of the detected faces and probably do not need eg facial landmarks or face identification. That's why the REP only specify very few mandatory subtopics for each identifiers. The other ones are primarily listed in the REP in order to set the topic naming convention (and that list could easily be extended based on what additional social signals the community want to use).

So to answer your question, it is always acceptable to introduce or remove subtopics (as long as the few mandatory ones are published)

Identification processes will have some uncertainty, so it's not uncommon for a recognizer to "flicker". Surely we don't want to bring up and take down a topic with every image message we mis-identify someone with, but then we don't want to leave them up indefinitely either.

The (multi-modal) identification process is specified to be handled in a distinct way in the REP: probabilistic 'candidate' match between faces/bodies/voices and persons are published on the /humans/candidate_matches topic by dedicated identification nodes, and up to a 'person fusion' node to effectively perform the association, accounting for uncertainty. See https://github.com/ros4hri/hri_person_manager for an example implementation.

With each human, the RFC discusses as many as 9 persons subtopics, 4 audio subtopics, 6 bodies subtopics, and 8 faces subtopics, with room for more if more identifiers are presented. So are we potentially having implementations where we're dynamically constructing 25+ new topics per person? I don't love that in settings where I know there are only a handful of people being recognized, and I predict it being prohibitively difficult to interact with in (for example) the mall setting where a single time step may have 30+ people.

I was not a big fan of it either at first. But in practice, it works better than what we feared. Especially because libhri (https://github.com/ros4hri/libhri) abstracts/hides away all the handling of the topics (similar to libtf with the /tf topic). The main downside is the output of rostopic list being a little bit polluted. But again, in practice it does not seem to be such a big issue: we routinely have hundreds of topics published by the robot anyway. A few additional tenths does not make such a big different, as we would use grep in any case to filter out the noise.

The good news is, I think both these concerns could be resolved by introducing a "human header" message and grouping topics into streams of per-person messages. As a result, I think the current approach could be adapted to have a topic per identifier, where the topic is responsible for publishing a stream of observations that are "tagged" by person.

I like a lot the idea of a human header. One major difficulty, though: we do not always have access to the ID of the person (either because the application does not need person identification, or because a face is detected, but not yet identified). So the semantic of the 'person' field of the header would need to be carefully designed.
In addition, we need a system be able to deal with the semantic of 'merging' people: two persons might actually be the same actual person, that was wrongly identified twice. The REP deals with that via 'aliasing', where a person can be marked as an alias for another. Something to bear in mind.

[severin-lemaignan]

@clalancette thanks a lot for the thorough proof-reading. Suggestions all accepted. I need to read more about ROS2 parameters to better understand how to change them. I'll work on that next.

[ymohamed08]

I would like to thank @severin-lemaignan for introducing this very well written, detailed document that represents the ROS4HRI system very well. I am one of the authors of the ROS for human robot interaction paper and co-developer of the system. to address @cst0 and @clalancette point of involving more people from the HRI community. In the HRI lab here at KTH sweden, more and more people are discussing the integration of the ROS4HRI pipeline in their systems. As it has been a challenge to robustly create systems that react to social situations in an appropriate manner. The system usage will be to react to human frustration. In addition, many other colleagues are expressing their interest in using an already standardized system. So, the system is gaining a bit of traction in the HRI community.

[ismaisma]

thank you for this initiative,

[AndreaTimperi]

Such a nice work! We definitely need a HRI standard in ROS!🦾

[itselisa]

Great work! it would be very useful have a HRI standard in ROS!

[SimoTani]

Well done, we need a standard for HRI in ROS!!!

[alphasrobotlab]

Thank you for this work. I think the HRI standard will be really helpful for a lot of people in our niche. Thanks!

[severin-lemaignan]

@cst0 : I've explored a bit more the idea of the 'HRI header', and I think it can indeed work well. An amended version of the REP is here: severin-lemaignan@c7e4134

It relies on new/additional hri_msgs that can be found here: https://github.com/ros4hri/hri_msgs/tree/hri_headers/msg

The main difficulties I can think of are:

• need to re-create 'HRI-stamped' standard messages, like BoolHRI, StringHRI, JointStateHRI
• because the new messages are not standard, need additional work for some tool to work. For instance, the TF frames of humans were published by running a robot_state_publisher for each /humans/bodies/<id>/joint_states. We now have /humans/bodies/joint_states publishing JointStateHRI, which can not be directly consumed anymore by robot_state_publisher. Need to either create + insert a republish_hri node to republish 'pure' JointState msg (not so nice, and bring us back to the previous 'one topic per ID' situation), or we need to implement a hri_state_publisher that will share 99% of the code with robot_state_publisher (might still do that, as we would have only one process for all the humans, instead of one process per detected skeleton)
• another example is cropped images from face, body: they can not be displayed directly anymore in eg rviz, rqt_image_view...

But overall, a big improvement/simplification. So I'm in favour of changing the design to follow your idea.

[ChrisReinke]

In the current standard groups are published under the /humans/interactions/groups topic with Group ID + list of person IDs.

But we might need more information about groups.
For example, useful information could be, the f-formation type (circular group, L-shaped group, ... ).
In the current form there is no way to store this information.

Thus, I suggest to use for groups the same structure as for persons, bodies, faces, and voices.

---

Groups:

The list of currently detected groups (list of group IDs) is published under /humans/groups/tracked (as a hri_msgs/IdsList message).
For each detected group, a namespace /humans/groups// is created.
The following subtopics MAY then be available, depending on available detectors:

• /member_ids | hri_msgs/IdsList | required | IDs of persons that are members of the group.
• /f_formation | std_msgs/String | not required  | Name of the F-formation type.

[severin-lemaignan]

@ChrisReinke sounds like a good idea. Thanks for the contribution!

[clalancette]

We've discussed this REP both internally in Open Robotics and in the ROS 2 TSC.

The consensus is that this is a well-written REP, and though we have some performance concerns about having so many topics in the system, there are also benefits to that approach (mainly in introspection).

With that said, this REP is approved as a Draft, and will be merged into the repository. @severin-lemaignan the next steps here are to get further consensus from the HRI community to eventually attempt to make this an Accepted REP.

Thanks for a well-written article and bearing with the long time delay here.

[severin-lemaignan]

Great news! Thanks @clalancette :-)

I've created a new PR for the proposed 'HRI headers' idea by @cst0 :
#362

@clalancette, would that address some of your concerns regarding performance?

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/ros-2-tsc-meeting-minutes-8-18-2022/27050/1