Timeout for map saver CLI too short / subscription not transient local

[Michael-Equi]

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04
• ROS2 Version:
  • Foxy binaries
• Version or commit hash:
  • 69977cd
• DDS implementation:
  • Fast-RTPS

Steps to reproduce issue

Running slam toolbox on the commit at this PR SteveMacenski/slam_toolbox#223 and attempting to save the map through the cli by calling the /save_map service. To verify /map was being published to correctly I ran a ros2 topic hz which settled out with around the following output.

average rate: 0.145
	min: 4.970s max: 30.000s std dev: 4.28447s window: 48

Given a call to the /save_map service I found that saving the map only succeeds about 1/3rd of the time, the rest of the time it outputs:

[async_slam_toolbox_node-1] [INFO] [1594529940.286451570] [map_saver]: 
[async_slam_toolbox_node-1] 	map_saver lifecycle node launched. 
[async_slam_toolbox_node-1] 	Waiting on external lifecycle transitions to activate
[async_slam_toolbox_node-1] 	See https://design.ros2.org/articles/node_lifecycle.html for more information.
[async_slam_toolbox_node-1] [INFO] [1594529940.286567916] [map_saver]: Creating
[async_slam_toolbox_node-1] [INFO] [1594529940.286682543] [map_saver]: Saving map from 'map' topic to '' file
[async_slam_toolbox_node-1] [WARN] [1594529940.286694799] [map_saver]: Free threshold unspecified. Setting it to default value: 0.250000
[async_slam_toolbox_node-1] [WARN] [1594529940.286705367] [map_saver]: Occupied threshold unspecified. Setting it to default value: 0.650000
[async_slam_toolbox_node-1] [ERROR] [1594529942.288657252] [map_saver]: Failed to save the map: timeout
[async_slam_toolbox_node-1] [INFO] [1594529942.288713783] [map_saver]: Destroying

Expected behavior

The timeout is easily configurable / longer when using slam_toolbox than the current 2 seconds or proper QoS configuration sets up the communications to be transient local (so they act like latching in ROS1) so that the last message is immediately sent over given that is hasn't expired past its deadline.

Additional Notes

I am not yet completely familiar with how different QoS combinations for the pub/subs behave but I would guess the best solution is to get the QoS configured so that it receives the last published map within a given time period. I know the slam_toolbox /map has the configuration of reliable and transient local but the map_saver /map subscription looks to be reliable and volatile (system defaults). Does that mean the map saver will not get the transient local behavior when subscribed to the /map topic? Potentially related to this, I noticed that after restarting slam_toolbox while keeping the same Rviz instance open it would not receive any new maps while it had the QoS profile of reliable and transient local. In order to get RViz to subscribe to the new slam_toolbox instance's /map topic correctly, I had to change the QoS on it to best effort and transient local.

[SteveMacenski]

A param for the saver to use that QoS would be a good move. Can you submit a PR letting the QoS be parameterized?

[Michael-Equi]

Yes. Just to be clear, a volatile subscriber + transient local publisher will act like a volatile topic overall, right? Should I just add something like the map_subscribe_transient_local parameter used on the costmap layers?

[SteveMacenski]

https://index.ros.org/doc/ros2/Concepts/About-Quality-of-Service-Settings/ see the charts for compatible QoS profiles.

You could add that param, yes.

[Michael-Equi]

https://index.ros.org/doc/ros2/Concepts/About-Quality-of-Service-Settings/ see the charts for compatible QoS profiles.

I read through that before and it does say that the publisher is responsible for persisting samples when set to transient local. If that was true though then this would not be an issue which is why I am a bit confused. Do only transient local subscribers ask for the the persisted message and therefore requiring both the subscriber and publisher to be transient local in order to get the latching effect?

[SteveMacenski]

Look at the charts for the QoS profiles that are compatible. Not all publishers profiles are valid for all subscribers profiles.

[Michael-Equi]

It says they are compatible

Publisher | Subscription | Compatible
Volatile  | Volatile         | Yes
Volatile  | Transient local | No
Transient local | Volatile | Yes <-----
Transient local | Transient local | Yes

But doesn't say exactly what the behavior should be.

[SteveMacenski]

Given a call to the /save_map service I found that saving the map only succeeds about 1/3rd of the time, the rest of the time it outputs:

It seems like then its working and maybe we just need to set another timeout if its not parameterized for your uses.

[Michael-Equi]

I made the PR which basically tells me that the "latching" behavior does require both the publisher and subscriber to be transient local.

[Mohit-Ak]

@SteveMacenski
I still get this most of the time. I also don't see the map_subscribe_transient_local in the command line.

Every 0.1s: ros2 run nav2_map_server map_saver_cli -f /home/bautomation/maps/sb_20201128021146                                                                 bav-dl: Sat Nov 28 02:17:39 2020

[INFO] [1606529860.280184194] [map_saver]:
        map_saver lifecycle node launched.
        Waiting on external lifecycle transitions to activate
        See https://design.ros2.org/articles/node_lifecycle.html for more information.
[INFO] [1606529860.280315330] [map_saver]: Creating
[INFO] [1606529860.280409596] [map_saver]: Saving map from 'map' topic to '/home/bautomation/maps/sb_20201128021146' file
[WARN] [1606529860.280426778] [map_saver]: Free threshold unspecified. Setting it to default value: 0.250000
[WARN] [1606529860.280441778] [map_saver]: Occupied threshold unspecified. Setting it to default value: 0.650000
[ERROR] [1606529862.283775030] [map_saver]: Failed to save the map: timeout
[INFO] [1606529862.283905999] [map_saver]: Destroying   

ros2 run nav2_map_server map_saver_cli -h
Usage:
  map_saver_cli [arguments] [--ros-args ROS remapping args]

Arguments:
  -h/--help
  -t <map_topic>
  -f <mapname>
  --occ <threshold_occupied>
  --free <threshold_free>
  --fmt <image_format>
  --mode trinary(default)/scale/raw

NOTE: --ros-args should be passed at the end of command line

Operating System:

• Ubuntu 20.04

ROS2 Version:

• Foxy binaries

Version or commit hash:

ros-foxy-nav2-map-server	0.4.3-1focal.20201103.155816
ros-foxy-nav2-map-server-dbgsym	0.4.3-1focal.20201103.155816

[Mohit-Ak]

Update

• I think the commit is not merged with the "foxy" branch and hence the problem.

[eric565648]

@Mohit-Ak Did you manage to solve the problem? I still get the same problem now. I tried to increase the timeout time but it doesn't work as well.

[Mohit-Ak]

It is still frequent in smaller systems but try using a beast PC configuration. Also, play around with image magick libraries.
#2109

I actually found a workaround of taking a screenshot of the map from RVIZ2 and doing some math on it to convert it into the required PGM using GIMP.