Both depth camera plugins stop publishing after upgrading gazebo9 #1175

corot · 2020-10-08T13:51:49Z

Both depth camera plugins stop publishing after upgrading gazebo9 from amd64 9.14.0-1 to 9.15.0-2
Looks like the plugin interface gets broken... no idea how, but the fact is that the plugin gets initialized and advertises the topics.
But the callback methods with the images (OnNewDepthFrame, OnNewRGBPointCloud, OnNewImageFrame) never get called by gazebo!

mitschron · 2020-10-08T18:54:11Z

I have the same problem with all gazebo plugins. I am working with gazebo11 ros-melodic.
In my case it seems to be caused by the performance metric subscriber added on this commit: 6c53c21
Changing the topic to from absolute namespace to relative seems to fix it. ("/gazebo/performance_metrics" -> "~/performance_metrics").
That is in line 202 of gazebo_ros_api_plugin.cpp

HighPriest · 2020-10-16T14:51:33Z

@mitschron Can you turn your fix into a pull request?
Alot of people have this problem, including my team. We have found a different fix, but it's on a much higher abstraction level.

mitschron · 2020-10-16T15:06:27Z

@mitschron Can you turn your fix into a pull request?
Alot of people have this problem, including my team. We have found a different fix, but it's on a much higher abstraction level.

#1180

ahcorde · 2020-10-19T13:12:09Z

Hello,

can you provide more details? are you installing the deb packages? compiling from source ?

HighPriest · 2020-10-19T13:24:58Z

@ahcorde
The issue appears in both source and package installation of gazebo11 against ROS2 Foxy.
Building the gazebo_ros_pkgs/dashing branch against Gazebo11 "fixes" the issue with gazebo_ros_camera and gazebo_ros_depth_camera plugins.

ahcorde · 2020-10-19T13:26:13Z

can you run it with --verbose ?

HighPriest · 2020-10-19T13:28:52Z

You mean run the build or run Gazebo, or run Gazebo and spawn an object with camera plugin?
There is no meaningful output out of Gazebo either way.
As I said before, debugging the plugging shows the "publish" function executes properly, but the data does not appear on the ROS topic.

The fix made by @mitschron appears to silence the PROFILER, which makes me think that it might be something related with it.

ahcorde · 2020-10-19T13:36:44Z

@HighPriest
I mean run Gazebo and spawn an object with camera plugin.

ros2 gazebo_ros gazebo.launch.py world:=<depth camera world> verbose:=true

Maybe there is something else wrong.

The proposed fix will resolve ~ to /gazebo/worldname which is the same topic name. I'm not really sure that this is the problem.

HighPriest · 2020-10-19T14:22:57Z

[INFO] [launch]: All log files can be found below /home/user/.ros/log/2020-10-19-16-17-18-065573-HieroDev-153874
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [gzserver-1]: process started with pid [153876]
[INFO] [gzclient --verbose  -2]: process started with pid [153878]
[INFO] [spawn_entity.py-3]: process started with pid [153882]
[INFO] [filtering_module-4]: process started with pid [153884]
[INFO] [state_controller-5]: process started with pid [153886]
[INFO] [target_counter-6]: process started with pid [153888]
[INFO] [movement_control-7]: process started with pid [153890]
[INFO] [path_planner-8]: process started with pid [153892]
[INFO] [can_out-9]: process started with pid [153925]
[INFO] [can_in-10]: process started with pid [153931]
[filtering_module-4] [WARN] [1603117038.815194913] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
[can_in-10] [INFO] [1603117038.873386327] [can_to_msg]: Publishing Can Frames on [/raw_data]
[can_out-9] [INFO] [1603117038.879786678] [sim.odometry_to_can]: Subscribed to [/sim/odometry], Wheel torque ID: [210], Wheel speed ID: [230], Steer angle ID: [220]
[can_out-9] [INFO] [1603117038.879998139] [sim.odometry_to_can]: Subscribed to [/sim/gps], GPS1 ID: [350], GPS2 ID: [351]
[can_out-9] [INFO] [1603117038.880030353] [sim.odometry_to_can]: Subscribed to [/sim/imu], Accelerometer ID: [400], Gyro ID: [401]
[gzclient --verbose  -2] Gazebo multi-robot simulator, version 11.2.0
[gzclient --verbose  -2] Copyright (C) 2012 Open Source Robotics Foundation.
[gzclient --verbose  -2] Released under the Apache 2 License.
[gzclient --verbose  -2] http://gazebosim.org
[gzclient --verbose  -2] 
[gzserver-1] Gazebo multi-robot simulator, version 11.2.0
[gzserver-1] Copyright (C) 2012 Open Source Robotics Foundation.
[gzserver-1] Released under the Apache 2 License.
[gzserver-1] http://gazebosim.org
[gzserver-1] 
[spawn_entity.py-3] [INFO] [1603117039.426191968] [spawn_entity]: Spawn Entity started
[spawn_entity.py-3] [INFO] [1603117039.426875882] [spawn_entity]: Loading entity XML from file 'censored'
[spawn_entity.py-3] [INFO] [1603117039.431429525] [spawn_entity]: Waiting for service /spawn_entity, timeout = 15
[spawn_entity.py-3] [INFO] [1603117039.432057042] [spawn_entity]: Waiting for service /spawn_entity
[gzclient --verbose  -2] [Msg] Waiting for master.
[gzclient --verbose  -2] [Msg] Connected to gazebo master @ http://127.0.0.1:11345
[gzclient --verbose  -2] [Msg] Publicized address: 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[gzclient --verbose  -2] [Err] [InsertModelWidget.cc:402] Missing model.config for model 'censored'
[spawn_entity.py-3] [INFO] [1603117045.697837274] [spawn_entity]: Calling service /spawn_entity
[gzserver-1] [Msg] Waiting for master.
[gzserver-1] [Msg] Connected to gazebo master @ http://127.0.0.1:11345
[gzserver-1] [Msg] Publicized address: 192.168.66.84
[gzserver-1] [Wrn] [msgs.cc:1842] Conversion of sensor type[depth] not supported.
[spawn_entity.py-3] [INFO] [1603117046.050501915] [spawn_entity]: Spawn status: SpawnEntity: Successfully spawned entity ['censored']
[INFO] [spawn_entity.py-3]: process has finished cleanly [pid 153882]
[gzserver-1] [INFO] [1603117046.353654122] [sim_camera_plugin]: Publishing camera info to [/camera/camera_info]
[gzserver-1] [INFO] [1603117046.355075690] [sim_camera_plugin]: Publishing depth camera info to [/camera/depth/camera_info]
[gzserver-1] [INFO] [1603117046.355682182] [sim_camera_plugin]: Publishing pointcloud to [/camera/points]
[gzserver-1] [WARN] [1603117046.453826520] [sim.vehicle]: Steering wheel joint [steering_wheel_joint] not found.
[gzserver-1] [INFO] [1603117046.468808404] [sim.vehicle]: Subscribed to [/sim/cmd_autonomy]
[gzserver-1] [INFO] [1603117046.468863622] [sim.vehicle]: Subscribed to [/sim/cmd_key]
[gzserver-1] [INFO] [1603117046.468885930] [sim.vehicle]: Subscribed to [/sim/cmd_joy]
[gzserver-1] [INFO] [1603117046.474569822] [sim.vehicle]: Advertising odometry on [/sim/odometry]
[gzserver-1] [INFO] [1603117046.509116098] [sim.gps]: Advertise GPS data on [/sim/gps]
[gzserver-1] [INFO] [1603117046.545875065] [sim.imu]: Advertise IMU data on [/sim/imu]
[gzserver-1] [Wrn] [msgs.cc:1842] Conversion of sensor type[depth] not supported.
[gzserver-1] [Wrn] [msgs.cc:1842] Conversion of sensor type[depth] not supported.

Here is the output of --verbose
"Conversion not supported" notice was pointed out in some other issue, where despite its appearance, everything worked.
Nothing related to camera output is here.

corot · 2020-10-20T02:05:02Z

In my case, I just compiled gazebo9's previous version (9.14.0-1) and it worked again.
Didn't try the fix yet, but can do this week.

ahcorde · 2020-10-20T13:14:36Z

I have tried to reproduce the issue in a docker container. I have used two different setups.

Gazebo 9 + melodic-gazebo-ros-pkgs. Installing gazebo 9.15 from packages I have tried to compile gazebo-ros-pkgs from sources too.
Gazebo 11 + foxy-gazebo-ros-pkgs. . Installing gazebo 11.2 from packages

Both setups works for me. Can you confirm the debs version of Gazebo and gazebo_ros_pkgs ?

mitschron · 2020-11-03T16:13:51Z

Gazebo 11 from packages

gazebo --version
Gazebo multi-robot simulator, version 11.2.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

Compiled gazebo-ros-pkgs from sources (melodic-devel)

gcc --version
gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

jacobperron · 2020-12-07T23:58:25Z

I'm unable to reproduce the issue. I've tried Gazebo 9.15 and Gazebo 9.16 from debs on Ubuntu Focal (20.04); with ROS 2 Foxy (from debs) and gazebo_ros_pkgs built from source.

I can echo all topics from the demo world, which come in at about 5Hz:

Run the demo world

 gazebo gazebo_ros_depth_camera_demo.world

Get message frequency:

 ros2 topic hz /demo_cam/mycamera/raw_cam_info_demo
 ros2 topic hz /demo_cam/mycamera/depth_demo
 ros2 topic hz /demo_cam/mycamera/image_demo
 ros2 topic hz /demo_cam/mycamera/points_demo
 ros2 topic hz /demo_cam/mycamera/raw_cam_info_demo

Note, I am not able to see the images in rqt_image_view. This is because of a QoS mismatch. But in RViz, if I change the topic QoS reliability setting to "best effort", then I can see the images.

I can try out ROS 1 next.

@corot Which version of gazebo_ros_pkgs are you using and where do the symptoms manifest (e.g. is there a particular visualization tool or command-line tool you are using to verify the lack the messages)?

scpeters · 2020-12-10T01:19:45Z

I believe this is related to gazebosim/gazebo-classic#2902, which is a deadlock in gazebo's sensor threads if you have multiple types of sensors in a world (such as a camera and an IMU, or a camera and a ray sensor) and subscribe to the gazebo topic /gazebo/performance_metrics. As of #1145 / #1147, gazebo_ros is subscribing to that gazebo topic by default.

For a quick workaround, comment out the following lines:

I'll work on disabling the subscription to /gazebo/performance_metrics by default until gazebosim/gazebo-classic#2902 has been fixed.

We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. The link_states and model_states topics currently use an advertise mechanism with callbacks when a subscriber connects or disconnects, so I've used that same pattern for the performance_metrics topic. This also helps workaround the deadlock documented in ros-simulation#1175 and gazebosim/gazebo-classic#2902. Signed-off-by: Steve Peters <scpeters@openrobotics.org>

scpeters · 2020-12-12T02:20:25Z

I'll work on disabling the subscription to /gazebo/performance_metrics by default until osrf/gazebo#2902 has been fixed.

I've opened #1202 targeting noetic-devel that will avoid subscribing to /gazebo/performance_metrics unless there are actual subscribers to the ROS topic that republishes this data. We are already doing this for link_states and model_states, so I copied that pattern. This should work around the deadlock in gazebosim/gazebo-classic#2902 as long as you don't subscribe to the ROS topic in a world with multiple camera types. I will backport to melodic-devel from this branch afterwards.

I don't know if there's a similar API in ros2; we don't seem to be using that type of approach for link_states or model_states in the ros2 branch.

We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. The link_states and model_states topics currently use an advertise mechanism with callbacks when a subscriber connects or disconnects, so I've used that same pattern for the performance_metrics topic. This also helps workaround the deadlock documented in #1175 and gazebosim/gazebo-classic#2902. This also adds a GAZEBO_ROS_HAS_PERFORMANCE_METRICS macro that reduces duplication of the version checking logic for performance metrics in gazebo and adds fixes some doc-string and typos in existing code Signed-off-by: Steve Peters <scpeters@openrobotics.org>

corot · 2020-12-21T21:54:43Z

Looks like this issue gets fixed on gazebo 9.16. Or at least I have reverted to use debs from compiled 9.14 without issue.
Can anyone confirm?

We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. This changes gazebo_ros_init to only subscribe to the gazebo topic if there are any subscribers to the corresponding ROS topic. While advertiser callbacks are used in ROS 1 but are not yet in ROS2, here we use polling in the GazeboRosInitPrivate::PublishSimTime callback to check for subscribers since it is called for each Gazebo time step. This also helps workaround the deadlock documented in ros-simulation#1175 and gazebosim/gazebo-classic#2902. Signed-off-by: Steve Peters <scpeters@openrobotics.org>

scpeters · 2020-12-22T04:02:03Z

I don't know if there's a similar API in ros2; we don't seem to be using that type of approach for link_states or model_states in the ros2 branch.

I've made a pull request targeting ros2 that uses polling instead of callbacks: #1205

scpeters · 2020-12-22T04:03:52Z

Looks like this issue gets fixed on gazebo 9.16. Or at least I have reverted to use debs from compiled 9.14 without issue.
Can anyone confirm?

I believe the underlying issue has not yet been fixed in osrf/gazebo; please see gazebosim/gazebo-classic#2902

I have merged workarounds to noetic-devel in #1202 and there are workarounds under review for melodic-devel (#1203) and ros2 (#1205)

Backport of #1202 to melodic-devel. We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. The link_states and model_states topics currently use an advertise mechanism with callbacks when a subscriber connects or disconnects, so I've used that same pattern for the performance_metrics topic. This also helps workaround the deadlock documented in #1175 and gazebosim/gazebo-classic#2902. This also adds a GAZEBO_ROS_HAS_PERFORMANCE_METRICS macro that reduces duplication of the version checking logic for performance metrics in gazebo and adds fixes some doc-string and typos in existing code Signed-off-by: Steve Peters <scpeters@openrobotics.org>

…1205) We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. This changes gazebo_ros_init to only subscribe to the gazebo topic if there are any subscribers to the corresponding ROS topic. While advertiser callbacks are used in ROS 1 but are not yet in ROS2, here we use polling in the GazeboRosInitPrivate::PublishSimTime callback to check for subscribers since it is called for each Gazebo time step. This also helps workaround the deadlock documented in #1175 and gazebosim/gazebo-classic#2902. This also adds a macro to reduce duplication of the version checking logic. Signed-off-by: Steve Peters <scpeters@openrobotics.org>

We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. The link_states and model_states topics currently use an advertise mechanism with callbacks when a subscriber connects or disconnects, so I've used that same pattern for the performance_metrics topic. This also helps workaround the deadlock documented in ros-simulation#1175 and gazebosim/gazebo-classic#2902. Signed-off-by: Steve Peters <scpeters@openrobotics.org>

Backport of #1202 to kinetic-level. We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. The link_states and model_states topics currently use an advertise mechanism with callbacks when a subscriber connects or disconnects, so I've used that same pattern for the performance_metrics topic. This also helps workaround the deadlock documented in #1175 and gazebosim/gazebo-classic#2902. This also adds a GAZEBO_ROS_HAS_PERFORMANCE_METRICS macro that reduces duplication of the version checking logic for performance metrics in gazebo and adds fixes some doc-string and typos in existing code Signed-off-by: Steve Peters <scpeters@openrobotics.org>

…1205) We are currently subscribing to the /gazebo/performance_metrics topic even if there are no subscribers to the ROS topic forwarding this data. This changes gazebo_ros_init to only subscribe to the gazebo topic if there are any subscribers to the corresponding ROS topic. While advertiser callbacks are used in ROS 1 but are not yet in ROS2, here we use polling in the GazeboRosInitPrivate::PublishSimTime callback to check for subscribers since it is called for each Gazebo time step. This also helps workaround the deadlock documented in #1175 and gazebosim/gazebo-classic#2902. This also adds a macro to reduce duplication of the version checking logic. Signed-off-by: Steve Peters <scpeters@openrobotics.org>

HighPriest · 2021-05-28T15:42:04Z

This issue has been fixed in latest versions of Gazebo11.
Anyone still having issues with this?

corot · 2021-05-30T12:01:55Z

It started working after using compiled version, but now also the latest gazebo9 release works (sorry, I should have closed the issue, my bad)

mitschron mentioned this issue Oct 18, 2020

gazebo_ros: Use relative namespace for performance metrics #1180

Closed

scpeters mentioned this issue Dec 12, 2020

Only subscribe to /gazebo/performance_metrics when necessary #1202

Merged

scpeters mentioned this issue Dec 15, 2020

Backport #1202 to melodic-devel: fix performance metrics #1203

Merged

scpeters mentioned this issue Dec 22, 2020

ros2: Only subscribe to /gazebo/performance_metrics when necessary #1205

Merged

scpeters mentioned this issue Dec 22, 2020

Backport #1202 to kinetic: fix performance metrics #1209

Merged

corot closed this as completed May 30, 2021

corot mentioned this issue May 30, 2021

gazebo_ros_depth_camera not publishing depth camera_info #1174

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Both depth camera plugins stop publishing after upgrading gazebo9 #1175

Both depth camera plugins stop publishing after upgrading gazebo9 #1175

corot commented Oct 8, 2020 •

edited

mitschron commented Oct 8, 2020

HighPriest commented Oct 16, 2020

mitschron commented Oct 16, 2020

ahcorde commented Oct 19, 2020

HighPriest commented Oct 19, 2020 •

edited

ahcorde commented Oct 19, 2020 •

edited

HighPriest commented Oct 19, 2020 •

edited

ahcorde commented Oct 19, 2020

HighPriest commented Oct 19, 2020 •

edited

corot commented Oct 20, 2020

ahcorde commented Oct 20, 2020 •

edited

mitschron commented Nov 3, 2020

jacobperron commented Dec 7, 2020

scpeters commented Dec 10, 2020

scpeters commented Dec 12, 2020

corot commented Dec 21, 2020

scpeters commented Dec 22, 2020

scpeters commented Dec 22, 2020

HighPriest commented May 28, 2021

corot commented May 30, 2021

Both depth camera plugins stop publishing after upgrading gazebo9 #1175

Both depth camera plugins stop publishing after upgrading gazebo9 #1175

Comments

corot commented Oct 8, 2020 • edited

mitschron commented Oct 8, 2020

HighPriest commented Oct 16, 2020

mitschron commented Oct 16, 2020

ahcorde commented Oct 19, 2020

HighPriest commented Oct 19, 2020 • edited

ahcorde commented Oct 19, 2020 • edited

HighPriest commented Oct 19, 2020 • edited

ahcorde commented Oct 19, 2020

HighPriest commented Oct 19, 2020 • edited

corot commented Oct 20, 2020

ahcorde commented Oct 20, 2020 • edited

mitschron commented Nov 3, 2020

jacobperron commented Dec 7, 2020

scpeters commented Dec 10, 2020

scpeters commented Dec 12, 2020

corot commented Dec 21, 2020

scpeters commented Dec 22, 2020

scpeters commented Dec 22, 2020

HighPriest commented May 28, 2021

corot commented May 30, 2021

corot commented Oct 8, 2020 •

edited

HighPriest commented Oct 19, 2020 •

edited

ahcorde commented Oct 19, 2020 •

edited

HighPriest commented Oct 19, 2020 •

edited

HighPriest commented Oct 19, 2020 •

edited

ahcorde commented Oct 20, 2020 •

edited