rospy publisher hanging when subscriber drops connection (e.g. because of wifi)

[advaitjain]

I'm submitting an issue for this question on ROS answers:
http://answers.ros.org/question/29387/rospy-publisher-hanging-when-subscriber-drops-connection-because-of-wifi-is-this-normal/

Summary:
If a rospy publisher is connected to a subscriber on a different computer and you disconnect the ethernet cable, the publisher hangs after about a minute.

Issue can be replicated with the rospy talker node running on a different computer than the listener node.

The cause might be here:
https://github.com/ros/ros_comm/blob/groovy-devel/clients/rospy/src/rospy/impl/tcpros_base.py#L621

[forrestv]

I ran into the same problem. It seems that rospy's use of blocking writes to sockets is indeed the cause.

[mikeferguson]

@tfoote I'm going to bring this back up -- as hydro tf/tf2 will use pure python publishers and this is going to cause a major change in how tf publishers operate over wifi/unreliable links compared to Groovy and earlier.

As can be seen in https://github.com/ros/ros_comm/blob/groovy-devel/clients/rospy/src/rospy/impl/tcpros_base.py#L620, Ken never finished implementing async or retry logic, which means that if you have say, a remote RVIZ connected to a python node publishing tf, you will now have the python node hang if the wifi link drops out. You might recall the problems we had with a certain robot when python nodes started randomly hanging (in that case, it was an /odom topic).

[flixr]

This is indeed a big problem... is anyone working on this?
I will probably need to wait until this is fixed before I can switch to hydro.

[dirk-thomas]

The actually issue has been in ROS since forever. rospy exposes a synchronous blocking publisher API. The problem is that addressing this issue requires asynchronous IO operations. In order to not block the publisher when one subscriber is not able to receive the information (drop-out of wireless, reading the data too slow) each message must be queued per connections and being delivered to the socket from a separate thread (as it is done in roscpp)

This can not be done easily since there is no good way to keep the current API for publishing. E.g. signaling error conditions can not be done via exceptions since the error will happen asynchronously.

May be the change @mikeferguson mentioned which he states to be new in "hydro tf/tf2" can be addressed within tf itself. @tfoote any ideas on that?

[tfoote]

@mikeferguson All python processes in tf were already using python based publishers. tf has always used native client communications. tf vs tf2 Was there more that you thought would exacerbate this issue?

[mikeferguson]

From the migration guide (http://wiki.ros.org/hydro/Migration):

tf2 continues to use the C++ core library, but with the isolation of the BufferCore class only the very core algorithm is the wrapped C++ class. All communications for the python bindings are done in pure python avoiding many of the corner cases where pytf would have issues.

I thought that meant tf had originally used C++ publishers in the wrapper. If not, then disregard the tf/tf2 aspects.

Regardless though, this bug pretty much makes rospy useless over a wifi link, are we really saying it just can't be fixed? I guess I haven't looked at the upper level APIs, but I've looked at the roscpp queues, and thought we could implement something similar in here.

[forrestv]

This can not be done easily since there is no good way to keep the current API for publishing. E.g. signaling error conditions can not be done via exceptions since the error will happen asynchronously.

I don't think the API creates any constraints - all the documented exceptions could happen immediately when publish() is called, not later when a write fails or something (since a write failing doesn't cause publish() to raise an exception in the caller).

[tfoote]

@mikeferguson The critical difference between tf2 and tf is that tf2's core library no longer uses time and all the timeout are implemented in python. That was where pytf had issues when there were timeouts insdie of the wrapped c++ but roscpp was not updating time because it was not initialized.

@dirk-thomas and I have spent quite a while on the whiteboard making sure that we can do this without significantly changing existing behavior. We're closing in on a solution that I think will work.

[mikeferguson]

@tfoote thanks for the explanation, and thanks for taking a look at this!

[dirk-thomas]

Please take a look at #308 and provide feedback. Would someone try to update tf2 to use this new option and verify that it fixes the describe problem adequately?

[tfoote]

Before closing this after releasing #308 we should collect a list of major publishers to be patched. So far I have

• tf
• tf2
• rospy tutorials (wiki)
• rospy tutorials (repo)

[dirk-thomas]

Updated http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers as well as http://wiki.ros.org/rospy_tutorials/Tutorials/WritingPublisherSubscriber

[tfoote]

all four have been checked.