rospy subscriber queue_size acting strange when buff_size is too small

[bsubei]

So I was having the same problem as the person here, and I found out that setting a queue_size of 1 for my subscriber node doesn't work (as in, it keeps queueing up messages and doesn't drop them) if my buff_size is not large enough.

I have a subscriber node that can process ROS images at say ~8Hz, and it subscribes to a topic with large messages (ROS images from bagfile) that publishes at ~24Hz. When I set queue_size to be 1 for my subscribing node and use the default buff_size of 65536, the messages start piling up (they're still processed at 8Hz), but none of the messages are dropped! This essentially makes it behave as if its queue_size was infinity. (if I kill the publishing bagfile, for example, my subscribing node still processes the old messages that were still in the queue although I explicitly set queue_size to 1).

Now when I set the buff_size on that same node to, say 2**24 bytes (a ridiculously large size big enough for images) with the same queue_size of 1, the queue behaves properly (it drops the messages it can't handle, and still processes the images at ~8Hz).

Is there a bug with queue_size if buff_size is too small?

I tried to dig in the source code but all I could think of was the possibility of a bug existing if buff_size is not big enough to capture the entire message coming in from the socket.

Sorry if this issue is a non-issue (or super obvious), but I don't see anywhere in the documentation the answer to this question:
How does rospy behave if buff_size is smaller than the received message?

[dirk-thomas]

The behavior you are seeing is not a bug.

In your case of sending data between processes there are multiple queues involved:

• the publisher will slow down if it is unable to write the message to the socket, as of Hydro you can specify a queue_size on the publisher too to use asynchronous publishing (see http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers)
• the OS will buffer data of the socket which will be ~20 msgs in your case (depending on the OS and settings)
• the subscriber uses a queue to buffer incoming messages, the buff_size determines how much data is read at once from the socket, a very high number will basically empty the OS queue and apply the queue limit of the subscriber, otherwise it reads them in 64k chunks and therefore in your case of large messages leaves the rest of the data in the OS buffer

Your example is also hiding what is actually happening. You should print a msg counter in the talker code, put the number into the message and also print this number in the listener. Then you will see which messages will actually be discarded and when the publisher is slowing down (since you are not setting a queue size for the publisher).

[JingleiY]

I also ran into the same problem! The subscriber just drop msgs due to insufficient buffer size. So how to increase the buffer size? Thanks!

[dirk-thomas]

@JingleiY please ask question on answers.ros.org as described in the ROS support guidelines.