Memory exhaustion on server with high rate large blob data

[jihoonl]

This is follow-up issue of #192.

Questions

• Should rosbridge have intelligence or configuration to adapt slow and fast clients seamlessly?
• Rosbridge server will suffer from memory exhaustion if slow client is subscribing to high rate large chuck data anyway. How should rosbridge react on this memory exhaustion?

[mvollrath]

Adaptation sounds complicated, but it seems like an additional per-client queue is needed in the outgoing implementation to prevent clients from killing the server with greedy subscriptions. This queue would be configured by the server's parameters, outside of the protocol's throttle and queue options. There should probably be a warning message the first time a message is dropped from a client's queue.

Additionally, a status message (as described in the protocol) should be sent to warn the client that they are dropping messages and need to add throttling.

[sevenbitbyte]

Should rosbridge have intelligence or configuration to adapt slow and fast clients seamlessly?

Yes ROS bridge should serve all clients as well as the available resources will allow. I think dynamically throttling a slow client may be a good direction to go in if it can be done in a way that also detects if the client's connection becomes faster.

Rosbridge server will suffer from memory exhaustion if slow client is subscribing to high rate large chuck data anyway. How should rosbridge react on this memory exhaustion?

A client should not be able to impact the server's performance so easily. Server development should expect clients to not be well behaved and expect that they are not ideally optimizated. If greater sever stability means dropping packets in new ways then by all means please do. The protocol does appear to have all needed features to inform the client that they are getting throttle by way of the status messages.

The incentive to app developers is to optimize clients(use throttling) so that users have the expected performance but when writing apps I don't always no apriori the exact network conditions my apps are used in. So its just not always possible to get throttle settings right for all users on all networks. Getting hits from the server about performance would make it easier from the app development perspective.

[mvollrath]

The rosbridge_library protocol has a throttle and queue system, but since it dumps messages into the Tornado queue asynchronously, the backpressure from the client doesn't make it back to the protocol's queue. Instead Tornado infinitely queues the backlog.

If we extend the RosbridgeWebSocket handler to hand off outgoing messages to an intermediate per-connection queue instead of pushing straight to Tornado, we can make synchronous Tornado writes from per-connection threads. This way we gain control of the backpressure on the server side and clients don't slow each other down.

[viktorku]

This backpressure in the server from slow clients can be solved with a simple circular buffer which drops stale messages after a certain cap limit. Wouldn't that be the simplest to implement? Does Tornado offer this out of the box in some way?

[mvollrath]

Tornado gives us a Future when we write to the socket, so we can either ignore it (as we do currently) or wait for it to finish to effectively make the write synchronous. Making the write synchronous would be the simplest way to connect the backpressure.

However, since all the web socket writes happen sequentially in the subscriber thread upon receiving a message, the last client would be waiting for the synchronous writes of all the other clients before it. This is why we would need a thread per connection to decouple the clients synchronous writes and give them their own queues.

[mvollrath]

Or better yet, decouple the threads further up the chain where MultiSubscriber is iterating over the callbacks. This way the queue handler works as intended and we aren't adding a queue after a queue.

[mvollrath]

I did some more looking around and the answer might be simpler than I thought.

Adding a small blurb to RosbridgeWebSocket.send_message lets us identify which thread is sending the message to Tornado:

    def send_message(self, message):
        """..."""
        import threading
        print 'writing from {}'.format(threading.current_thread())
        IOLoop.instance().add_callback(partial(self.write_message, message, binary))

Now the results depend completely on the settings the client used to subscribe to the topic. I tested two clients running identical applications subscribed to the same topic.

With queue_length unset or set to 0, both send from the subscriber thread:

writing from <Thread(/bench/pcl, started daemon 140434852521728)>
writing from <Thread(/bench/pcl, started daemon 140434852521728)>

With queue_length set to any value > 0, the sends are decoupled by QueueMessageHandler:

writing from <QueueMessageHandler(Thread-7, started daemon 140427042748160)>
writing from <QueueMessageHandler(Thread-15, started daemon 140427034355456)>

So if we can figure out how to make writes synchronous and default queue_length to some non-zero value, we hook up backpressure to the QueueMessageHandler like users would expect.

I also found Queue support in Tornado which might be useful, but haven't looked at it closely.

[mvollrath]

I've had some success with making web socket writes synchronous in Tornado 4.5.3 by locking both the callback addition and the write itself (with a future callback releasing the Lock). Tornado is still invisibly queuing some unknown amount of data, maybe down in the socket. If the write_message Future ever fails to run callbacks (which is not uncommon), the server's IOLoop is completely frozen. Unless newer Tornado versions are more reliable, this is on the shelf for now.

[mvollrath]

For the curious, https://github.com/EndPointCorp/rosbridge_suite/tree/fix_ws_backpressure