Slow web socket performance

[mvollrath]

The rosbridge web socket server is very CPU hungry when a lot of messages are being routed to browser clients.

Our use case is subscription to multiple sensors from multiple browser clients for distributed, interactive visuals. ROS is a great platform for this kind of thing, but when we ran rosbridge on a modest server (some kind of Atom processor with plenty of RAM), it couldn't keep up. The server would eat a lot of CPU as message traffic ramped up, and sometimes choked up resulting in apparent system breakdown. We ended up offloading the rosbridge server to a more powerful server, but we'd like for rosbridge to be efficient enough to handle hundreds of messages per second without burning up CPU cycles on a user-facing machine.

I looked around and found this discussion pointing at Tornado's web socket masking function as the culprit, with native speedups making an order of magnitude difference:

https://groups.google.com/forum/#!topic/python-tornado/n6VpUr8zk2c

Then this Tornado pull request to add native speedups:

tornadoweb/tornado#579

Looks like it was released in Tornado 3.2. Ubuntu's python-tornado package is still at 3.1.1 in trusty, but pip will install 4.0.2 as of today. Presently rosbridge includes Tornado 2.3.

This is related to #121 which should resolve this issue if Tornado comes from pip or is updated to 3.2+ in the rosbridge tree.

Thanks
mv

[megawac]

Correct me if I'm wrong but this is fixed on master+develop as of #126 and has been published to the debs

[mvollrath]

Thanks, #126 does resolve this.

[mvollrath]

Using 4.0.2 Tornado sources doesn't resolve this. The tornado.speedups module has to be compiled native to get full speed websockets. See Tornado's setup.py:

https://github.com/tornadoweb/tornado/blob/v4.0.2/setup.py

We may be able to chain load Tornado's setup.py or just copy the custom_build_ext bits into rosbridge_server's setup.py.

Here is test code I've verified with pip-installed Tornado:

#!/usr/bin/env python
try:
    import tornado.speedups
except ImportError:
    print 'no speedups!'
else:
    print 'blazing fast websockets!'

[gladkikhartem]

1. You'll need to build tornado speedups.
2. If you'll send multiple small websocket messages - it will eat up your CPU!
   I've met this during data upload to tornado:

Uploading file by 1024 bytes chunks will eat your 100% CPU at 5-10 Mbit/s

Uploading file by 1024*20 bytes chunks will eat 10% CPU at 100Mbit/s

Magic.