Single Process

[cancan101]

Is there anyway to use a similar approach to https://github.com/jrief/django-websocket-redis which:

• Runs a seperate Django main loop in a cooperative concurrency model using gevent, thus only one thread/process is required to control all open websockets simultaneously.
• No dependency to any other asynchronous event driven framework, such as Tornado, Twisted or Node.js.

That way the user can just start their server and not worry about also running the Tornado server.py.

[hagsteel]

It would be nice for development but it feels like a low priority.

When I want to run both Django and SD in one terminal I create a run.sh file and add the following:

#!/bin/bash

python server.py &
python manage.py runserver &

trap "kill -TERM -$$" SIGINT
wait

(this solution does not work well with pdb / ipdb).

[cancan101]

Any reason not to take advantage of a framework such as django-websocket-redis for the communication rather than using tornado+socketio?

[hagsteel]

Tornado is good for dealing with web sockets and async stuff.

SwampDragon is partially a result of me doing lots of different web socket projects over time.
I used to go with gevent and Socket.io but over time I've come to like SockJS more.

It's also good to be able to separate the two so you can serve your Django views from one server and SD on another server. You should (in theory, as I haven't fully tested this yet as there has been no need) be able to load balance SD as well.

This lends it self to doing things like having Django serving up a couple of light views and having more powerful servers for SD.

In short: I like working with Tornado and SockJS over Gevent and Socket.IO

[cancan101]

If I understand the full architecture, it would look something like:

(N x Django) <---> Redis <---> (M x Tornado+Socket.io) <---ws---> browsers

where you can scale up both the Django or the Tornado servers?

[ghost]

Well I think the author is using SockJS and not socket.io. I'll defer to Jonah of course, but it seems like the scaling happens on the tornado side. SockJS is designed to be load-balanced; it uses what amounts to "sticky sessions" so that a load balancer can connect a browser's websocket consistently to the same server. So in other words, if you have 4 quad-core servers, you could run 4 tornado instances on each of them and stick a load balancer like ha-proxy or nginx in front of them. As you need more capacity, you just add servers/tornado instances behind the load balancer.

Anyway, that's my take, based on an admittedly cursory overview.

[cancan101]

Right, I mean SockJS:

(N x Django) <---> Redis <---> (M x Tornado+SockJS) <---ws---> load balancer <---ws--> browsers

The question is whether you can also load balance on the Django side, i.e. scale out to multiple instances of Django pushingto Redis and I think the answer is yes, but wanted to check.

[ghost]

My offhand response is "yes." I imagine you could follow the usual scaling rules for Django. Probably be pretty smooth unless you're Pinterest or someone similar.

I know nothing about scaling Redis, but I feel sure that, first, a single instance can handle a hell of a lot, and second, there must be a clustering facility of some kind.

[hagsteel]

Since the processes are actually separated you can easily scale either one.

You could put a couple of cores off for Django and a few off for SwampDragon (Tornado), If everything was merged into one process then this would be a problem.

To run both Tornado and Django using python manage.py runserver you would have to do some hacky bits to start both at the same time.

I would say for development purposes it would nice to have it, but the cost feels a bit too high.

I'll close this issue for now.