More django examples

[sontek]

More people would be attracted to gevent-socketio if we showed more django support. Currently django-socketio relies on a very old version and is really no longer needed with our new API.

[dillongreen]

Good idea, some Django example(s) would get more people interested. Maybe some geocoding with Google Maps (JavaScript running in the browser), send the latitude/longitude values to the server (Django), store them, and send lat/lng values back to the browser based on a user query, and then draw some markers on the map. I actually have a use case for that using mongoengine (storing lat/lng values on MongoDB for its geo-spatial query capabilities).

[dswarbrick]

I looked at using django-socketio, before deciding that it was too outdated, and I didn't need most of the stuff it had. I just hooked up gevent-socketio to my existing Django app, and it works very well, except for one problem that I've just encountered in the last few days.

Up until recently, none of my websocket views hit the database. Now, I have a couple that do some very simple queries, and what I've found is that I'm getting postgres "idle in transaction", because Django does not finish the transaction (or close the DB connection) until a view returns.

Since websocket connections are long-running connections, Django basically never finishes the transaction (even for SELECT-only queries) or closes the DB connection (if you manage transactions manually), for the duration that the client is connected to the websocket. As you can imagine, this will potentially be a problem if every websocket connection has a constantly open DB connection.

This is obviously more a Django problem than a gevent-socketio problem, but I think it would be worth mentioning this to prospective developers. At the moment, I'm working around it a in a fairly brutal fashion by manually calling db.close_connection() after any queries in websocket views.

[sontek]

@dswarbrick Yeah, that makes sense. We could definitely make an extension that does the transaction handling manually, so it would open, try/except, rollback or commit on every single call to an event in socket.io. That way django-socketio still isn't needed but you get the transactional stuff still

[neokeats]

@dswarbrick did you managed to connect gevent-socketio and the django app whitout having to have them both launch by the same server/process ?
i'm thinking about using 0mq for that in order to keep thing separated...

[dillongreen]

@neokeats you say you're thinking zmq... https://github.com/mozilla-services/circus/ by any chance? That's what I thought I might do.

[neokeats]

@markusgattol no simply something like that https://github.com/sdiehl/zeromq-chat using the last api of gevent-socketio

[neokeats]

i don't think circus will work or will be easy to make it run on windows

[dswarbrick]

@neokeats I use a modified version of django-socketio's "runserver_socketio" as my dev server, and run the whole app under Gunicorn in production, using an SSL-enabled version of GeventSocketIOWorker. It should theoretically be possible however to split the websocket server out of the Django server process, so long as both are using the same session storage backend. What you will run into is potential problems if trying to run a "chat" type websocket application, with multiple worker / child processes. There needs to be some common lobby / user roster sharing between the child processes, as well as message passing between them, if for example person A on process 123 sends a message to person B on process 456. The simple_chat example app in the repo, which stores a user roster in the socket's parent server process, only works for a single-process setup.

[neokeats]

@dswarbrick i thought that was the purpose of things like 0mq
person A on process 123 sends a message to something "tcp://127.0.0.1:5000" and listen to messages from it
person B on process 456 was listening to "tcp://127.0.0.1:5000", so he'll get the message...

[neokeats]

@dswarbrick i thought that was the purpose of things like 0mq
person A on process 123 sends a message to something "tcp://127.0.0.1:5000" and listen to messages from it
person B on process 456 was listening to "tcp://127.0.0.1:5000", so he'll get the message...
https://github.com/sdiehl/zeromq-chat/blob/master/apps/chat/views.py

i'm thinking of that in order to have realtime as services
i have severals django sites and some of realtime need to be shared accross site, therefore listening internally to the same source will do trick...

[neokeats]

seems, i got it to work
https://github.com/neokeats/gvent-socketio-django-zmq

first launch

 #will act as "main" server      
 python chat_zmq.py 

and

 #django site
 python run.py 

if you add a second django site as long as the chat register to the same zmp PUB and SUB, the chat will be common.

[dswarbrick]

@neokeats Is the chat_zmq.py really necessary? I haven't used zmq before, and only read a little about it, but I was under the impression that you could simply have multiple websocket servers connect directly to a zmq daemon, publish messages to it, and those messages will be heard by any subscribers. What exactly is chat_zmq.py doing, in your case, that could not be done in the websocket process?

[neokeats]

i m new to zmq but from what i understand there is no daemon...

http://www.zeromq.org/area:faq

How do I start the ØMQ daemon?
There are no services or daemons to start, unless you build them yourself. ØMQ is a library.
Compile and link an > example and run it. ØMQ applications speak to each other directly.

that what's do chat_zmq.py it start a daemon and brodcast to subscriber what it receives

[dswarbrick]

Ah ok... I was thinking it was more like ActiveMQ or RabbitMQ.

[wuzuf]

Not sure if it is exactly what you were looking for but I have adapted the chat example from django-socketio to work with the latest version of gevent-socketio here: https://github.com/wuzuf/django-socketio.
It is quite easy to set up and to get running. (there is a dependency on my modified version of django-socketio, though)

[sontek]

@wuzuf Want to separate out the chat example into its own project so that we can include it as a good way to use gevent-socketio?

[wuzuf]

@sontek I have just submitted a pull request adding a new example. I removed all the dependencies to django_socketio. It should hopefully be easy to set up, the (very) short documentation to start it is in the README.rst. Let me know if you believe it deserves more polishing.
One thing I forgot to mention, is that of course it works only when you have one single process/worker. I will try to work on a ZMQ example when I have time.

[vinodc]

I know this is a little late, but for anyone who arrived here while searching for how to resolve idle transaction issues, I've started using Django with autocommit enabled, and connecting to PgBouncer with transaction pooling. This combination prevents IDLE connections in the DB.