tornadio cluster behind a load balancer

[specialunderwear]

Hi I am trying to set up a cluster of tornado servers behind a load balancer and noticed that SessionContainer is just a dict. This means I MUST send clients to the same server every request. Maybe you already thought about this? What I want to do is implement a memcached session backend so it doesn't matter on which server you end up for each request.

Or maybe you've got some other ideas?

[mrjoes]

First of all - check sockjs-tornado, because SockJS is more stable than Socket.IO and has built-in support for load balancing using sticky sessions (clients will be redirected to the same node).

Problem is not really in session backend, it is in your application logic. And I'm not sure that session container running against MemCached will help and will perform very well.

If your application does not work with user groups (in any form), you can use decent load balancer that can do sticky sessions (all subsequent requests for a user will go to same server). Load balancer will balance active connections against your servers, so it does not matter if users go to same server between requests or not.

If your application works with user groups, well, you will have to reimplement your server logic.

Let's check simple example: chat server.

Properties of the chat:

1. Users
2. Rooms
3. Private messages

Lets say all users have unique id (database PK, for example).

In case of multiple server nodes, you will have to figure out how to pass messages between servers, so users on ServerA will see messages sent by users connected to ServerB.

So, you need to add a meeting point: whenever UserA sends message to the RoomA, you need to broadcast to all users that are connected to the same server and in RoomA. Then you need to push message to rest of the servers. They will pick it up and will send it to their own RoomA users.

If UserA sends message to UserB, you check if UserB is on the same server. If he is, send it over. If no - push to the message bus, server with the user will pick it up and send.

And so on.

As you can see, in this case, having SessionContainer won't help. You need to have your own logic on top of it.

[specialunderwear]

I'm afraid that sticky sessions is not an option for us.

Correct me if I'm wrong but isn't the session only used to choose a transport method and as soon as the socket is established it is no longer needed. Only in case the user has no websocket/flash support the session is used. Eg. long polling, polling.

passing messages from users across servers is not a problem at all in our application.

[mrjoes]

I see. If it is not a secret, why sticky sessions are not an option?

Yes, session is only used to persist data between polls. However, session object works together with your connection object. If you have any properties in your connection class (some kind of state), you will have to rewrite it to use session too.

Plus, you will need to change how it works with the session: for incoming requests it should check with memcached, create new instance of the SocketConnection class and then call necessary callbacks. So, your application will become completely stateless.

Can't say it is very hard to accomplish, but it is not supported.

On a side note, if you go with memcached sessions you won't have on_close callback working most of the time.

[specialunderwear]

Thank you for your response very enlightening. We wanted to use lvs for load balancing because that would be super efficient when using websockets.

Getting long polling/polling to work in that scenario would be very hard indeed if SocketConnection holds state. This is not nessescerily so in our application though. We could have a problem that a message for a user can be lost if he switches server before delivery, because it is directed to a specific server.

I guess that using tornado prefork is also out of the question then.

I couldn't quite make what you meant with your last remark. Why would on_close callback not work? Is it because the socket remains open with long polling for some time?

[mrjoes]

Well, LVS is just solution for fail-over and you can use load balancer on top of it. Like HAProxy. See here, for example: http://serverfault.com/questions/173391/lvs-vs-haproxy-which-should-i-choose

HAProxy supports websockets natively and can do sticky sessions.

Anyway, regarding load balancing. User won't switch server before getting reply from the server he's on. For example, for xhr-polling transport, he's making AJAX request to the server, waiting for response, gets or not gets the data, closes connection. So, if there was no data for the user during his poll, he won't receive anything.

As for the on_close: if memcached will expire session, your server(s) won't know about it and won't call the on_close method in your class.