Invalid session (even with sticky sessions)

[bui]

My setup: Flask -> uWSGI + gevent (8 separate processes running via supervisord) -> nginx
RabbitMQ as a message queue

nginx config:

upstream socketio_nodes {
    ip_hash;

    server 127.0.0.1:34000;
    server 127.0.0.1:34001;
    server 127.0.0.1:34002;
    server 127.0.0.1:34003;
    server 127.0.0.1:34004;
    server 127.0.0.1:34005;
    server 127.0.0.1:34006;
    server 127.0.0.1:34007;
}

...

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $server_name;
        proxy_read_timeout 7200s;
        proxy_pass http://socketio_nodes;
        proxy_max_temp_file_size 50000m;
    }

    location /socket.io {
        proxy_connect_timeout 7d;
        proxy_send_timeout 7d;
        proxy_read_timeout 7d;
        proxy_buffering off;
        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_pass http://socketio_nodes/socket.io;
    }

Example uWSGI run command:

uwsgi --http :34000 --gevent 1000 --http-websockets --master --wsgi-file web.py --callable app --workers 1 --disable-logging

Flask-SocketIO initialisation:

SocketIO(app, engineio_logger=True, message_queue='amqp://', async_mode='gevent_uwsgi', ping_timeout=600)

(I have also applied gevent monkey patching at the start of the app)

This is the error that occurs on connection:

4455a8dd219f48c283c7c81128d06ba1: Sending packet OPEN data {'pingInterval': 25000, 'pingTimeout': 600000, 'upgrades': ['websocket'], 'sid': '4455a8dd219f48c283c7c81128d06ba1'}
4455a8dd219f48c283c7c81128d06ba1: Sending packet MESSAGE data 0
4455a8dd219f48c283c7c81128d06ba1: Received packet MESSAGE data 0/mynamespace,
4455a8dd219f48c283c7c81128d06ba1: Sending packet CLOSE data None
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask_socketio/__init__.py", line 42, in __call__
    start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/middleware.py", line 47, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/socketio/server.py", line 353, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/server.py", line 258, in handle_request
    socket = self._get_socket(sid)
  File "/usr/local/lib/python2.7/dist-packages/engineio/server.py", line 411, in _get_socket
    raise KeyError('Session is disconnected')
KeyError: 'Session is disconnected'
Invalid session 4455a8dd219f48c283c7c81128d06ba1
Invalid session 4455a8dd219f48c283c7c81128d06ba1

I've spent hours trying to figure out what's wrong, but I'm completely stumped now. Any ideas on what might be the issue here?

[bui]

FWIW there's another exception that can randomly occur instead of the previous KeyError, but it's always one of the two:

0ffc6d791a464caca2327e5953c5d73e: Sending packet OPEN data {'pingInterval': 25000, 'pingTimeout': 600000, 'upgrades': ['websocket'], 'sid': '0ffc6d791a464caca2327e5953c5d73e'}
0ffc6d791a464caca2327e5953c5d73e: Sending packet MESSAGE data 0
0ffc6d791a464caca2327e5953c5d73e: Received packet MESSAGE data 0/mynamespace,
0ffc6d791a464caca2327e5953c5d73e: Sending packet CLOSE data None
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/flask_socketio/__init__.py", line 42, in __call__
    start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/middleware.py", line 47, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/socketio/server.py", line 353, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/engineio/server.py", line 289, in handle_request
    self.disconnect(sid)
  File "/usr/local/lib/python2.7/dist-packages/engineio/server.py", line 203, in disconnect
    del self.sockets[sid]
KeyError: '0ffc6d791a464caca2327e5953c5d73e'

[jwg4]

Hi @bui

Firstly, you have --workers 1 in your uwsgi command line, is this right or should it be 8?

Secondly, I am not exactly sure of how nginx should be configured. But I have the directive
proxy_redirect off; in my config which I am correctly using to proxy socket.io connections. (The rest of my config looks very like yours.) Does this help you at all?

[bui]

1 uWSGI worker per process is correct, since the app needs to be available on multiple ports in order to use nginx load balancing. I believe this is what @miguelgrinberg recommends in this scenario.

proxy_redirect off in the nginx config did not fix the problem, unfortunately. I feel like this is more of a problem with the app server tracking sessions than with nginx, but I could be wrong.

[PremyslTalich]

I keep getting these two exceptions and I am running on flask-socketio development server (socketio.run(app) -> localhost:5000).

Any progress guys?

[jwg4]

@Kisslick can you post the output of pip freeze? Have you been able to reduce your app to something minimal and still get this error, and can you post details of your app, preferably with the code?

[miguelgrinberg]

@Kisslick @bui Are your clients connecting over WebSocket, or do they stay on long polling? If you are using a browser app as the client, you can look in the Network section of the browser's debugger. If you see a constant stream of requests, then the client is doing long polling. For WebSocket, there will be one WebSocket request in the list, and no new requests issued.

Also, do you get the same errors if you use the example application in this repository instead of your own?

Do you get the error if you deploy a single worker instead of 8?

[miguelgrinberg]

I"m going to assume this was caused by the disconnection bug in python-engineio I recently fixed. Try upgrading to python-engineio==1.6.0 and reopen the issue if you continue to experience problems.