Unable to setup Flask-SockeIO for Ubuntu + SSL + uWsgi + NGINX configured as load balancer

[ghost]

Hello,

I spend the last 3 days on this, reading a lot of documentation (there is not a lot regarding uWsgi + ngix+ssl+ load balancer(nginx)), but still didnt able to configure my app to connect to the Flask-SocketIO
with a configuration of :
Flask App (2 apps, each has different port/uwsgi socket)
uWSGI services ( 2 services in systemd configured)
Nginx configured for Load Balance between these 2 apps.

the flask app with flask-SockectIO app.py :

---

from flask_socketio import SocketIO, Namespace, emit, join_room, leave_room, close_room, rooms, disconnect
..
..
socketio = SocketIO(app, logger=True, engineio_logger=True, debug=True, cors_allowed_origins = None)
..

running the app directly
(kjVirtualEnv) $ python app.py
works fine!

running using uWSGI directly with command line - works fine !
(kjVirtualEnv) $ uwsgi --socket 0.0.0.0:7024 --http-websockets --protocol=http -w wsgi:app

the app deployed twice for horizontal scaling:
/home/berry/projects/python/kjBerry1
/home/berry/projects/python/kjBerry2

(with 2 different uwsgi ini files, 2 sockects , 2 services in systemd , etc.. )
they both work just fine also with NGINX - its just the Flask-SocketIO i still CANT setup to work

the nginx configured to use SSL (self signed certificate, the http to https redirect works fine)

using NGINX i get error only for the Web Sockets -
the app is working, nginx serving pages from kjBerry1 or kjBerry2 with the LOAD BALANCER setup just fine (but not for web sockets)
the errror is:
[error] 53946#53946: *24 peer closed connection in SSL handshake while SSL handshaking to upstream

PLEASE any answer/help or link to a blog or stack overflow that i didnt find will be appreciated

THANKS !

+--------------------------------------------------------------------------------------+
| kjBerry1.service
+--------------------------------------------------------------------------------------+
[Unit]
Description=uWSGI instance to serve kjBerry1
After=network.target

[Service]
User=berry
Group=www-data
WorkingDirectory=/home/berry/projects/python/kjBerry1
Environment="PATH=/home/berry/projects/python/kjBerry1/kjBerry1Venv/bin"
ExecStart=/home/berry/projects/python/kjBerry1/kjBerry1Venv/bin/uwsgi --ini kjBerry1.ini

[Install]
WantedBy=multi-user.target
+--------------------------------------------------------------------------------------+

uwsgi ini file : kjBerry1.ini

[uwsgi]
module = wsgi:app
master = true
processes = 1
socket = kjBerry1.sock
chmod-socket = 660
http-websockets = true
gevent = 1000
vacuum = true
logto = /home/berry/projects/python/kjBerry1/uwsgi-berry.log
die-on-term = true

nginx:

upstream uwsgi_berry {
    ip_hash;
    server unix:/home/berry/projects/python/kjBerry1/kjBerry1.sock;
    server unix:/home/berry/projects/python/kjBerry2/kjBerry2.sock;
}


server {
   listen 80;
   listen [::]:80;
   server_name ilmmrpi01.com www.ilmmrpi01.com;
   return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    include snippets/my-self-signed.conf;
    server_name sample.com www.sample.com;

    access_log /home/berry/projects/python/kjBerry1/kjberry1-nginx-access.log;
    error_log  /home/berry/projects/python/kjBerry1/kjberry1-nginx-error.log error;

    location / {
        include uwsgi_params;
        uwsgi_pass  uwsgi_berry;
    }

    location /socket.io {
        include proxy_params;
        proxy_http_version 1.1;
        proxy_buffering off;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass https://uwsgi_berry/socket.io;
    }

    location /static/ {
        root /home/berry/projects/python/kjBerry1/kjFlask;
        autoindex on;
    }
}

---

[miguelgrinberg]

It does not appear that you have SSL in your uWSGI processes, correct? Then this line:

proxy_pass https://uwsgi_berry/socket.io;

should be:

proxy_pass http://uwsgi_berry/socket.io;

You also seem to be mixing two different communication protocols between nginx and uWSGI. For the HTTP routes you are using uwsgi_pass and for Socket.IO you are using proxy_pass. I've never seen this, I suggest you move everything to proxy_pass. Once you get it to work that way, feel free to experiment and try other options, but that is what I have personally tested.

[ghost]

Miguel Thanks for your reply.

you suggest to move everything to proxy_pass.
location /socket.io --> proxy_pass --> ok

but I really dont know how to change this
location / {
include uwsgi_params;
uwsgi_pass uwsgi_berry;
}

to proxy_pass, because uwsgi_berry is the load balancing of
upstream uwsgi_berry {
ip_hash;
server unix:/home/berry/projects/python/kjBerry1/kjBerry1.sock;
server unix:/home/berry/projects/python/kjBerry2/kjBerry2.sock;
}

my guess is i need probably change this to
upstream uwsgi_berry {
ip_hash;
server http://0.0.0.0:7001;
server http://0.0.0.0:7002;
}
???
and than to replace socket = kjBerry1.sock with socket = 0.0.0.0:7001 ? in the ini file of the uwsgi

[uwsgi]
module = wsgi:app
master = true
processes = 1
socket = 0.0.0.0:7001 // remove this : kjBerry1.sock
chmod-socket = 660
http-websockets = true
gevent = 1000
vacuum = true
logto = /home/motim/projects/python/kjBerry1/uwsgi-berry.log
die-on-term = true

i dont like guessing, i preffer to read and learn, i wish there were a better documentation
however i was able to find documentation only for nginx, uwsgi, but not the how they integrate + how websockets can be configured

Thanks

[miguelgrinberg]

The Flask-SocketIO documentation shows working configurations for uWSGI and nginx. You can start from that and then adapt those configs to your needs one step at a time, making sure things continue to work with each small change you make: https://flask-socketio.readthedocs.io/en/latest/#uwsgi-web-server

[ghost]

I saw this documentation, but

1. in this example the load balance in the nginx server block is only for the socket.io
   while the "location / { proxy_pass http://127.0.0.1:5000; } " is for specific port 5000 only - no load balance.
2. in this example the uWSGI protocol is http, while
   the benefit of using uwsgi is to use wsgi protocol to communicate with nginx that also support wsgi natively
   the uwsgi protocol mention that this is the recommended way regarding performance.
3. your documentation show the nginx settings
   and for the uWSGI server you have only the command line:
   $ uwsgi --http :5000 --gevent 1000 --http-websockets --master --wsgi-file app.py --callable app
   How do i make the uWSGI server running
   2 processes of my application kjBerry1, kjBerry2
   in systemd ?
4. your documentation is for http,
   production application must be configured with https, i cant see documentation on this.

Miguel - I'm very sure as the developer of this amazing package that you already test it
is several architectures and deployments and I believe you have at least one code example you can share.
in my experience the demo application is prety small, few files and configuration for:
simple flask app with route to some dynamic page, serving static file like .js or even image
and a simple flask-socketio using one namespace,
then a simple example how to run this app in uWSGI server including configuring service in systemd for that
and finally a nginx server block with load balanacer.

as long i dont have a sample code for that, i'll keep my trials and probably my next steps
will be to configure systemd to run 2 services of the uWSGI with http port and not using .sock file in the *.ini file

this is from the docs:

"""
The uWSGI server is a fairly complex package that provides a large and comprehensive set of options.
It must be compiled with WebSocket and SSL support for the WebSocket transport to be available.
...
HOW ??? can you share link for documentation or sample code/configuration ?
...
The next example adds the support for load balancing multiple Socket.IO servers:
HOW we load balance BOTH http traffic for the Flask app and Load balancing for the Flask-SocketIO ?
"""

upstream socketio_nodes {
ip_hash;

server 127.0.0.1:5000;
server 127.0.0.1:5001;
server 127.0.0.1:5002;
# to scale the app, just add more nodes here!

}

server {
listen 80;
server_name _;

how to configure Load balancer ?
location / {
include proxy_params;
proxy_pass http://127.0.0.1:5000;
}

is this possible?
location / {
include proxy_params;
proxy_pass http://socketio_nodes/;
}

locaton /static {
    alias <path-to-your-application>/static;
    expires 30d;
}

location /socket.io {
    include proxy_params;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_pass http://socketio_nodes/socket.io;
}

}

[miguelgrinberg]

Unfortunately I don't have the time or the bandwidth to create documentation for projects other than my own. As I mentioned above, I provide minimal configuration examples that I have confirmed personally. It would be impossible for me to provide documentation for every possible way to configure nginx, uWSGI, etc.

the benefit of using uwsgi is to use wsgi protocol to communicate with nginx

As I said before, I do not know if you can use both the uwsgi and the http protocols in the same server instance. You'll have to experiment yourself with this, or ask on a uWSGI forum for assistance. I stand by my recommendation to switch everything to the http protocol, get everything to work, then start tweaking the configuration until you get what you want.

The uWSGI server is a fairly complex package that provides a large and comprehensive set of options.
It must be compiled with WebSocket and SSL support for the WebSocket transport to be available.

HOW ??? can you share link for documentation or sample code/configuration ?

I don't. This isn't my project and I'm not very familiar with it. When I needed to test with uWSGI I found what I needed from Google searches. Stack overflow in particular answers many of these questions.

HOW we load balance BOTH http traffic for the Flask app and Load balancing for the Flask-SocketIO ?

The syntax to load balance is the same for websocket and HTTP. The example in the docs shows how it is done for WebSocket, you can do the same for HTTP. If you intend to use the uwsgi protocol for your http and/or websocket traffic then you'll have to figure out how to do it yourself, I've always used the http protocol in my tests.

[ghost]

Miguel,

Thank you for taking the time to answer that.

I'll follow your recommendation to use the http protocol for both the flask app itself and the websockets.

Thanks again !

[ghost]

update:
there is solution for this question that is under testing + adding the redis message queue to this project and also emit an event from outside the app using the example in the docs:
socketio = SocketIO(message_queue='redis://')
socketio.emit('my event', {'data': 'foo'}, namespace='/test')

the steps to resolve that are:

1. follow Miguel advise, start from simple example and test the app step by step.
   first run the app using
   $ python app.py
   than configure wsgi (but DONT use the .sock file !) configure the app to run as uwsgi service but with port number
   note! you must add ssl support to the uwsgi, otherwise the websockets dont work.
   the uwwsgi service should run both http https ports
   for example:
   App1 ==> --https 4431 --http 5051
   App2 ==> --https 4432 --http 5052

last thing is to config NGINX as Load balancer with SSL listening to 443
means load balance on ports : 4431 & 4432
and finally add to nginx a redirect rule
to redirect all http traffic to https

its works !