Allow deploy on uwsgi

[Wtower]

Current status

Currently the deployment assumes Apache with mod_wsgi. Apache is the only option for Plesk running on 80.

Incentive

By default mod_wsgi runs on 1 process with 15 threads. For an average project with an average render time 150msec, as the number of requests per minute goes up (eg 200) the throughput deteriorates rapidly until wsgi no longer serves when it runs out of threads and apache serves a static html if present or 403. This takes a couple of seconds to settle after rpm falls down (if it falls down).

Increasing the number of threads partially solves the 403 issue but still the mod_wsgi throughput is terrible, especially comparing to a node server.

This has lead into some investigation on how much better an alternative wsgi performs and how is it possible to use with Apache. Some hints from a relevant investigation reveal that uwsgi is a good choice for high throughput, which is reasonable since it is based on C. Indeed, after some tests with the same project, although render time is mostly the same (this should be optimised at application level), throughput for 200rpm even with only 1 process and 2 threads was stable.

About how to work with Apache, uwsgi recommends basically two alternatives: mod_uwsgi and mod_proxy_uwsgi. The first is unnecessarily complicated and unconventional, leaving the second as a good option. Unfortunately, this does not seem to work for older servers running Apache 2.2. But it seems that using plain mod_proxy works very well anyway, with the downside of losing some Apache functionality such as gz compression.

How to setup uwsgi

This is how uwsgi was installed on an older Ubuntu 12.04:

vf new -p python3.4 uwsgi
pip install -U pip setuptools wheel
pip install uwsgi

How to run

This is how uwsgi was started for the above evaluation:

uwsgi --chdir=/var/www/vhosts/(subscription host)/(domain)/ \
  --wsgi-file=(app)/wsgi.py \
  --env DJANGO_SETTINGS_MODULE=(app).settings_live \
  --master \
  --pidfile=/tmp/(app).pid --vacuum \
  --http=127.0.0.1:3031 \
  --processes=1 --threads=2 \
  --home=/var/virtualenvs/(app) \
  --stats 127.0.0.1:9191

This needs fine tuning and to use an ini file. More options here. The initial recommendation from Django docs was not helpful.

Note that the wsgi file that was required for mod_wsgi was not running. The simple Django generated wsgi.py was enough. Also note that because mod_proxy was used, the option --http was specified. Otherwise for mod_proxy_uwsgi --socket is required.

This was the apache conf under /var/www/vhosts/(domain)/conf/vhost.conf that was used:

ProxyPass /static !
ProxyPass /media !

Alias /static/ /var/www/vhosts/(subscription)/(domain)/static/
<Location "/static/">
    Options -Indexes
</Location>

Alias /media/ /var/www/vhosts/(subscription)/(domain)/media/
<Location "/media/">
    Options -Indexes
</Location>

<Location />
#  ProxyPass uwsgi://127.0.0.1:3031/
  ProxyPass http://127.0.0.1:3031/
  ProxyPassReverse http://127.0.0.1:3031/
</Location>

The above will be used in ansible.

Monitoring

Install uwsgitop with pip install uwsgitop. Run uwsgitop 127.0.0.1:3031. Measure memory with sudo pmam -x (pid).

More info on threads here. Also consider using emperor.

[Wtower]

It is worth noting for general interest that when using mod_wsgi and it runs out of threads, we can use a small static 'oops' page for apache to server in document root.

If using google analytics, we can track this root page with manually setting it to something different:

https://developers.google.com/analytics/devguides/collection/analyticsjs/single-page-applications

[Wtower]

After careful read of the mod_wsgi documentation, the line of thought above in the Incentive section is incorrect. mod_wsgi runs by default on embedded mode, where mod_wsgi runs inside the Apache processes, and Apache manages these processes based on its configuration.

The default Apache configuration for Plesk is prefork MPM and is optimised for serving php, but not suitable for Django. Therefore daemon mode is explicitly required for mod_wsgi. Therefore the performance deterioration experienced comes from the above fact.

In the case of mod_wsgi. Apache should queue up the requests if no process/thread is available, up to a certain limit (default 100, there is a setting) and then throw 50x gateway timeout. A custom page can be specified to serve in that case with the above analytics track.

Nevertheless, a uwsgi feature would be nice.