Migrate application & database to the same server

[cuducos]

Currently Jarbas is a slow application for four reasons:

• Loading time is terrible beacuse it is hosted in a free tier at Heroku; this means Heroku might have to wake up the application when user reaches it
• No CDN for assets.static files, all is being served by Heroku (with WhiteNoise)
• Searching is slow because Heroku free tier just allows 10k rows in the database, thus we’re are hosting the app at Heroku but the database at AWS
• Due to the benefit-cost of paying a decent server doesn't seem to be priority to Serenata de Amor at this point

[cuducos]

As far as I can remember, the specifications would be something like that:

• Server with Python 3 and NodeJS
• Postgres (we use more than 2 million rows)
• Git
• nginx

[cuducos]

Almost everything set at our DigitalOcean droplet. I'm having trouble with /static/ — nginx is returning 403 Forbidden (e.g. http://45.55.37.68/static/app.css?8706439a).

The sites-available/jarbas nginx config reads:

…
location /static/ {
    alias /root/jarbas/staticfiles/;
}
…

The file permissions seems right:

root@serenata-jarbas:~/jarbas# ll
total 76
drwxr-xr-x   9 root www-data 4096 Sep 15 22:23 ./
drwx------  11 root root     4096 Sep 15 22:33 ../
drwxr-xr-x   2 root www-data 4096 Sep 15 22:23 bin/
drwxr-xr-x   2 root www-data 4096 Sep 15 20:56 contrib/
-rw-r--r--   1 root www-data  562 Sep 15 20:56 elm-package.json
drwxr-xr-x   4 root www-data 4096 Sep 15 21:39 elm-stuff/
drwxr-xr-x   8 root www-data 4096 Sep 15 22:24 .git/
-rw-r--r--   1 root www-data  172 Sep 15 20:56 .gitignore
drwxr-xr-x   6 root www-data 4096 Sep 15 22:23 jarbas/
-rwxr-xr-x   1 root www-data  804 Sep 15 20:56 manage.py*
drwxr-xr-x 188 root www-data 4096 Sep 15 22:18 node_modules/
-rw-r--r--   1 root www-data  288 Sep 15 20:56 package.json
-rw-r--r--   1 root www-data   39 Sep 15 20:56 Procfile
-rw-r--r--   1 root www-data 3272 Sep 15 20:56 README.md
-rw-r--r--   1 root www-data  143 Sep 15 22:23 requirements-dev.txt
-rw-r--r--   1 root www-data   57 Sep 15 20:56 requirements.txt
-rw-r--r--   1 root www-data   13 Sep 15 20:56 runtime.txt
drwxr-xr-x   4 root www-data 4096 Sep 15 22:18 staticfiles/

root@serenata-jarbas:~/jarbas# ll staticfiles/
total 532
drwxr-xr-x 4 root www-data   4096 Sep 15 22:18 ./
drwxr-xr-x 9 root www-data   4096 Sep 15 22:23 ../
drwxr-xr-x 6 root www-data   4096 Sep 15 21:20 admin/
-rwxr-xr-x 1 root www-data   3098 Sep 15 22:18 app.8706439a4744.css*
-rwxr-xr-x 1 root www-data    816 Sep 15 22:18 app.8706439a4744.css.gz*
-rwxr-xr-x 1 root www-data 208598 Sep 15 21:39 app.ac7397ecb8a7.js*
-rwxr-xr-x 1 root www-data  38857 Sep 15 22:18 app.ac7397ecb8a7.js.gz*
-rwxr-xr-x 1 root www-data   3098 Sep 15 22:18 app.css*
-rwxr-xr-x 1 root www-data    816 Sep 15 22:18 app.css.gz*
-rwxr-xr-x 1 root www-data 208598 Sep 15 22:18 app.js*
-rwxr-xr-x 1 root www-data  38857 Sep 15 22:18 app.js.gz*
drwxr-xr-x 6 root www-data   4096 Sep 15 21:20 rest_framework/
-rwxr-xr-x 1 root www-data   8950 Sep 15 22:18 staticfiles.json*

Any ideas?

[cv]

I'd strongly recommend not running stuff as root, or even having these files be owned by root. If there's a security issue somewhere, you're completely hosed.

My guess is that nginx, if you installed it from the apt-get package, is running as the www-data user, so maybe try chowning those files to that user as well, and see how things go?

[cuducos]

Many thanks for the advice of avoiding running as root. As a terrible devops I had completely forgotten about this risk. Gonna change that today.

The chown -R www-data:www-data staticfiles/ hasn't helped though. Gonna explore that later.

[cuducos]

Everything is working at http://45.55.37.68/ now. A short TODO list before closing this Issue:

• Files are owned by another user, not root, but processes (nginx and gunicorn) still ran by root: can someone more experienced confirm if this is ok (security-wise) or if is there another way to wire things up?
• One can deploy with a git pull to jarbas@45.55.37.68:/opt/jarbas.git (thanks to a Git hook), but I'm not sure if restarting gunicorn and/or nginx is needed after deploying; if it is how could we automatize that? Right now I started gunicorn with --reload, but I'm not sure if this is enough
• We might want to use a domain or a sub-domain (e.g. jarbas.datasciencebr.com) (cc @Irio)
• Before shutting down Heroku (or make it forward users to the new server) we need to update our collaborators and our documents with the new URL (cc @vilapedro)

[ElSaico]

• nginx should be ran by www-data, and it'd be wise to create a separate user for both owning the application folder and running gunicorn; static should still be accessible by nginx, though.
• Only gunicorn needs restarting. A possible way to automate this is by make it write a PID file somewhere (option -p or --pid) and then send it a SIGHUP at the end of deployment.

[Irio]

@cuducos I just created http://jarbas.datasciencebr.com/. Should be live soon.

[cuducos]

Closing this issue, as it's live, but gonna move some concerns to a more specific issue in a sec.