deployment.md

Depoyment

This document describes how to get your Science Flask app to AWS. Before we start, I highly recommend this great tutorial. Also you should probably have a look at this and this, especially if you get stuck somewhere in the process.

We'll start a new Ubuntu EC2 instance and install apache webserver with mod_wsgi HTTP server module.

Then we'll install everything we need for Science Flask, instantiate the database, set up Celery and RabbitMQ and by the end of it, we'll have a running web-app in less than an hour.

1. Register to Amazon AWS.

2. Follow this tutorial and launch an EC2 instance. I used Ubuntu Server 16.04 LTS (HVM), SSD Volume Type AMI and launched a medium istance which is quite expensive but it's much easier to install all Python packages on that one then scale it down, otherwise we run into issues like this

3. Create an elastic IP for your instance following this so you can reconnect to it even if you had to restart it from the AWS console.

Elastic IPs have a terrible name, in fact they are quite the opposite. They are static and will stay attached to your instance even after a shutdown. This is NOT true for the IPv4 public IP address you get when you launch an instance. This IPv4 is allocated randomly and will change if you restart your instance, which means you'll need to reconfigure your webserver every time if you need to restart/scale your instance. This is obviously no fun, so elastic IPs come to the rescue. Once you've assigned an elastic IP to your instance, it's IPv4 address will be the same even after restarts.

Once you have it, note it down, we'll use it later in almost every step.

4. Connect to your EC2 instance via SSH as shown in the article. If you've used ubuntu AMI, then your username will be ubuntu so you should connect with

ssh -i ~/.ssh/your_key.pen ubuntu@elastic_ip_of_your_EC2

5. Great, let's install Apache webserver, with mod_wgi and pip for Python package management.

sudo apt-get update
sudo apt-get install apache2
sudo apt-get install libapache2-mod-wsgi
sudo apt-get install python-pip

6. Now clone the Science Flask repo and register the app with apache and make a few folders:

git clone https://github.com/danielhomola/science_flask.git
sudo ln -sT ~/science_flask /var/www/html/science_flask
mkdir logs
mkdir failedAnalyses
mkdir userData

If you have changed the name of the app from science_flask, make sure you update frontend/frontend.wsgi accordingly.

7. Install all Python packages that we need for Science Flask:

cd science_flask
sudo pip install -r requirements.txt

Now it's a good time to scale back your EC2 instance to a cheaper/smaller one. I used the nano instance which is the cheapest/smallest. Here's how to do it.

Once that's done reconnect via SSH.

Note, your public IP is now different but the elastic IP should still work and you should be able to connect you your instance via SSH with the same ssh command that we used in point 4.

8. We want to serve our users through a secure HTTPS connection. If you have a domain, you can get a free SSL certificate from Amazon or Let's Encrypt

If you don't have a domain and just want to use the elastic IP generated by AWS to point to your service, follow this tutorial to roll your own SSL certificate for free on an Ubuntu server.

Feel free to skip Step 3. of this tutorial (Firewall). The rest should be the same.

Ignore the end of Step 2. and Step 6. as well, we'll deal with that in the next point.

Note: you should use your public_IPv4_address_for_your_EC2 to setup default-ssl.conf. This will contain your elastic IP address if you've done everything right in step 3.

9. Create config file for Apache:

sudo nano /etc/apache2/sites-enabled/000-default.conf

Delete the contents of this file (Ctrl + K will delete an entire line in nano). Then copy and paste the following:

<VirtualHost *:80>
       ServerName public_IPv4_address_for_your_EC2
       ServerAdmin admin@science_flask.com
       DocumentRoot /var/www/html
       Redirect permanent / https://public_IPv4_address_for_your_EC2
</VirtualHost>

<VirtualHost *:443>
       ServerName Then follow this to add TCP port in EC2 security groups
change config.py to CELERY_BROKER_URL = 'amqp://<username>:<password>@public:5672//'

       ServerAdmin admin@science_flask.com
       DocumentRoot /var/www/html
       WSGIDaemonProcess frontend user=ubuntu group=ubuntu processes=1 threads=1
       WSGIScriptAlias / /var/www/html/science_flask/frontend/frontend.wsgi

       <Directory /var/www/html/science_flask/frontend>
           WSGIProcessGroup frontend
           WSGIApplicationGroup %{GLOBAL}
           Order deny,allow
           Allow from all
       </Directory>

       <Directory /var/www/html/science_flask>
           WSGIProcessGroup frontend
           WSGIApplicationGroup %{GLOBAL}
           Order deny,allow
           Allow from all
       </Directory>

       # Static Directories
       Alias /static /var/www/html/science_flask/frontend/static/
       <Location "/static">
               SetHandler None
       </Location>

       # SSL Stuff
       SSLEngine On
       SSLCertificateFile      /etc/ssl/certs/apache-selfsigned.crt
       SSLCertificateKeyFile /etc/ssl/private/apache-selfsigned.key
       <Location />
           SSLRequireSSL On
           SSLVerifyClient optional
           SSLVerifyDepth 1
           SSLOptions +StdEnvVars +StrictRequire
       </Location>
</VirtualHost>

Make sure you actually go through this file and make sure everything makes sense for your app.

Note: your public_IPv4_address_for_your_EC2 should contain your elastic IP address.

10. Customize frontend/config_example.py and rename it to frontend/config.py

    1. Generate a secret key for your app like this
    2. Setup the username, email, password for the admin. You can then log in with these credentials and go to the Admin profile from the Profile page. Then you can edit all the tables of the database from online.
    3. Setup mail sending. You can use AWS's SES service if you have a domain name, (that's what I did for CorrMapper and for this demo). Alternatively you can use Gmail also. You'll need to allow less secure apps to use your account for this to work which might not be a great idea. You can also create an app-key.

11. If you'd like to use Alembic to migrate your database if you update it's schema, then read this blog post and the docs here and do:

./manage.py db init
./manage.py db migrate
./manage.py db upgrade

12. We are getting close. Let's install SQLite and create the database for the app.

sudo apt-get install sqlite
sudo apt-get install python-tk
python db_create.py

13. Install RabbitMQ and setup a user with it, add it to config.py

sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user <username> <password>
sudo rabbitmqctl set_permissions -p / <username> ".*" ".*" ".*"

Change config.py to

CELERY_BROKER_URL = 'amqp://<username>:<password>@public_IPv4_address_for_your_EC2:5672//'

Start the service with:

sudo rabbitmq-server

If you get an error saying it's already running, that's OK.

14. Update security group of your EC2 instance to add inbound traffic for HTTP, HTTPS and RabbitMQ:

    • go to Security Groups and edit the Inbound rules of the security group of your instance.
    • add the following:
      • SSH TCP 22 0.0.0.0/0
      • HTTP TCP 80 0.0.0.0/0
      • HTTPS TCP 443 0.0.0.0/0
      • Custom TCP Rule TCP 5672 0.0.0.0/0
      • Custom UDP Rule UDP 5672 0.0.0.0/0

15. Setup celery

Celery would need to run as a daemon process in the background. For this we'll use the very powerful supervisord package. It can do much more than this so definitely have a look at the docs.

Here's a quick tutorial on how to setup supervisor with Celery.

Science Flask repo already has a supervisord.conf file that should work out of the box if you've followed everything till this point.

Let's start supervisord by typing supervisord in science_flask folder.

When you try to start supervisord and get an error like this:

Error: Another program is already listening on a port that one of our HTTP servers is configured to use.

follow this trick

You can restart celeryd process via the supervisorctl like this:

supervisorctl restart celeryd

16. You can check Apache's logs like this:

less -S /var/log/apache2/error.log
less -S /var/log/apache2/access.log

Celery and Science Flask also logs. These are in ~/science_flask/logs. These are very useful to check on the status of the app.

17. Restart server and enjoy!

sudo systemctl restart apache2

You should be able to log in with the admin credentials you specified in frontend/config.py. Go to the Profile page and then to the Admin panel (botton right corner).