Skip to content

Deployment

Jump to bottom
Martin Müller edited this page Dec 19, 2017 · 12 revisions

Deployment to production on AWS EC2 (Ubuntu 16.04)

A few things to note:

  • Needs at least a t2.small instance to work properly, however should also work on a micro
  • Create ssh keys first using ssh-keygen -t rsa -C "email@example.com" and add the public key to the deployment keys in this repo.

Install anaconda

cd && mkdir downloads && cd downloads
wget https://repo.continuum.io/archive/Anaconda3-5.0.1-Linux-x86_64.sh
bash Anaconda3-5.0.1-Linux-x86_64.sh -b
echo 'export PATH="~/anaconda3/bin:$PATH"' >> ~/.bashrc
source .bashrc
conda update conda
rm ~/downloads/Anaconda3-5.0.1-Linux-x86_64.sh

Pull repo

# cd && git clone git@github.com:salathegroup/crowdbreaks-flask-api.git && cd crowdbreaks-flask-api/ # If using ssh
cd && git clone git@github.com:salathegroup/crowdbreaks-flask-api.git && cd crowdbreaks-flask-api/
conda create --name flask-api
source activate flask-api
pip install -r requirements.txt
source deactivate
mkdir instance
cp config.py.example instance/config.py
# Add secrets to instance/config.py

# Download SVM binaries
curl https://s3.eu-central-1.amazonaws.com/crowdbreaks-dev/binaries/sent2vec_v1.0.p --create-dirs -o ~/crowdbreaks-flask-api/bin/vaccine_sentiment/sent2vec_v1.0.p

Redis

Mostly following this

sudo apt-get install build-essential tcl
cd /tmp
wget http://download.redis.io/redis-stable.tar.gz
tar xzvf redis-stable.tar.gz
cd redis-stable
make
sudo make install

Configure and run Redis:

sudo mkdir /etc/redis
sudo cp ~/crowdbreaks-flask-api/lib/configs/redis.conf /etc/redis
# Set password under requirepass 
# Note that this config is using port 6389

# Create service
sudo cp ~/crowdbreaks-flask-api/lib/configs/redis.service /etc/systemd/system/redis.service
sudo adduser --system --group --no-create-home redis
sudo mkdir /var/lib/redis
sudo chown redis:redis /var/lib/redis
sudo chmod 770 /var/lib/redis
sudo systemctl start redis
sudo systemctl enable redis

See status using sudo systemctl status redis

Gunicorn webserver

Mostly following this: this

Test whether stuff is working: cd ~/crowdbreaks-flask-api && gunicorn --bind 0.0.0.0:5000 wsgi:app

Create system daemon script

sudo cp ~/crowdbreaks-flask-api/lib/configs/flask-api.service /etc/systemd/system/flask-api.service
sudo systemctl start flask-api
sudo systemctl enable flask-api

Make sure server is running properly: sudo systemctl status flask-api

Nginx

sudo apt-get install nginx
sudo cp ~/crowdbreaks-flask-api/lib/configs/api-stg.crowdbreaks.org /etc/nginx/sites-available/
sudo ln -s /etc/nginx/sites-available/api-stg.crowdbreaks.org /etc/nginx/sites-enabled
# configure elasticsearch-dev.crowdbreaks.org in order to use kibana through nginx
sudo systemctl restart nginx
sudo rm /etc/nginx/sites-available/default
sudo rm /etc/nginx/sites-enabled/default

SSL certificate with Let's encrypt

Following this. In order to use basic auth on the Flask API endpoint, we should use SSL certificates on the EC2 instance. Make sure both port 80 and port 443 are open for inbound traffic in the AWS security group.

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx -y
sudo certbot --nginx -d api-stg.crowdbreaks.org

This should automatically modify the nginx config as well as add all necessary certificates. The current certificate will need to be renewed every 2 months. Run /opt/letsencrypt/letsencrypt-auto —debug or set up a cron job to renew certificates.

Logstash

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get -y install oracle-java8-installer
echo "deb http://packages.elastic.co/logstash/2.3/debian stable main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install logstash
# allow write on config files folder by Flask
sudo chown -R www-data:www-data /etc/logstash/conf.d/

Clone this wiki locally