Skip to content

Deployment Automation (resume)

Jump to bottom
Severin FERRAND edited this page Dec 14, 2017 · 13 revisions

Overview

In this section I will explain how I automated deployment of my website (resume) across multiple hosting solutions.

Here is the scenario:

  1. website source file are hosted on a git (github) repository (In this specific case my resume website source files are stored in https://github.com/Seburan/resume)

  2. developer is developing website, working with branches (develop branch etc.). When dev/test is complete, developer will merge changes to master branch and push changes to repository.

  3. Repository Webhook is configured to trigger upon a push event. (More documentation about https://developer.github.com/webhooks/)

  4. Repository Webhook will call a Webhook URL we have previously configured on one of the hosting service running nginx.

  5. the Webhook URL will be hosting a simple script, this script will analyze the event payload (it's signature based on the secret key, and its content).

  6. If event payload matches the criteria (push on master, legitimate event from authenticated repository) then it will trigger the deployment script.

  7. Deployment script will perform the following activities:

7.1. Pull from Github to Local Nginx (Website hosted @ https://resume.severin.jp )

7.2. Push to https://github.com/Seburan/seburan.github.io repository (Website hosted @ https://github.severin.jp )

7.3. Sync to S3 Bucket s3://aws.severin.jp (Website hosted @ http://aws.severin.jp )

Prerequisites

Configuration Files

  • /etc/nginx/
    • nginx.conf
    • global/
      • letsencrypt.conf
      • restrictions.conf
      • fcgiwrap.conf
    • sites-available
      • resume.severin.jp
    • sites-enabled
      • resume.severin.jp

Website Deployment from GitHub repository

Configure deployment keys

I want to be able to deploy website content using www-data user, so we will assign a Github SSH deployment key to www-data and configure this Deploy key in the repository containing our website resources.

based on documentation fro https://gist.github.com/zhujunsan/a0becf82ade50ed06115

First generate a key from my user:

$ ssh-keygen -t rsa -b 4096 -C "deploy@severin.jp"

saved as .ssh/git_deploy_id_rsa

Create a config file for ssh client under ~/.ssh/config

Host github.com
	HostName github.com
	User git
	IdentityFile ~/.ssh/git_deploy_id_rsa

Then copy the config to www-data ~/.ssh (see cat /etc/passwd if you don't know where his home dir is)

$ sudo mkdir /var/www/.ssh
$ sudo cp git_deploy_id_rsa* /var/www/.ssh/
$ sudo cp config /var/www/.ssh/
$ sudo chown -R www-data:www-data /var/www/.ssh
$ ll /var/www/.ssh/

Private key git_deploy_id_rsa should have a 600 permission :

total 12
-rw-r--r-- 1 www-data www-data   87  9月 28 22:14 config
-rw------- 1 www-data www-data 3243  9月 28 22:14 git_deploy_id_rsa
-rw-r--r-- 1 www-data www-data  743  9月 28 22:14 git_deploy_id_rsa.pub

we can now confirm :

$ sudo -u www-data ssh -T git@github.com

we should get a successfully authenticated message :

The authenticity of host 'github.com (192.30.255.112)' can't be established.
RSA key fingerprint is 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.255.112' (RSA) to the list of known hosts.
Hi Seburan/resume! You've successfully authenticated, but GitHub does not provide shell access.

Deploy content locally

Well this is pretty straightforward, in just one command:

Initial deploy, i am cloning the repository:

$ sudo -u www-data git clone git@github.com:Seburan/resume.git /web/resume.severin.jp/www/

Further deploy, I will pull the content:

$  cd /web/resume.severin.jp/www/
$  sudo -u www-data git pull origin master

Deploy on remote repository

Second step, we want to deploy the website content on https://github.com/Seburan/seburan.github.io which is a free hosting solution gracely offered by Github to host a static website (this may be for a github project or not actually).

what we'll do here is very simple. If you have followed the first part and already cloned the repo, we will just add a second remote and push the changes there.

before adding a second remote we need to register deployment user's public key (here www-data) into the target repository Deploy Keys, and select Allow write access.

then we can add a remote based on the documentation https://help.github.com/articles/adding-a-remote/

Let's check current remote repositories:

$  cd /web/resume.severin.jp/www/
$ sudo -u www-data git remote -v
origin	git@github.com:Seburan/resume.git (fetch)
origin	git@github.com:Seburan/resume.git (push)

then let's add the remote repository for hosting the live version:

$ sudo -u www-data git remote add live git@github.com:Seburan/seburan.github.io.git

then again check the remote repositories:

$ sudo -u www-data git remote -v
live	git@github.com:Seburan/seburan.github.io.git (fetch)
live	git@github.com:Seburan/seburan.github.io.git (push)
origin	git@github.com:Seburan/resume.git (fetch)
origin	git@github.com:Seburan/resume.git (push)

Finally we can push the changes to live version.

based on https://help.github.com/articles/pushing-to-a-remote/

$ sudo -u www-data git push live master:master

Deploy on AWS S3 bucket

ToDo: configure AWS credentials under deployment user's home

 $ ll .aws/
total 8
-rw------- 1 severin severin 118 11月 12 13:02 config
-rw------- 1 severin severin 246 11月 12 13:02 credentials

will use the following command to synchronize the local folder with the s3 bucket hosting the website.

Note that the commence will actually only include files type (based on extension) required for the live version.

ws --profile site-admin@severin.jp s3 sync --acl public-read --delete --exclude "*" --include "*.html" --include "*.css" --include "*.js" --include "*.jpg" --include "*.ico" --include "*.eot" --include "*.otf" --include "*.svg" --include "*.ttf" --include "*.woff" --include "*.woff2" ./ s3://aws.severin.jp

Deployment Script

ToDo: upload script update_resume_www-data.sh

#!/bin/bash

# pull latest version from github

cd /web/resume.severin.jp/www/
sudo -u www-data git pull origin master

# push to seburan.github.io
sudo -u www-data git push live master:master

# sync to aws
# this command will synchronize current folder with target s3 bucket
# --acl public-read will allow public read for new files
# --delete (boolean) Files that exist in the destination but not in the source are deleted during sync.
# --exclude "*" will exclude everything unless it is specified in --include
# --include the following file types : html, css, js, jpg, ico, eot, otf, svg, tff, woff, woff2
sudo -u www-data aws --profile site-admin@severin.jp s3 sync --acl public-read --delete --exclude "*" --include "*.html" --include "*.css" --inc
lude "*.js" --include "*.jpg" --include "*.ico" --include "*.eot" --include "*.otf" --include "*.svg" --include "*.ttf" --include "*.woff" --inc
lude "*.woff2" ./ s3://aws.severin.jp

if everything OK, the execution should show something like this:

$ sudo -u www-data ./deploy_resume.sh 
From github.com:Seburan/resume
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Everything up-to-date

Webhook

  • ToDo: setup webhook to run script deploy_resume.sh

source: https://blog.sleeplessbeastie.eu/2017/09/18/how-to-execute-cgi-scripts-using-fcgiwrap/

install FastCGI wrapper

we need to install the FastCGI component:

$ sudo apt-get install fcgiwrap

let's check it is running:

$ sudo service fcgiwrap status
● fcgiwrap.service - Simple CGI Server
   Loaded: loaded (/lib/systemd/system/fcgiwrap.service; static)
   Active: active (running) since 水 2017-12-13 15:54:02 JST; 55s ago
 Main PID: 13851 (fcgiwrap)
   CGroup: /system.slice/fcgiwrap.service
           └─13851 /usr/sbin/fcgiwrap

12月 13 15:54:02 severin-pi systemd[1]: Starting Simple CGI Server...
12月 13 15:54:02 severin-pi systemd[1]: Started Simple CGI Server.

Prepare the script to trigger

we prepare the directory which will contain cgi script, a best practice for Debian system is to locate these cgi scripts under /usr/lib/cgi-bin. we could have put the script in the /web location but this might be dangerous as a user could in some case upload some file in the web dir.

$ sudo mkdir -p /usr/lib/cgi-bin
$ sudo chown www-data:www-data /usr/lib/cgi-bin

then let's prepare a sample script bashcgiscript.cgi. we will replace by our final update_resume_www-data.sh once the basic configuration has been tested.

#!/bin/bash

echo "Content-type: text/html"
echo ""

echo "<html>"
echo "<head>"
echo "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">"
echo "<title>Bash CGI script</title>"
echo "</head>"
echo "<body>"
echo "<p>Hello, Your IP address is $REMOTE_ADDR</p>"
echo "<pre>"
env
echo "</pre>"
echo "</body>"
echo "</html>"

exit 0

we have to make the script executable:

$ sudo -u www-data chmod +x /usr/lib/cgi-bin/bashcgiscript.cgi

Nginx configuration

We will copy the sample provided in FastCGI Wrapper documentation:

$ sudo cp /usr/share/doc/fcgiwrap/examples/nginx.conf /etc/nginx/global/fcgiwrap.conf

we will change a few things so we can reuse this configuration for any other sites: location to scripts

$ cat /etc/nginx/global/fcgiwrap.conf

# fcgiwrap
location ~ \.cgi$ {

  # Zero-day exploit defense.
  try_files $uri =404;

  # Determines whether FastCGI server responses with codes greater than or equal to 300 should be passed to a client or be intercepted and redi$
  fastcgi_intercept_errors on;

  # Disable gzip (it makes scripts feel slower since they have to complete
  # before getting gzipped)
  gzip off;

  # Set the root to /usr/lib (inside this location this means that we are
  # giving access to the files under /usr/lib/cgi-bin)
  root  /usr/lib/cgi-bin;

  # Fastcgi parameters, include the standard ones
  include /etc/nginx/fastcgi_params;

  # Adjust non standard parameters (SCRIPT_FILENAME)
  fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;

  # Fastcgi socket
  fastcgi_pass  unix:/var/run/fcgiwrap.socket;

}

then finally we update the website configuration and add a directive to specify the exact script path we want to allow execution. (we can allow any *.cgi or only specific path if you don't want to allow execution of any script under /usr/lib/cgi-bin)

$ cat /etc/nginx/sites-available/resume.severin.jp

        # Specify location of Maintenance script
        # Include the default FastCGI conf location to be able to execute scripts under /usr/lib/cgi-bin/
        location ~ \.cgi$ {
                include global/fcgiwrap.conf;
        }

we are good, check configuration and reload nginx.

$ sudo nginx -t
$ sudo service nginx reload

Final Web hook script

now we are read to proceed with web hook script. For reference, you can view some example of Bash Shell CGI Script here http://www.yolinux.com/TUTORIALS/BashShellCgi.html

I just added a logger instruction to log to syslog /var/log/messages and redirected the stderr output to stdout to display both outputs to the html page.

/usr/lib/cgi-bin/resume.severin.jp_update.cgi

#!/bin/bash

echo "Content-type: text/html"
echo ""

echo "<html>"
echo "<head>"
echo "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\">"
echo "<title>Bash CGI script</title>"
echo "</head>"
echo "<body>"
echo "<p>Hello Requester, Your IP address is $REMOTE_ADDR</p>"
echo "<pre>"

#logging
logger -s "$SCRIPT_FILENAME execution requested from $HTTP_HOST by $REMOTE_ADDR"
echo "User `whoami` is running the script."

# pull from master
echo ""
echo "Updating resume.severin.jp..."
echo ""
cd /web/resume.severin.jp/www/
git pull origin master 2>&1

# push to seburan.github.io
echo ""
echo "github.severin.jp..."
echo ""
git push live master:master 2>&1

# sync to aws
# this command will synchronize current folder with target s3 bucket
# --acl public-read will allow public read for new files
# --delete (boolean) Files that exist in the destination but not in the source are deleted during sync.
# --exclude "*" will exclude everything unless it is specified in --include
# --include the following file types : html, css, js, jpg, ico, eot, otf, svg, tff, woff, woff2
echo ""
echo "aws.severin.jp..."
echo ""
aws --profile site-admin@severin.jp s3 sync --acl public-read --delete --exclude "*" --include "*.html" --include "*.css" --include "*.js" --include "*.jpg" --include "*.ico" --include "*.eot" --include "*.otf" --include "*.svg" --include "*.ttf" --include "*.woff" --include "*.woff2" ./ s3://aws.severin.jp 2>&1

echo "</pre>"
echo "<p>Update complete</p>"
echo "</body>"
echo "</html>"

exit 0

You can run the file as www-data to check:

sudo -u www-data ./resume.severin.jp_update.cgi

or directly access the page from your browser to trigger the script (no worry, I'll change the name of the script to some name not easy to guess since there is not yet a measure to prevent someone to call it repeatedly)

https://resume.severin.jp/resume.severin.jp_update.cgi

then you should get a nice output like this if there was a change on the master branch on source repo:

Hello Requester, Your IP address is 162.158.178.196

User www-data is running the script.


Updating resume.severin.jp...

From github.com:Seburan/resume
 * branch            master     -> FETCH_HEAD
   56fff22..13416a9  master     -> origin/master
Updating 56fff22..13416a9
Fast-forward
 index.html | 1 +
 1 file changed, 1 insertion(+)

github.severin.jp...

To git@github.com:Seburan/seburan.github.io.git
   56fff22..13416a9  master -> master

aws.severin.jp...

upload: ./index.html to s3://aws.severin.jp/index.html
Update complete

Webhook setup on Github

This is the easiest part. Just go to your Github repository > settings > Webhooks and follow instructions from the documentation https://developer.github.com/webhooks/.

In this case we will select to trigger Webhook on push events only.

Improvements

A nice-to-have improvement would be to parse the payload sent by the webhook and verify the payload signature based on the shared secret. This would guaranty that the deploy script can only be triggered by Github Webhook and not an attacker who would the the url. In my case I have simply obfuscated the script name so an attacker would not be easily able to find it but this is NOT an elegant security measure.

Clone this wiki locally