fullstack.org

full stack

1. We have one server that has the everything [viirtual private server]

and then the second server that houses the testing framework VPS has the OS. The OS has the Web Server[nginx] - it has an engine to run the code. For eg, we can use WSGI Server to run python code (eg. gunicorn), the web server can take the code/other dependencies from github etc. there are also the static files, key value datastores etc that are in the OS and are used by the server.

1. fabric - provides a convient python wrapper for ssh commands
2. ansible - configuration management automation tool
3. nginx - it is a web server
4. green unicorn - a WSGI server, web server gateway interface
5. WSGI is a standard defined by the python community to for deploying web applications to web servers. a server that impliments the WSGI standard is the WSGI server. django/flask etc are wsgi frameworks.

We must run our web application on a server or a cluster of servers called the production enviornment. we can rent a virtual server from AWS, Linode, DigitalOcean for eg. There are several options:

bare metal servers : physical hardware, eg dell, lenovo virtual private servers - shared portions of servers eg, Linode, DO IaaS - virtualized abstractions of resources, AWS, MSAzure PaaS - Abstracted execution enviornments eg Heroku, Amazon BeanStalk

Virtual Private servers - are virtualized slices of hardware which run on top of physical server. They use software like VMWare to run many different server instances on the same server.

when you create a new EC2 instance, you will get a key-value pair - it will be a .pem file. go to the file with the pem, then do this: ssh -i “fullstack-one.pem” ubuntu@ec2-52-90-52-115.compute-1.amazonaws.com it is ubuntu@public-dns of the instance

this wont work if your key is publicaly viewable. so, make it private by: chmod 400 fullstack-one.pem

then do the last step.

AWS creates the key-value pair for us directly, we dont need to do it explicitly.

secure your fresh server

apt-get install fail2ban

change UsePAM to true in : /etc/ssh/sshd_config

this will not allow root access to the server. now, we will create a non-root user and add it to a non-root group.

before that, I was getting a Write failed: Broken pipe error, this was when I did not query the server for sometime. To fix this, refer to: http://askubuntu.com/questions/127369/how-to-prevent-write-failed-broken-pipe-on-ssh-connection

so, use : sudo /usr/sbin/groupadd deployers to add deployers as the new group

/usr/sbin/groupadd deployers

mv /etc/sudoers /etc/sudoers-backup

(cat /etc/sudoers-backup ; echo “%deployers ALL=(ALL) ALL”) > /etc/sudoers

chmod 0440 /etc/sudoers

**OQ: not able to do this for some reason. check the pages 30-36 and implement that

you have the entire procedure in a script so that you don’t have to do this everytime you use a new VPS. we use Fabric for that. install it in a virtualenv using : pip install fabric==1.10.2 Execute the script from the local command line with fab bootstrap . You’ll be prompted for a password for the new user and then the script will connect to the server again with that user to complete the steps we walked through manually earlier this chapter.

So, we have Fabric - which is used to automate the server setting up process

Next up is the Operating System. Earlier we secured the server itself. Now, we will talk about the OS that runs on the server.

Ubuntu earlier used GNOME and now uses Unity for the GUI. We will need several Python libs to use the Python Stack

python-dev - header files and static library for Python python-virtualenv - to create and use Python virtualenvs for isolation git-core - for the git VCS nginx - a web server that will answer our incoming HTTP requests supervisor - will control the state of our WSGI and other programs

set up the firewall: our system still needs a firewall to lock down ports other than 22 (ssh), 80 (HTTP) and 443 (HTTPS).

sudo ufw allow 22 sudo ufw allow 80 sudo ufw allow 443

sudo ufw enable

we can automate this and more using Ansible. Ansible is an open source configuration management tool written in Python that will allow us to automate the steps we performed manually above. We’ll also build on the Ansible scripts, which are called “playbooks” in Ansible terminology.

earlier, we created a dir called `prod` which housed the fabric file [which set up a new user - deploy and then banned root access to the server]

Now, we will add Ansible files to the same dir. the ansible playbook will be an YAML file called deploy.yml we will also create a hosts file that’ll tell Ansible what server IP address we should deploy to.

we will basically have a dir structure that will house all the different parts of the automation process. for eg, there will be seperate scripts for database etc. then, there will be a main.yml that will orcherstrate the entire deployment process.

Now, that we have an Ansible script that will create a new deploy user, prevent root access, set up firewall, and more, we will start talking about Web Servers

Nginx is an example of a web server. recall, earlier, when we wanted to host pages, eg for jekyll, we used jekyll serve - this created a server and it hosted the static pages. Similarly, we have SimpleHttpServer in the Python API. So, inside the OS, we need a server that would handle the HTTP requests for the page - it would accept the GETs, process them and return JSON, XML, HTML etc.

In our example, the server does two things:

1. serves the static CSS and JS files.
2. it also serves as a proxy that passes requests to the WSGI server and sends the responses back to the web browser.

We will use Nginx (other options include Apache HTTP server etc)

we will store the static files in the static/ dir in static/ we will have various subfolder housing the JS and CSS etc.

for the second one, we will use our server as a reverse proxy - i.e. it will send the Python requests to the WSGI server, it will run the Python code and return the response to our web server which will send it as the response for the HTTP request.

before we install and configure Nginx, we will need to do two things:

1. set a domain name to point to our server
2. create an SSL certificate (for HTTPS)

the domain name is just a alias for an IP. The webserver is located on the internet by the IP address. But, it is difficult to remember the IP, so we use a domain name. the domain name resolution is the process of converting the name to IP.

there are special hosts that do this conversion, called Domain Name System hosts.

HTTP is without encryption, HTTPS is not. We can configure Nginx without HTTPS, via SSL/TLS

create a config file with the app name and extension as .conf /etc/nginx/conf.d/loccart.conf

here, we define the port to listen to for connections, define locations for access log, error log etc. say where to find the static files, we set the proxy settings.

we can create a self-signed certifiicate and upload it on the nginx server for security.

we can create the certificate in a few neat openssl commands.

you can securely copy items to your server like this:

scp -i path/on/local/machine/file.txt fil2.txt deployer@{your.server.ip.address}:/path/on/server/

we need to copy the cert files in a certain folder of the nginx dirs.

now, in the config file we created earlier, we will add the certificate location and its key.

we can automate all this, using Ansible.

we can have our code hosted on a github, and make our webserver grab it from there directly. for large projects, it is recommended that we have deployment pipelines package the source code to deploy it and never have production enviornment touch the source control system directly.

we need a deploy key. this will grant read only access from our server to the single repo that houses our code. this is important if your repo is private. for public repos, anyone can read the code.

whenever you want to authorize two systems, you use public and private keys.

we create a public-private key pair. we do this using ssh-keygen -t rsa -b 2048 -t is the algo -b is the #bytes to use in the key

it will prompt us to name the pair. this will create two files. one is the the private key foo and the public key foo.pub

we can create them like this: ssh-keygen -t rsa -b 2048

we put the public ket in github and keep the private key with us. so, the server can show everyone its key and then check if the client has the matching key, if yes, that client is authorized. else, not. so, you keep the secret one with you.

go to your private repo and add the deploy key to the project. share the public key. put the private key in your server.

back on server, install the git-core

then, do this to get the code on ther server ssh-agent bash -c ‘ssh-add /home/deployer/deploy_key/deploy_key; git clone git@github.com:makaimc/cyoa.git’

to update the code, we just pull the master branch ssh-agent bash -c ‘ssh-add /home/deployer/deploy_key/deploy_key; git pull origin master’

“A persistant database is an abstraction on top of an operating systems file system” that makes it easier for applicaions to create, read, write, update and delete data.

PostgreSQL is a traditional relational database. Redis is an in-memory key-value pair store.

we will use the former for long term/persistant data, and the latter for transient data for example, to keep a count of the number of votes received.

the database runs on the database server. we can install PostgreSQL by getting the following packages:

1. postgresql - core database software
2. libpq-dev - deveploment files so that you can use psycopg2 [the package that lets you interface postgresql from python]
3. postgresql-client-common and postgresql-client - programs for connecting to the postgresql database server.

apart from the relational storage scheme, we can also have: key-value pair, document-oriented, graph type.

Redis is an open source in-memory key-value pair data store. It can be used for caching, queuing, and storing session data for faster access than a traditional relational database, among many other use cases.

you set keys using: set keyname value you can get it by : get keyname <value>

strings are the basic data type in Redis. but we can perform increment and decrement also incr cyoa

again, all this can be automated.

application dependencies are the libraries that our project needs to work. for eg, if we need Flask, we can put it in the requirements.txt file.

we will install the dependencies on our server in a virtualenv, not directly. each virtualenv is its own copy of the python intrepreter and dependencies in the site-packages directory.

When you build a Python web application you should always include a requirements.txt file with your project’s pegged dependencies. “Pegging” dependencies means attaching a specific version number after the name of the dependency.

we can install what is in the requirements.txt file by first activating the enviornment and then issueing: pip install -r /home/deployer/cyoa/requirements.txt