Branch: master
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
546 lines (383 sloc) 20.4 KB
<li><a href="#ubuntu_vps">Spinning Ubuntu VPS</a></li>
<li><a href="#locales_fix">Fixing&nbsp;locales</a></li>
<li><a href="#common_programs">Installing common programs</a></li>
<li><a href="#setting_potgresql">Setting PostgreSQL</a></li>
<li><a href="#users_and_permissions">Setting up users and permissions</a></li>
<li><a href="#virtualenv">Virtualenv</a></li>
<li><a href="#start_django_project">Starting sample&nbsp;Django project</a></li>
<li><a href="#django_configure_postgresql">Configuring PostgreSQL to work with Django</a></li>
<li><a href="#setting_folders">Setting up folders</a></li>
<li><a href="#gunicorn">Gunicorn</a></li>
<li><a href="#supervisor">Supervisor</a></li>
<li><a href="#nginx">NginX</a></li>
<li><a href="#useful_commands">Useful commands</a></li>
<hr />
<p>Hello. Today we will be learning how to set up usual Django project.</p>
<p>We will be using such tehnology stack Ubuntu 16.04 x64, Django, PostgreSQL, Gunicorn, Supervisor, Virtualenv and NginX.</p>
<p>For hosting we will be using <a href="">DigitalOcean</a></p>
<hr />
<h3><a id="ubuntu_vps" name="ubuntu_vps"><strong>Spinning Ubuntu VPS</strong></a></h3>
<p>Visit <a href="">DigitalOcean</a> and register account , you can use <a href="" target="_blank">my invite link</a> (you&#39;ll get $10 in credit)</p>
<p>Than add your droplet for example you can use the cheapear one (5$ droplet)</p>
<p>After choosing region where your droplet will be located (choose where your customers will be located) set up name and you&#39;ve done - you&#39;ve created your first droplet.</p>
<p>Now check your email in a few minutes there will be a mailed with login info.</p>
<p>Then we will connect to Ubuntu&nbsp;</p>
<code class="language-bash">ssh root@ourServerIpAdress</code></pre>
<hr />
<h3><a id="locales_fix" name="locales_fix"><strong>Fixing&nbsp;locales</strong></a></h3>
<p>Let&#39;s start with ensuring that our system is up to date. PS `<code>-y</code>` means automatic yes to prompts</p>
<code class="language-bash">$ sudo apt-get update
$ sudo apt-get upgrade -y</code></pre>
<p>There is common problem with locales to fix it use this</p>
<code class="language-bash">$ nano /etc/environment
<p>and add this line add the end</p>
<p>Now reboot to finish fixing locales</p>
<code class="language-bash">$ sudo reboot</code></pre>
<hr />
<h3><a name="common_programs"><strong>Installing common programs like top, mc, tree and python-dev etc</strong></a></h3>
<p>Now wait at least 30 seconds and login again to our server and run below command to install all `bone structure` packages</p>
<code class="language-bash">$ sudo apt-get install htop mc tree python3-dev libpq-dev python-dev libxslt1-dev libxml2 -y</code></pre>
<hr />
<h3><a id="setting_potgresql" name="setting_potgresql"><strong>Setting PostgreSQL</strong></a></h3>
<p>Installing postgresql</p>
<code class="language-bash">$ sudo apt-get install postgresql postgresql-contrib -y</code></pre>
<p><span style="font-size:12px">Now we will create PostgreSQL user and set up database for him.</span></p>
<p><span style="font-size:12px">You need to change `<code>USERNAME</code>`, `<code>USERPASSWORD</code>` and `<code>DATABASE_NAME</code>` to your own values. <span class="marker">Note that `<code>USERNAME</code>` and `<code>DATABASE_NAME</code>` should be lovercase</span></span></p>
<code class="language-bash">$ sudo -u postgres psql --command="create user USERNAME"
$ sudo -u postgres psql --command="alter role USERNAME with password 'USERPASSWORD'"
$ sudo -u postgres psql --command="create database DATABASE_NAME with owner USERNAME template template0 encoding 'UTF8' "
$ sudo -u postgres psql --command="grant all privileges on database DATABASE_NAME to USERNAME"</code></pre>
<p>If you have problems with postgresql try to start cluster, because in some system cluster won&#39;t start automatically (remember to change <code>`9.5`</code> to your postgresql version)</p>
<code class="language-bash">$ pg_createcluster 9.5 main --start</code></pre>
<hr />
<h3><a id="users_and_permissions" name="users_and_permissions"><strong>Setting up users and permissions</strong></a></h3>
<p>Creating project directories</p>
<code class="language-bash">$ sudo mkdir -p /var/webapps/testproject/</code></pre>
<p>Here we will create a user for our&nbsp;website, named `<code>testuser</code>`&nbsp;and assigned to a system group called `<code>webapps`</code>.</p>
<code class="language-bash">$ sudo groupadd -r webapps
$ sudo useradd -mr -g webapps -s /bin/bash -d /var/webapps/testproject/home testuser</code></pre>
<p>Adding permissions to folders</p>
<code class="language-bash">$ sudo chown -R testuser:users /var/webapps/testproject/
$ sudo chmod -R g+w /var/webapps/testproject/
<hr />
<h3><a id="virtualenv" name="virtualenv"><strong>Virtualenv</strong></a></h3>
<p>Installing package</p>
<code class="language-bash">$ sudo apt-get install python-virtualenv -y</code></pre>
<p>Now login to our `<code>testuser</code>`, note that `<code>-</code>` will move you to user&#39;s home directory</p>
<code class="language-bash">$ su - testuser</code></pre>
<p>go one directory up</p>
<code class="language-bash">testuser@server:~$ cd ..</code></pre>
<p>and create virtualenv with python 3 (for python 2 use &#39;=python2&#39;)</p>
<code class="language-bash">testuser@server:/var/webapps/testproject$ virtualenv --python=python3 env
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/webapps/testproject/env/bin/python3
Also creating executable in /var/webapps/testproject/env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
<p>than activate it to use all packages from it</p>
<code class="language-bash">testuser@server:/var/webapps/testproject$ source env/bin/activate</code></pre>
<p>and finally create directory where we will be storing source code, in this directory you can either clone your code from git or follow next step</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ mkdir code</code></pre>
<hr />
<h3><a id="start_django_project" name="start_django_project"><strong>Starting sample&nbsp;Django project</strong></a></h3>
<p>Assuming that you have done all previous steps than we will create sample Django project</p>
<p>Installing Django is as easy as</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ pip install django
Collecting django
Installing collected packages: django
Successfully installed django</code></pre>
<p>Starting testproject, note that dot at the end means that we will re use current directory</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ cd code
(env) testuser@server:/var/webapps/testproject/code$ django-admin startproject testproject .</code></pre>
<p>And test it by&nbsp;running&nbsp;</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject/code$ python runserver
Performing system checks...
System check identified no issues (0 silenced).
You have 13 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.
Run 'python migrate' to apply them.
March 26, 2017 - 12:57:32
Django version 1.10.6, using settings 'testproject.settings'
Starting development server at
Quit the server with CONTROL-C.
<p>And you well be able to access it from&nbsp;<a href="">http://ourServerIpAdress:8000</a>&nbsp;</p>
<hr />
<h3><a id="django_configure_postgresql" name="django_configure_postgresql"><strong>Configuring PostgreSQL to work with Django</strong></a></h3>
<p>Installing PostgreSQL&nbsp;database adapter&nbsp;</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject/code$ pip install psycopg2</code></pre>
<p>You can now configure the databases in your `<code></code>`:</p>
<code class="language-python">DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'HOST': 'localhost',
'PORT': '', # Set to empty string for default.
<p>Than we need to create tables for our newly created database</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject/code$ python makemigrations
(env) testuser@server:/var/webapps/testproject/code$ python migrate</code></pre>
<hr />
<h3><a id="setting_folders" name="setting_folders"><strong>Setting up folders</strong></a></h3>
<p>Go one directory up</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ </code></pre>
<p>For now if you were following all steps your project directory should look like this</p>
<p>├── code<br />
&nbsp;&nbsp; ├──<br />
&nbsp;&nbsp; └── testproject<br />
├── env<br />
└── home</p>
<p>But for real life project you will need folders&nbsp;for logs, configs and etc</p>
<p>For example in `<code>media</code>` and `<code>static</code>`&nbsp;we will be storing media files and files from `<code>collectstatic</code>`&nbsp;</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ mkdir logs conf
(env) testuser@server:/var/webapps/testproject$ mkdir -p www/media
(env) testuser@server:/var/webapps/testproject$ mkdir -p www/static</code></pre>
<p>And for&nbsp;now it should look like</p>
<p>├── code<br />
&nbsp;&nbsp; ├──<br />
&nbsp;&nbsp; └── testproject<br />
├── conf<br />
├── env<br />
├── logs<br />
└── www<br />
&nbsp; &nbsp; ├── media<br />
&nbsp; &nbsp; └── static</p>
<hr />
<h3><a id="gunicorn" name="gunicorn"><strong>Gunicorn</strong></a></h3>
<p>In production we won&rsquo;t be using Django&rsquo;s single-threaded development server, but a special&nbsp;application server called&nbsp;<a href="">gunicorn</a>.</p>
<p>Installing it via</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ pip install gunicorn
Collecting gunicorn
Installing collected packages: gunicorn
Successfully installed gunicorn
<p>And test it by running</p>
<code class="language-bash">(env) testuser@server:/var/webapps/testproject$ cd code
(env) testuser@server:/var/webapps/testproject/code$ gunicorn testproject.wsgi:application --bind ourServerIpAdress:8001
[2017-03-26 13:01:34 +0000] [11414] [INFO] Starting gunicorn
[2017-03-26 13:01:34 +0000] [11414] [INFO] Listening at: http://ourServerIpAdress:8001 (11414)
[2017-03-26 13:01:34 +0000] [11414] [INFO] Using worker: sync
[2017-03-26 13:01:34 +0000] [11417] [INFO] Booting worker with pid: 11417
<p>Gunicorn is set up&nbsp;and ready to serve your website. Now we will create config file and save it in `<code>conf/</code>`.</p>
<p><span style="font-size:12px">Note: don&#39;t change <code>bind</code> , we will later use it for NginX</span></p>
<code class="language-python">proc_name = 'testproject'
bind = ''
workers = 3
user = 'testuser'
group = 'webapps'
loglevel = 'debug'
errorlog = '/var/webapps/testproject/logs/gunicorn.error.log'
accesslog = '/var/webapps/testproject/logs/gunicorn.access.log'</code></pre>
<p>As a rule-of-thumb set the <code>workers</code> according to the following formula: 2&nbsp;*&nbsp;CPUs&nbsp;+&nbsp;1. The idea being, that at any given time half of your workers will be busy doing I/O. For a single CPU machine it would give you 3.</p>
<p>Be careful not to forget to change paths and filenames to match your project&nbsp;in `<code>`</code></p>
<hr />
<h3><a id="supervisor" name="supervisor"><strong>Supervisor</strong></a></h3>
<p>We need to ensure&nbsp;sure that gunicorn is starting&nbsp;automatically with the system and that it can automatically restarts if for some reason it exits unexpectedly.</p>
<p>These is a job for a service called&nbsp;<a href="">supervisord</a>.&nbsp;</p>
<code class="language-bash">$ sudo apt-get install supervisor -y</code></pre>
<p>Specifically on Ubuntu 16.04 we need to execute this in order to work with supervisor correctly</p>
<code class="language-bash">$ sudo systemctl enable supervisor
$ sudo systemctl start supervisor</code></pre>
<p>In order to force Supervisor to run gunicorn&nbsp; we will need to create `<code>/etc/supervisor/conf.d/testproject.conf</code>`&nbsp;in `<code>/etc/supervisor/conf.d</code>`&nbsp;directory with this content</p>
<code class="language-bash">[program:testproject]
environment=LANG="en_US.utf8", LC_ALL="en_US.UTF-8", LC_LANG="en_US.UTF-8"
directory = /var/webapps/testproject/code
command = /var/webapps/testproject/env/bin/gunicorn testproject.wsgi:application -c /var/webapps/testproject/conf/
user = testuser
loglevel = debug
stdout_logfile = /var/webapps/testproject/logs/supervisor.log
stderr_logfile = /var/webapps/testproject/logs/supervisor.error.log
autostart = true
autorestart = true
redirect_stderr = true</code></pre>
<p>And create log files</p>
<code class="language-bash">$ touch /var/webapps/testproject/logs/supervisor.log
$ touch /var/webapps/testproject/logs/supervisor.error.log</code></pre>
<p>In order to recognise file and run it you should execute this</p>
<code class="language-bash">$ sudo supervisorctl reread
testproject: available
$ sudo supervisorctl update
testproject: added process group
<p>You can now check status of our site</p>
<code class="language-bash">$ sudo supervisorctl status testproject
testproject RUNNING pid 11987, uptime 0:00:35</code></pre>
<p>Your website should now be able to automatically start and restart in the case of failure or system reboot.</p>
<h3><a id="nginx" name="nginx"><strong>NginX</strong></a></h3>
<p>Here we will be setting up Nginx to work with our static files and media</p>
<p>To install</p>
<code class="language-bash">$ sudo apt-get install nginx -y
$ sudo service nginx start</code></pre>
<p>Check status of&nbsp;setup by visiting page&nbsp;<a href="">http://ourServerIpAdress</a>&nbsp;.&nbsp;Nginx should greet you with words &quot;Welcome to nginx!&quot;</p>
<p>Here we will create&nbsp;Nginx virtual server configuration for our Django website. Every&nbsp;Nginx virtual server should be configured&nbsp;by a file in the `<code>/etc/nginx/sites-available</code>` directory and enabling them by&nbsp;making symbolic links&nbsp;in the `<code>/etc/nginx/sites-enabled</code>` directory.</p>
<p>Create file `<code>/etc/nginx/sites-available/testproject</code>` with following content (remember to adapt it to your project variables and paths)</p>
<code class="language-nginx">server {
listen 80;
server_name ourServerIpAdress;
client_max_body_size 15m;
access_log /var/webapps/testproject/logs/nginx-access.log;
error_log /var/webapps/testproject/logs/nginx-error.log;
# Enable Gzip compression
gzip on;
# Compression level (1-9)
gzip_comp_level 5;
# Don't compress anything under 256 bytes
gzip_min_length 256;
# Compress output of these MIME-types
location /static/ {
alias /var/webapps/testproject/www/static/;
expires 30d;
add_header Pragma public;
add_header Cache-Control "public";
location /media/ {
alias /var/webapps/testproject/www/media/;
expires 30d;
add_header Pragma public;
add_header Cache-Control "public";
location / {
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header Host $server_name;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header REMOTE_ADDR $remote_addr;
# uncomment below to enable custom 404 (and other error pages)
# proxy_intercept_errors on;
# uncomment this and create file 404.html in /var/webapps/testproject/www/static
# directory to create custom 404 error page
# error_page 404 /404.html;
# location = /404.html {
# root /var/webapps/testproject/www/static;
# internal;
# }
<p>You can also check syntax for NginX via</p>
<code class="language-bash">$ nginx -t</code></pre>
<p>And create symbolik link to enable it in NginX</p>
<code class="language-bash">$ sudo ln -s /etc/nginx/sites-available/testproject /etc/nginx/sites-enabled/testproject</code></pre>
<p>And restart NginX</p>
<code class="language-bash">$ sudo service nginx restart
<p>As for now you should achieve up and running website&nbsp;on&nbsp;<a href="">http://ourServerIpAdress</a>&nbsp;</p>
<p>Final directory structure in&nbsp;`<code>/var/webapps/testproject</code>` will be</p>
<p>├── code<br />
&nbsp;&nbsp; ├──<br />
&nbsp;&nbsp; └── testproject<br />
├── conf<br />
&nbsp;&nbsp; └──<br />
├── env<br />
├── home<br />
├── logs<br />
&nbsp;&nbsp; ├── gunicorn.access.log<br />
&nbsp;&nbsp; ├── gunicorn.error.log<br />
&nbsp;&nbsp; ├── nginx-access.log<br />
&nbsp;&nbsp; ├── nginx-error.log<br />
&nbsp;&nbsp; ├── supervisor.error.log<br />
&nbsp;&nbsp; └── supervisor.log<br />
└── www<br />
&nbsp;&nbsp;&nbsp; ├── media<br />
&nbsp;&nbsp;&nbsp; └── static</p>
<h3><a id="useful_commands" name="useful_commands"><strong>Useful commands</strong></a></h3>
<p>To refresh changes in project use this, it will restart gunicorn and supervisor and as a result your changes to source files will be visible on site.</p>
<code class="language-bash">$ sudo service supervisor restart</code></pre>
<p><strong>How to backup database&nbsp; (<span style="color:#ff0000">unsure about this, help wanted</span>) </strong></p>
<p>This command wil create db.json file with your database backup</p>
<code class="language-bash">python dumpdata --exclude auth.permission --exclude contenttypes &gt; db.json
<p>When you want to load your backup you need to drop database and load in freshly created database</p>
<p>So in Django you need to migrate before loading backup and then</p>
<code class="language-bash">python loaddata db.json</code></pre>
<p><span style="font-size:12px">SIdenotes: this can&#39;t be used for large databases, backup and restore databases should be identical in structure.</span></p>
<p><span style="font-size:12px">If you know about <code>Ansible</code> and <code>Vagrant</code> you can check my <a href="" target="_blank">Django Seed Project</a> which is deployable via <code>ansible</code>.</span></p>