It’s time to write an Ansible playbook to deploy Mezzanine to a server. We’ll go through it step by step, but if you’re the type of person who starts off by reading the last page of a book to see how it ends,[1] you can find the full playbook at the end of this chapter as mezzanine.yml: the complete playbook. It’s also available on GitHub. Check out the README file before trying to run it directly.
I’ve tried to hew as closely as possible to the original Fabric scripts that Mezzanine author Stephen McDonald wrote.[2]
Before we dive into the guts of our playbook, let’s get a high-level view. The ansible-playbook command-line tool supports a flag called --list-tasks. This flag prints out the names of all the tasks in a playbook. It’s a handy way to summarize what a playbook is going to do. Here’s how you use it:
$ ansible-playbook --list-tasks mezzanine.ymlList of tasks in Mezzanine playbook shows the output for the mezzanine.yml playbook in mezzanine.yml: the complete playbook.
playbook: mezzanine.yml
play #1 (web): Deploy mezzanine TAGS: []
tasks:
install apt packages TAGS: []
create project path TAGS: []
create a logs directory TAGS: []
check out the repository on the host TAGS: []
install Python requirements globally via pip TAGS: []
create project locale TAGS: []
create a DB user TAGS: []
create the database TAGS: []
ensure config path exists TAGS: []
create tls certificates TAGS: []
remove the default nginx config file TAGS: []
set the nginx config file TAGS: []
enable the nginx config file TAGS: []
set the supervisor config file TAGS: []
install poll twitter cron job TAGS: []
set the gunicorn config file TAGS: []
generate the settings file TAGS: []
install requirements.txt TAGS: []
install required python packages TAGS: []
apply migrations to create the database, collect static content TAGS: []
set the site id TAGS: []
set the admin password TAGS: []
As we discussed earlier, Mezzanine is built atop Django. In Django, a web app is called a project. We get to choose what to name our project, and I’ve chosen to name it mezzanine_example.
Our playbook deploys into a Vagrant machine, and will deploy the files into the home directory of the Vagrant user’s account.
Directory structure under /home/vagrant shows the relevant directories underneath /home/vagrant:
-
/home/vagrant/mezzanine/mezzanine-example will contain the source code that will be cloned from a source code repository on GitHub.
-
/home/vagrant/.virtualenvs/mezzanine_example is the virtualenv directory, which means that we’re going to install all of the Python packages into that directory.
-
/home/vagrant/logs will contain log files generated by Mezzanine.
.
├── logs
├── mezzanine
│ └── mezzanine_example
└── .virtualenvs
└── mezzanine_example
As you can see in Defining the variables, this playbook defines quite a few variables.
vars:
user: "{{ ansible_user }}"
proj_app: mezzanine_example
proj_name: "{{ proj_app }}"
venv_home: "{{ ansible_env.HOME }}/.virtualenvs"
venv_path: "{{ venv_home }}/{{ proj_name }}"
proj_path: "{{ ansible_env.HOME }}/mezzanine/{{ proj_name }}"
settings_path: "{{ proj_path }}/{{ proj_name }}"
reqs_path: requirements.txt
manage: "{{ python }} {{ proj_path }}/manage.py"
live_hostname: 192.168.33.10.xip.io
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
repo_url: git@github.com:ansiblebook/mezzanine_example.git
locale: en_US.UTF-8
# Variables below don't appear in Mezzanine's fabfile.py
# but I've added them for convenience
conf_path: /etc/nginx/conf
tls_enabled: True
python: "{{ venv_path }}/bin/python"
database_name: "{{ proj_name }}"
database_user: "{{ proj_name }}"
database_host: localhost
database_port: 5432
gunicorn_procname: gunicorn_mezzanine
num_workers: "multiprocessing.cpu_count() * 2 + 1"
vars_files:
- secrets.ymlI’ve tried for the most part to use the same variable names that the Mezzanine Fabric script uses. I’ve also added some extra variables to make things a little clearer. For example, the Fabric scripts directly use proj_name as the database name and database username. I prefer to define intermediate variables named database_name and database_user and define these in terms of proj_name.
It’s worth noting a few things here. First off, we can define one variable in terms of another. For example, we define venv_path in terms of venv_home and proj_name.
Also, note how we can reference Ansible facts in these variables. For example, venv_home is defined in terms of the ansible_env fact collected from each host.
Finally, note that we have specified some of our variables in a separate file, called secrets.yml, by doing this:
vars_files:
- secrets.ymlThis file contains credentials such as passwords and tokens that need to remain private. The repository on GitHub does not actually contain this file. Instead, it contains a file called secrets.yml.example that looks like this:
db_pass: e79c9761d0b54698a83ff3f93769e309
admin_pass: 46041386be534591ad24902bf72071B
secret_key: b495a05c396843b6b47ac944a72c92ed
nevercache_key: b5d87bb4e17c483093296fa321056bdc
# You need to create a Twitter application at https://dev.twitter.com
# in order to get the credentials required for Mezzanine's
# twitter integration.
#
# See http://mezzanine.jupo.org/docs/twitter-integration.html
# for details on Twitter integration
twitter_access_token_key: 80b557a3a8d14cb7a2b91d60398fb8ce
twitter_access_token_secret: 1974cf8419114bdd9d4ea3db7a210d90
twitter_consumer_key: 1f1c627530b34bb58701ac81ac3fad51
twitter_consumer_secret: 36515c2b60ee4ffb9d33d972a7ec350aTo use this repo, you need to copy secrets.yml.example to secrets.yml and edit it so that it contains the credentials specific to your site. Also note that secrets.yml is included in the .gitignore file in the Git repository to prevent someone from accidentally committing these credentials.
It’s best to avoid committing unencrypted credentials into your version-control repository because of the security risks involved. This is just one possible strategy for maintaining secret credentials. We also could have passed them as environment variables. Another option, which we will describe in [more_on_playbooks_a], is to commit an encrypted version of the secrets.yml file by using Ansible’s vault functionality.
We’re going to need two types of packages for our Mezzanine deployment. We need to install some system-level packages, and because we’re going to deploy on Ubuntu, we use apt as our package manager for the system packages. We also need to install some Python packages, and we’ll use pip to install the Python packages.
System-level packages are generally easier to deal with than Python packages, because system-level packages are designed specifically to work with the operating system. However, the system package repositories often don’t have the newest versions of the Python libraries we need, so we turn to the Python packages to install those. It’s a trade-off between stability and running the latest and greatest.
Installing system packages shows the task we’ll use to install the system packages.
- name: install apt packages
apt: pkg={{ item }} update_cache=yes cache_valid_time=3600
become: True
with_items:
- git
- libjpeg-dev
- libpq-dev
- memcached
- nginx
- postgresql
- python-dev
- python-pip
- python-psycopg2
- python-setuptools
- python-virtualenv
- supervisorBecause we’re installing multiple packages, we use Ansible’s iteration functionality, the with_items clause. We could have installed the packages one at a time, like this:
- name: install git
apt: pkg=git
- name: install libjpeg-dev
apt: pkg=libjpeg-dev
...However, it’s easier to write the playbook if we group the packages in a list. When we invoke the apt module, we pass it {{ item }}. This is a placeholder variable that will be populated by each of the elements in the list of the with_items clause.
|
Note
|
By default, Ansible uses item as the name of the loop iteration variable. In [more_on_playbooks_a], we’ll show how to change this variable name. |
In addition, the apt module contains an optimization making it more efficient to install multiple packages by using the with_items clause. Ansible will pass the entire list of packages to the apt module, and the module will invoke the apt program only once, passing it the entire list of packages to be installed. Some modules, like apt, have been designed to handle lists intelligently like this. If a module doesn’t have native support for lists, Ansible will simply invoke the module multiple times, once for each element of the list.
You can tell that the apt module is intelligent enough to handle multiple packages at once, because the output looks like this:
TASK: [install apt packages] ************************************************** ok: [web] => (item=[u'git', u'libjpeg-dev', u'libpq-dev', u'memcached', u'nginx', u'postgresql', u'python-dev', u'python-pip', u'python-psycopg2', u'python-setuptools', u'python-virtualenv', u'supervisor'])
On the other hand, the pip module does not handle lists intelligently, so Ansible must invoke it once for each element of the list, and the output looks like this:
TASK [install required python packages] **************************************** ok: [web] => (item=gunicorn) ok: [web] => (item=setproctitle) ok: [web] => (item=psycopg2) ok: [web] => (item=django-compressor) ok: [web] => (item=python-memcached)
In the playbook examples of [playbooks_a_beginning], we wanted the whole playbook to run as root, so we added the become: True clause to the play. When we deploy Mezzanine, most of the tasks will be run as the user who is SSHing to the host, rather than root. Therefore, we don’t want to run as root for the entire play, only for select tasks.
We can accomplish this by adding become: True to the tasks that do need to run as root, such as Installing system packages.
|
Note
|
All of the example commands in this subsection are run on the (Ubuntu) remote host, not the control machine. |
Ubuntu maintains a cache with the names of all of the apt packages that are available in the Ubuntu package archive. Let’s say you try to install the package named libssl-dev. We can use the apt-cache program to query the local cache to see what version it knows about:
$ apt-cache policy libssl-devThe output is shown in apt-cache output.
libssl-dev:
Installed: (none)
Candidate: 1.0.1f-1ubuntu2.21
Version table:
1.0.1f-1ubuntu2.21 0
500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
1.0.1f-1ubuntu2 0
500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
As we can see, this package is not installed locally. According to the local cache, the latest version is 1.0.1f-1ubuntu2.21. We also see some information about the location of the package archive.
In some cases, when the Ubuntu project releases a new version of a package, it removes the old version from the package archive. If the local apt cache of an Ubuntu server hasn’t been updated, then it will attempt to install a package that doesn’t exist in the package archive.
To continue with our example, let’s say we attempt to install the libssl-dev package:
$ apt-get install libssl-devIf version 1.0.1f-1ubuntu2.21 is no longer available in the package archive, we’ll see the following error:
Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main libssl-dev amd64 1.0.1f-1ubuntu2.21 404 Not Found [IP: 91.189.88.153 80] Err http://security.ubuntu.com/ubuntu/ trusty-security/main libssl-dev amd64 1.0.1f-1ubuntu2.21 404 Not Found [IP: 91.189.88.149 80] Err http://security.ubuntu.com/ubuntu/ trusty-security/main libssl-doc all 1.0.1f-1ubuntu2.21 404 Not Found [IP: 91.189.88.149 80] E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl-dev_1.0.1f-1ubuntu2. 21_amd64.deb 404 Not Found [IP: 91.189.88.149 80] E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/o/openssl/libssl-doc_1.0.1f-1ubuntu2. 21_all.deb 404 Not Found [IP: 91.189.88.149 80] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
On the command line, the way to bring the local apt cache up-to-date is to run apt-get update. When using the apt Ansible module, the way to bring the local apt cache up-to-date is to pass the update_cache=yes argument when invoking the module, as shown in Installing system packages.
Because updating the cache takes additional time, and because we might be running a playbook multiple times in quick succession in order to debug it, we can avoid paying the cache update penalty by using the cache_valid_time argument to the module. This instructs to update the cache only if it’s older than a certain threshold. The example in Installing system packages uses cache_valid_time=3600, which updates the cache only if it’s older than 3,600 seconds (1 hour).
Although Mezzanine can be used without writing any custom code, one of its strengths is that it is written on top of the Django platform, and Django is a great web application platform if you know Python. If you just wanted a CMS, you’d likely just use something like WordPress. But if you’re writing a custom application that incorporates CMS functionality, Mezzanine is a good way to go.
As part of the deployment, you need to check out the Git repository that contains your Django applications. In Django terminology, this repository must contain a project. I’ve created a repository on GitHub that contains a Django project with the expected files. That’s the project that gets deployed in this playbook.
I created these files using the mezzanine-project program that ships with Mezzanine, like this:
$ mezzanine-project mezzanine_example
$ chmod +x mezzanine_example/manage.pyNote that I don’t have any custom Django applications in my repository, just the files that are required for the project. In a real Django deployment, this repository would contain subdirectories that contain additional Django applications.
Checking out the Git repository shows how we use the git module to check out a Git repository onto a remote host.
- name: check out the repository on the host
git: repo={{ repo_url }} dest={{ proj_path }} accept_hostkey=yesI’ve made the project repository public so that readers can access it, but in general, you’ll be checking out private Git repositories over SSH. For this reason, I’ve set the repo_url variable to use the scheme that will clone the repository over SSH:
repo_url: git@github.com:ansiblebook/mezzanine_example.gitIf you’re following along at home, to run this playbook you must have the following:
-
A GitHub account
-
A public SSH key associated with your GitHub account
-
An SSH agent running on your control machine, with agent forwarding enabled
-
Your SSH key added to your SSH agent
Once your SSH agent is running, add your key:
$ ssh-add
If successful, the following command will output the public key of the SSH you just added:
$ ssh-add -l
The output should look like something this:
2048 SHA256:o7H/I9rRZupXHJ7JnDi10RhSzeAKYiRVrlH9L/JFtfA /Users/lorin/.ssh/id_rsa
To enable agent forwarding, add the following to your ansible.cfg:
[ssh_connection] ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ForwardAgent=yes
You can verify that agent forwarding is working by using Ansible to list the known keys:
$ ansible web -a "ssh-add -l"
You should see the same output as when you run ssh-add -l on your local machine.
Another useful check is to verify that you can reach GitHub’s SSH server:
$ ansible web -a "ssh -T git@github.com"
If successful, the output should look like this:
web | FAILED | rc=1 >> Hi lorin! You've successfully authenticated, but GitHub does not provide shell access.
Even though the word FAILED appears in the output, if the message from GitHub appears, then it was successful.
In addition to specifying the repository URL with the repo parameter and the destination path of the repository as the dest parameter, we also pass an additional parameter, accept_hostkey, which is related to host-key checking. We discuss SSH agent forwarding and host-key checking in more detail in [Appendix].
As mentioned earlier in this chapter, we’re going to install some of the packages as Python packages because we can get more recent versions of those than if we installed the equivalent apt package.
We can install Python packages systemwide as the root user, but it’s better practice to install these packages in an isolated environment to avoid polluting the system-level Python packages. In Python, these types of isolated package environments are called virtualenvs. A user can create multiple virtualenvs, and can install Python packages into a virtualenv without needing root access.
Ansible’s pip module has support for installing packages into a virtualenv and for creating the virtualenv if it is not available.
Install Python requirements shows how to use pip to install several packages globally. Note that this requires become: True.
- name: install Python requirements globally via pip
pip: name={{ item }} state=latest
with_items:
- pip
- virtualenv
- virtualenvwrapper
become: TrueInstall Python packages shows the two tasks that we use to install Python packages into the virtualenv, both of which use the pip module, although in different ways.
- name: install requirements.txt
pip: requirements={{ proj_path }}/{{ reqs_path }} virtualenv={{ venv_path }}
- name: install required python packages
pip: name={{ item }} virtualenv={{ venv_path }}
with_items:
- gunicorn
- setproctitle
- psycopg2
- django-compressor
- python-memcachedA common pattern in Python projects is to specify the package dependencies in a file called requirements.txt. And, indeed, the repository in our Mezzanine example contains a requirements.txt file. It looks like requirements.txt.
Mezzanine==4.2.2
The requirements.txt file is missing several other Python packages that we need for the deployment, so we explicitly specify these as a separate task.
Note that the Mezzanine Python package in requirements.txt is pinned to a specific version (4.2.2), whereas the other packages aren’t pinned; we just grab the latest versions of those. If we did not want to pin Mezzanine, we simply could have added Mezzanine to the list of packages, like this:
- name: install python packages
pip: name={{ item }} virtualenv={{ venv_path }}
with_items:
- mezzanine
- gunicorn
- setproctitle
- south
- psycopg2
- django-compressor
- python-memcachedAlternately, if we wanted to pin all of the packages, we have several options. We could have specified all the packages in the requirements.txt file. This file contains information about the packages and the dependencies. An example file looks like Example requirements.txt.
beautifulsoup4==4.5.3 bleach==1.5.0 chardet==2.3.0 Django==1.10.4 django-appconf==1.0.2 django-compressor==2.1 django-contrib-comments==1.7.3 filebrowser-safe==0.4.6 future==0.16.0 grappelli-safe==0.4.5 gunicorn==19.6.0 html5lib==0.9999999 Mezzanine==4.2.2 oauthlib==2.0.1 olefile==0.43 Pillow==4.0.0 psycopg2==2.6.2 python-memcached==1.58 pytz==2016.10 rcssmin==1.0.6 requests==2.12.4 requests-oauthlib==0.7.0 rjsmin==1.0.12 setproctitle==1.1.10 six==1.10.0 tzlocal==1.3
If you have an existing virtualenv with the packages installed, you can use the pip freeze command to print out a list of installed packages. For example, if your virtualenv is in ~/mezzanine_example, you can activate your virtualenv and print out the packages in the virtualenv like this:
$ source ~/mezzanine_example/bin/activate
$ pip freeze > requirements.txtInstalling from requirements.txt shows how we could have installed all the packages by using a requirements.txt file.
- name: copy requirements.txt file
copy: src=files/requirements.txt dest=~/requirements.txt
- name: install packages
pip: requirements=~/requirements.txt virtualenv={{ venv_path }}Alternatively, we could have specified both the package names and their versions in the list, as shown in Specifying package names and version. We pass a list of dictionaries, and dereference the elements with item.name and item.version.
- name: python packages
pip: name={{ item.name }} version={{ item.version }} virtualenv={{ venv_path }}
with_items:
- {name: mezzanine, version: 4.2.2 }
- {name: gunicorn, version: 19.6.0 }
- {name: setproctitle, version: 1.1.10 }
- {name: psycopg2, version: 2.6.2 }
- {name: django-compressor, version: 2.1 }
- {name: python-memcached, version: 1.58 }Up until this point in the book, every time we have invoked a module, we have passed the argument as a string. Taking the pip example from Specifying package names and version, we passed the pip module a string as an argument:
- name: install package with pip
pip: name={{ item.name }} version={{ item.version }} virtualenv={{ venv_path }}If we don’t like long lines in our files, we could break up the argument string across multiple lines by using YAML’s line folding, which we originally wrote about in [line_folding]:
- name: install package with pip
pip: >
name={{ item.name }}
version={{ item.version }}
virtualenv={{ venv_path }}Ansible also provides another option for breaking up a module invocation across multiple lines. Instead of passing a string, we can pass a dictionary in which the keys are the variable names. This means we can invoke Specifying package names and version like this instead:
- name: install package with pip
pip:
name: "{{ item.name }}"
version: "{{ item.version }}"
virtualenv: "{{ venv_path }}"The dictionary-based approach to passing arguments is also useful when invoking modules that take complex arguments. A complex argument is an argument to a module that is a list or a dictionary. The ec2 module, which creates new servers on Amazon EC2, is a good example of a module that takes complex arguments. Calling a module with complex arguments shows how to call a module that takes a list as an argument for the group parameter, and a dictionary as an argument to the instance_tags parameter. We’ll cover this module in more detail in [cloud].
- name: create an ec2 instance
ec2:
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: productionYou can even mix it up by passing some arguments as a string and others as a dictionary, by using the args clause to specify some of the variables as a dictionary. We could rewrite our preceding example as follows:
- name: create an ec2 instance
ec2: image=ami-8caa1ce4 instance_type=m3.medium key_name=mykey
args:
group:
- web
- ssh
instance_tags:
type: web
env: productionIf you’re using the local_action clause (we’ll cover this in more detail in [more_on_playbooks_b]), the syntax for complex args changes slightly. You need to add module: <modulename> as shown here:
- name: create an ec2 instance
local_action:
module: ec2
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: productionYou can also mix simple arguments and complex arguments when using local_action:
- name: create an ec2 instance
local_action: ec2 image=ami-8caa1ce4 instance_type=m3.medium key_name=mykey
args:
image: ami-8caa1ce4
instance_type: m3.medium
key_name: mykey
group:
- web
- ssh
instance_tags:
type: web
env: production|
Warning
|
Ansible allows you to specify file permissions, which are used by several modules, including file, copy, and template. If you are specifying an octal value as a complex argument, it must either start the value with a 0 or quote it as a string. For example, note how the mode argument starts with a 0: If you do not start the mode argument with a 0 or quote it as a string, Ansible will interpret the value as a decimal number instead of an octal, and will not set the file permissions the way you expect. For details, see GitHub. |
If you want to break your arguments across multiple lines, and you aren’t passing complex arguments, which form you choose is a matter of taste. I generally prefer dictionaries to multiline strings, but in this book I use both forms.
When Django runs in development mode, it uses the SQLite backend. This backend will create the database file if the file does not exist.
When using a database management system such as Postgres, we need to first create the database inside Postgres and then create the user account that owns the database. Later, we will configure Mezzanine with the credentials of this user.
Ansible ships with the postgresql_user and postgresql_db modules for creating users and databases inside Postgres. Creating the database and database user shows how we invoke these modules in our playbook.
When creating the database, we specify locale information through the lc_ctype and lc_collate parameters. We use the locale_gen module to ensure that the locale we are using is installed in the operating system.
- name: create project locale
locale_gen: name={{ locale }}
become: True
- name: create a DB user
postgresql_user:
name: "{{ database_user }}"
password: "{{ db_pass }}"
become: True
become_user: postgres
- name: create the database
postgresql_db:
name: "{{ database_name }}"
owner: "{{ database_user }}"
encoding: UTF8
lc_ctype: "{{ locale }}"
lc_collate: "{{ locale }}"
template: template0
become: True
become_user: postgresNote the use of become: True and become_user: postgres on the last two tasks.
When you install Postgres on Ubuntu, the installation process creates a user named
postgres that has administrative privileges for the Postgres installation.
Note that the root account does not have administrative privileges in Postgres
by default, so in the playbook, we need to become the Postgres user in order to
perform administrative tasks, such as creating users and databases.
When we create the database, we set the encoding (UTF8) and locale categories (LC_CTYPE, LC_COLLATE) associated with the database. Because we are setting locale information, we use template0 as the template.[3]
Django expects to find project-specific settings in a file called settings.py. Mezzanine follows the common Django idiom of breaking these settings into two groups:
-
Settings that are the same for all deployments (settings.py)
-
Settings that vary by deployment (local_settings.py)
We define the settings that are the same for all deployments in the settings.py file in our project repository. You can find that file on GitHub.
The settings.py file contains a Python snippet that loads a local_settings.py file that contains deployment-specific settings. The .gitignore file is configured to ignore the local_settings.py file, since developers will commonly create this file and configure it for local development.
As part of our deployment, we need to create a local_settings.py file and upload it to the remote host. local_settings.py.j2 shows the Jinja2 template that we use.
from __future__ import unicode_literals
SECRET_KEY = "{{ secret_key }}"
NEVERCACHE_KEY = "{{ nevercache_key }}"
ALLOWED_HOSTS = [{% for domain in domains %}"{{ domain }}",{% endfor %}]
DATABASES = {
"default": {
# Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
"ENGINE": "django.db.backends.postgresql_psycopg2",
# DB name or path to database file if using sqlite3.
"NAME": "{{ proj_name }}",
# Not used with sqlite3.
"USER": "{{ proj_name }}",
# Not used with sqlite3.
"PASSWORD": "{{ db_pass }}",
# Set to empty string for localhost. Not used with sqlite3.
"HOST": "127.0.0.1",
# Set to empty string for default. Not used with sqlite3.
"PORT": "",
}
}
SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTOCOL", "https")
CACHE_MIDDLEWARE_SECONDS = 60
CACHE_MIDDLEWARE_KEY_PREFIX = "{{ proj_name }}"
CACHES = {
"default": {
"BACKEND": "django.core.cache.backends.memcached.MemcachedCache",
"LOCATION": "127.0.0.1:11211",
}
}
SESSION_ENGINE = "django.contrib.sessions.backends.cache"Most of this template is straightforward; it uses the {{ variable }} syntax to insert the values of variables such as secret_key, nevercache_key, proj_name, and db_pass. The only nontrivial bit of logic is the line shown in Using a for loop in a Jinja2 template.
ALLOWED_HOSTS = [{% for domain in domains %}"{{ domain }}",{% endfor %}]
If we look back at our variable definition, we have a variable called domains that’s defined like this:
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.ioOur Mezzanine app is going to respond only to requests that are for one of the hostnames listed in the domains variable: http://192.168.33.10.xip.io or http://www.192.168.33.10.xip.io in our case. If a request reaches Mezzanine but the host header is something other than those two domains, the site will return "Bad Request (400)."
We want this line in the generated file to look like this:
ALLOWED_HOSTS = ["192.168.33.10.xip.io", "www.192.168.33.10.xip.io"]
We can achieve this by using a for loop, as shown in Using a for loop in a Jinja2 template. Note that it doesn’t do exactly what we want. Instead, it will have a trailing comma, like this:
ALLOWED_HOSTS = ["192.168.33.10.xip.io", "www.192.168.33.10.xip.io",]
However, Python is perfectly happy with trailing commas in lists, so we can leave it like this.
You might have noticed that the domains we are using look a little strange: 192.168.33.10.xip.io and www.192.168.33.10.xip.io. They are domain names, but they have the IP address embedded within them.
When you access a website, you pretty much always point your browser to a domain name such as http://www.ansiblebook.com, instead of an IP address such as http://151.101.192.133. When we write our playbook to deploy Mezzanine to Vagrant, we want to configure the application with the domain name or names that it should be accessible by.
The problem is that we don’t have a DNS record that maps to the IP address of our Vagrant box. In this case, that’s 192.168.33.10. There’s nothing stopping us from setting up a DNS entry for this. For example, I could create a DNS entry from mezzanine-internal.ansiblebook.com that points to 192.168.33.10.
However, if we want to create a DNS name that resolves to a particular IP address, there’s a convenient service called xip.io, provided free of charge by Basecamp, that we can use so that we don’t have to avoid creating our own DNS records. If AAA.BBB.CCC.DDD is an IP address, the DNS entry AAA.BBB.CCC.DDD.xip.io will resolve to AAA.BBB.CCC.DDD. For example, 192.168.33.10.xip.io resolves to 192.168.33.10. In addition, www.192.168.33.10.xip.io also resolves to 192.168.33.10.
I find xip.io to be a great tool when I’m deploying web applications to private IP addresses for testing purposes. Alternatively, you can simply add entries to the /etc/hosts file on your local machine, which also works when you’re offline.
Let’s examine the Jinja2 for loop syntax. To make things a little easier to read, we’ll break it up across multiple lines, like this:
ALLOWED_HOSTS = [
{% for domain in domains %}
"{{ domain }}",
{% endfor %}
]
The generated config file looks like this, which is still valid Python.
ALLOWED_HOSTS = [
"192.168.33.10.xip.io",
"www.192.168.33.10.xip.io",
]Note that the for loop has to be terminated by an {% endfor %} statement. Also note that the for statement and the endfor statement are surrounded by {% %} delimiters, which are different from the {{ }} delimiters that we use for variable substitution.
All variables and facts that have been defined in a playbook are available inside Jinja2 templates, so we never need to explicitly pass variables to templates.
Django applications use a special script called manage.py that performs administrative actions for Django applications such as the following:
-
Creating database tables
-
Applying database migrations
-
Loading fixtures from files into the database
-
Dumping fixtures from the database to files
-
Copying static assets to the appropriate directory
In addition to the built-in commands that manage.py supports, Django applications can add custom commands. Mezzanine adds a custom command called createdb that is used to initialize the database and copy the static assets to the appropriate place. The official Fabric scripts do the equivalent of this:
$ manage.py createdb --noinput --nodataAnsible ships with a django_manage module that invokes manage.py commands. We could invoke it like this:
- name: initialize the database
django_manage:
command: createdb --noinput --nodata
app_path: "{{ proj_path }}"
virtualenv: "{{ venv_path }}"Unfortunately, the custom createdb command that Mezzanine adds isn’t idempotent. If invoked a second time, it will fail like this:
TASK: [initialize the database] ***********************************************
failed: [web] => {"cmd": "python manage.py createdb --noinput --nodata", "failed"
: true, "path": "/home/vagrant/mezzanine_example/bin:/usr/local/sbin:/usr/local/b
in:/usr/sbin: /usr/bin:/sbin:/bin:/usr/games:/usr/local/games", "state": "absent"
, "syspath": ["", "/usr/lib/python2.7", "/usr/lib/python2.7/plat-x86_64-linux-gnu
", "/usr/lib/python2.7/lib-tk", "/usr/lib/python2.7/lib-old", "/usr/lib/python2.7
/lib-dynload", "/usr/local/lib/python2.7/dist-packages", "/usr/lib/python2.7/dist
-packages"]}
msg:
:stderr: CommandError: Database already created, you probably want the syncdb or
migrate command
Fortunately, the custom createdb command is effectively equivalent to two idempotent built-in manage.py commands:
- migrate
-
Create and update database tables for Django models
- collectstatic
-
Copy the static assets to the appropriate directories
By invoking these commands, we get an idempotent task:
- name: apply migrations to create the database, collect static content
django_manage:
command: "{{ item }}"
app_path: "{{ proj_path }}"
virtualenv: "{{ venv_path }}"
with_items:
- syncdb
- collectstaticTo initialize our application, we need to make two changes to our database:
-
We need to create a Site model object that contains the domain name of our site (in our case, that’s 192.168.33.10.xip.io).
-
We need to set the administrator username and password.
Although we could make these changes with raw SQL commands or Django data migrations, the Mezzanine Fabric scripts use Python scripts, so that’s how we’ll do it.
There are two tricky parts here. The Python scripts need to run in the context of the virtualenv that we’ve created, and the Python environment needs to be set up properly so that the script will import the settings.py file that’s in ~/mezzanine/mezzanine_example/mezzanine_example.
In most cases, if we needed some custom Python code, I’d write a custom Ansible module. However, as far as I know, Ansible doesn’t let you execute a module in the context of a virtualenv, so that’s out.
I used the script module instead. This will copy over a custom script and execute it. I wrote two scripts, one to set the Site record, and the other to set the admin username and password.
You can pass command-line arguments to script modules and parse them out, but I decided to pass the arguments as environment variables instead. I didn’t want to pass passwords via command-line argument (those show up in the process list when you run the ps command), and it’s easier to parse out environment variables in the scripts than it is to parse command-line arguments.
|
Note
|
You can set environment variables with an environment clause on a task, passing it a dictionary that contains the environment variable names and values. You can add an environment clause to any task; it doesn’t have to be a script. |
In order to run these scripts in the context of the virtualenv, I also needed to set the path variable so that the first Python executable in the path would be the one inside the virtualenv. Using the script module to invoke custom Python code shows how I invoked the two scripts.
- name: set the site id
script: scripts/setsite.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
PROJECT_APP: "{{ proj_app }}"
WEBSITE_DOMAIN: "{{ live_hostname }}"
- name: set the admin password
script: scripts/setadmin.py
environment:
PATH: "{{ venv_path }}/bin"
PROJECT_DIR: "{{ proj_path }}"
PROJECT_APP: "{{ proj_app }}"
ADMIN_PASSWORD: "{{ admin_pass }}"#!/usr/bin/env python
# A script to set the site domain
# Assumes two environment variables
#
# WEBSITE_DOMAIN: the domain of the site (e.g., www.example.com)
# PROJECT_DIR: root directory of the project
# PROJECT_APP: name of the project app
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
proj_app = os.environ['PROJECT_APP']
os.environ['DJANGO_SETTINGS_MODULE'] = proj_app + '.settings'
import django
django.setup()
from django.conf import settings
from django.contrib.sites.models import Site
domain = os.environ['WEBSITE_DOMAIN']
Site.objects.filter(id=settings.SITE_ID).update(domain=domain)
Site.objects.get_or_create(domain=domain)#!/usr/bin/env python
# A script to set the admin credentials
# Assumes two environment variables
#
# PROJECT_DIR: the project directory (e.g., ~/projname)
# PROJECT_APP: name of the project app
# ADMIN_PASSWORD: admin user's password
import os
import sys
# Add the project directory to system path
proj_dir = os.path.expanduser(os.environ['PROJECT_DIR'])
sys.path.append(proj_dir)
proj_app = os.environ['PROJECT_APP']
os.environ['DJANGO_SETTINGS_MODULE'] = proj_app + '.settings'
import django
django.setup()
from django.contrib.auth import get_user_model
User = get_user_model()
u, _ = User.objects.get_or_create(username='admin')
u.is_staff = u.is_superuser = True
u.set_password(os.environ['ADMIN_PASSWORD'])
u.save()Next, we set the configuration file for Gunicorn (our application server), Nginx (our web server), and Supervisor (our process manager), as shown in Setting configuration files. The template for the Gunicorn configuration file is shown in templates/gunicorn.conf.py.j2, and the template for the Supervisor configuration file is shown in templates/supervisor.conf.j2.
- name: set the gunicorn config file
template:
src: templates/gunicorn.conf.py.j2
dest: "{{ proj_path }}/gunicorn.conf.py"
- name: set the supervisor config file
template:
src: templates/supervisor.conf.j2
dest: /etc/supervisor/conf.d/mezzanine.conf
become: True
notify: restart supervisor
- name: set the nginx config file
template:
src: templates/nginx.conf.j2
dest: /etc/nginx/sites-available/mezzanine.conf
notify: restart nginx
become: TrueIn all three cases, we generate the config files by using templates. The Supervisor and Nginx processes are started by root (although they drop down to nonroot users when running), so we need to sudo so that we have the appropriate permissions to write their configuration files.
If the Supervisor config file changes, Ansible will fire the restart supervisor handler. If the Nginx config file changes, Ansible will fire the restart nginx handler, as shown in Handlers.
handlers:
- name: restart supervisor
supervisorctl: name=gunicorn_mezzanine state=restarted
sudo: True
- name: restart nginx
service: name=nginx state=restarted
sudo: Truefrom __future__ import unicode_literals
import multiprocessing
bind = "127.0.0.1:{{ gunicorn_port }}"
workers = multiprocessing.cpu_count() * 2 + 1
loglevel = "error"
proc_name = "{{ proj_name }}"[program:{{ gunicorn_procname }}]
command={{ venv_path }}/bin/gunicorn -c gunicorn.conf.py -p gunicorn.pid \
{{ proj_app }}.wsgi:application
directory={{ proj_path }}
user={{ user }}
autostart=true
stdout_logfile = /home/{{ user }}/logs/{{ proj_name }}_supervisor
autorestart=true
redirect_stderr=true
environment=LANG="{{ locale }}",LC_ALL="{{ locale }}",LC_LANG="{{ locale }}"
The only template that has any template logic (other than variable substitution) is templates/nginx.conf.j2. It has conditional logic to enable TLS if the tls_enabled variable is set to true. You’ll see some if statements scattered about the templates that look like this:
{% if tls_enabled %}
...
{% endif %}
It also uses the join Jinja2 filter here:
server_name {{ domains|join(", ") }};
This code snippet expects the variable domains to be a list. It will generate a string with the elements of domains connected together, separated by commas. Recall that in our case, the domains list is defined as follows:
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.ioWhen the template renders, the line looks like this:
server_name 192.168.33.10.xip.io, www.192.168.33.10.xip.io;
upstream {{ proj_name }} {
server unix:{{ proj_path }}/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
{% if tls_enabled %}
listen 443 ssl;
{% endif %}
server_name {{ domains|join(", ") }};
client_max_body_size 10M;
keepalive_timeout 15;
{% if tls_enabled %}
ssl_certificate conf/{{ proj_name }}.crt;
ssl_certificate_key conf/{{ proj_name }}.key;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
# ssl_ciphers entry is too long to show in this book
# See https://github.com/ansiblebook/ansiblebook
# ch06/playbooks/templates/nginx.conf.j2
ssl_prefer_server_ciphers on;
{% endif %}
location / {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Protocol $scheme;
proxy_pass http://{{ proj_name }};
}
location /static/ {
root {{ proj_path }};
access_log off;
log_not_found off;
}
location /robots.txt {
root {{ proj_path }}/static;
access_log off;
log_not_found off;
}
location /favicon.ico {
root {{ proj_path }}/static/img;
access_log off;
log_not_found off;
}
}
The convention with Nginx configuration files is to put your configuration files in /etc/nginx/sites-available and enable them by symlinking them into /etc/nginx/sites-enabled.
The Mezzanine Fabric scripts just copy the configuration file directly into sites-enabled, but I’m going to deviate from how Mezzanine does it because it gives me an excuse to use the file module to create a symlink. We also need to remove the default configuration file that the Nginx package sets up in /etc/nginx/sites-enabled/default.
As shown in Enabling Nginx configuration, we use the file module to create the symlink and to remove the default config file. This module is useful for creating directories, symlinks, and empty files; deleting files, directories, and symlinks; and setting properties such as permissions and ownership.
- name: enable the nginx config file
file:
src: /etc/nginx/sites-available/mezzanine.conf
dest: /etc/nginx/sites-enabled/mezzanine.conf
state: link
become: True
- name: remove the default nginx config file
file: path=/etc/nginx/sites-enabled/default state=absent
notify: restart nginx
become: TrueOur playbook defines a variable named tls_enabled. If this variable is set to true, the playbook will install TLS certificates. In our example, we use self-signed certificates, so the playbook will create the certificate if it doesn’t exist.
In a production deployment, you would copy an existing TLS certificate that you obtained from a certificate authority.
Installing TLS certificates shows the two tasks involved in configuring for TLS certificates. We use the file module to ensure that the directory that will house the TLS certificates exists.
- name: ensure config path exists
file: path={{ conf_path }} state=directory
sudo: True
when: tls_enabled
- name: create tls certificates
command: >
openssl req -new -x509 -nodes -out {{ proj_name }}.crt
-keyout {{ proj_name }}.key -subj '/CN={{ domains[0] }}' -days 3650
chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt
sudo: True
when: tls_enabled
notify: restart nginxNote that both tasks contain this clause:
when: tls_enabledIf tls_enabled evaluates to false, Ansible will skip the task.
Ansible doesn’t ship with modules for creating TLS certificates, so we need to use the command module to invoke the openssl command in order to create the self-signed certificate. Since the command is very long, we use YAML line-folding syntax (see [line_folding]) so that we can break the command across multiple lines.
These two lines at the end of the command are additional parameters that are passed to the module; they are not passed to the command line:
chdir={{ conf_path }}
creates={{ conf_path }}/{{ proj_name }}.crt
The chdir parameter changes the directory before running the command. The creates parameter implements idempotence: Ansible will first check whether the file {{ conf_path }}/{{ proj_name }}.crt exists on the host. If it already exists, Ansible will skip this task.
If you run manage.py poll_twitter, Mezzanine will retrieve tweets associated with the configured accounts and show them on the home page. The Fabric scripts that ship with Mezzanine keep these tweets up-to-date by installing a cron job that runs every five minutes.
If we followed the Fabric scripts exactly, we’d copy a cron script into the /etc/cron.d directory that had the cron job. We could use the template module to do this. However, Ansible ships with a cron module that allows us to create or delete cron jobs, which I find more elegant. Installing cron job for polling Twitter shows the task that installs the cron job.
- name: install poll twitter cron job
cron: name="poll twitter" minute="*/5" user={{ user }} job="{{ manage }} \
poll_twitter"If you manually SSH to the box, you can see the cron job that gets installed by using crontab -l to list the jobs. Here’s what it looks like for me when I deploy as the Vagrant user:
#Ansible: poll twitter */5 * * * * /home/vagrant/.virtualenvs/mezzanine_example/bin/python \ /home/vagrant/mezzanine/mezzanine_example/manage.py poll_twitter
Notice the comment at the first line. That’s how the Ansible module supports deleting cron jobs by name. If you were to do this:
- name: remove cron job
cron: name="poll twitter" state=absentthe cron module would look for the comment line that matches the name and delete the job associated with that comment.
mezzanine.yml: the complete playbook shows the complete playbook in all its glory.
link:mezzanine.yml[role=include]The live_hostname and domains variables in our playbook assume that the host we are going to deploy to is accessible at 192.168.33.10. The Vagrantfile shown in Vagrantfile configures a Vagrant machine with that IP address.
VAGRANTFILE_API_VERSION = "2"
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
config.vm.box = "ubuntu/trusty64"
config.vm.network "private_network", ip: "192.168.33.10"
endDeploy Mezzanine into the Vagrant machine:
$ ansible-playbook mezzanine.ymlYou can then reach your newly deployed Mezzanine site at any of the following URLs:
You might hit a few speed bumps when trying to run this playbook on your local machine. This section describes how to overcome some common obstacles.
You may see the task named "check out the repository on the host" fail with this error:
fatal: Could not read from remote repository.
A likely fix is to remove a preexisting entry for 192.168.33.10 in your ~/.ssh/known_hosts file. See [badhostkey_sb] for more details.
Some WiFi routers ship with DNS servers that won’t resolve the hostname 192.168.33.10.xip.io. You can check whether yours does by typing on the command line:
dig +short 192.168.33.10.xip.io
The output should be as follows:
192.168.33.10
If the output is blank, your DNS server is refusing to resolve xip.io hostnames. If this is the case, a workaround is to add the following to your /etc/hosts file:
192.168.33.10 192.168.33.10.xip.io
If your browser returns the error "Bad Request (400)," it is likely that you are trying to reach the Mezzanine site by using a hostname or IP address that is not in the ALLOWED_HOSTS list in the Mezzanine configuration file. This list is populated using the domains Ansible variable in the playbook:
domains:
- 192.168.33.10.xip.io
- www.192.168.33.10.xip.io
In this scenario, we’ve deployed Mezzanine entirely on a single machine. However, it’s common to deploy the database service on a separate host from the web service. In [roles], we’ll show a playbook that deploys across the database and web services on separate hosts.
You’ve now seen what it’s like to deploy a real application with Mezzanine. The next chapter covers some more advanced features of Ansible that didn’t come up in our example.