The intention of this project is to provide automated configuration management for production, development, and staging environments with the same set of files.
You may follow our
latest repository branch for
the latest stable release.
This project no longer represents the majority of our applications.
Installation, VM setup:
If you've already been working with an earlier version of
Prerequisites and Dependencies
At least when running all of the web application and database services together, which is the case with the default Vagrantfile mentioned below, we assume a host machine with at least 8 GB of memory.
Please install the following tools as documented on their websites:
- VirtualBox (Version 5.2.x)
- Vagrant (Version 2.0.x)
- vagrant-vbguest (
vagrant plugin install vagrant-vbguest)
- Ansible (Version 1.9) (Version 2.x will definitely not work) installation instructions
- Additional dependencies described in the
piprequirements file (see below)
- Clone this project with Git or download the latest zipfile and open it. If you download the zipfile you won't be able to update it as easily when we issue updates.
- Install the additional dependencies using
pip install -r requirements.txt
- Copy the following files to their "live" equivalents (removing ".dist") and
edit them with values specific to your installation.
- Note that user shell accounts are configured in
ansible/group_vars/all, and that they require SSH public keys in their ssh_authorized_keys fields. The
adminusersvariable is for administrative users who will run ansible-playbook.
- Note that user shell accounts are configured in
- If you're going to be developing DPLA applications, you might want to
*_use_local_sourcevariables in some of the roles'
defaultsdirectories, as well as the variables related to the source directories.
- If you're going to be developing DPLA applications, you might want to override the
Vagrantfile. If you want to perform development work on any of the DPLA applications that will be installed, see the comments in
Vagrantfileabout mounting their working copy directories for "local" deployments.
- Make sure that Vagrant has downloaded the base server image that we'll need for our VMs:
$ vagrant box add ubuntu/trusty64
- Add the following entries to your /etc/hosts file or the equivalent for your operating system:
192.168.50.2 loadbal local.dp.la 192.168.50.6 webapp1 192.168.50.50 es
- Run these commands in your terminal:
$ cd /path/to/automation/ansible # Replace with your actual path $ vagrant up $ ansible-playbook -i development -u vagrant --private-key=$HOME/.vagrant.d/insecure_private_key playbooks/package_upgrade.yml dev_all.yml $ vagrant reload
The various applications will be online at:
- http://local.dp.la/ (Some redirects and static resources, but not the frontend site)
- http://local.dp.la/pssapi/ (Primary Source Sets API)
- https://local.dp.la/primary-source-sets/admins/sign_in (Primary Source Sets admin. This is not served on the production site.)
- http://local.dp.la:9201/ (Elasticsearch API)
- http://webapp1:8008/munin/ (Resource monitoring graphs, after they've had time to gather statistics and update)
You may set
ansible/group_vars/all to control
whether your development VM's sites go over SSL. The default is
means the development VMs will not use HTTPS.
The development VMs use a self-signed certificate, which will cause your browser
to issue a harsh security warning when you try to load
We recommend that you add the
local.dp.la SSL certificate to your operating
system's list of trusted certificates to get around this. If you use Chrome,
the blog for the Postman utility has an article at
that provides a useful walkththrough of the steps that you can take to achieve
this. It's intended for users of Postman, which is a Chrome plugin, but will
obviously work to get Chrome to accept the certificate, which is what you need.
If you use another browser, the procedure should be similar. You need to
download the certificate and add it to your certificate chain, or your operating
system's certificate chain, with trusted status.
After the hosts are spooled up (as VMs or cloud servers), subsequent commands
with ansible-playbook can be run with one of the administrative user accounts
defined in the file
ansible/group_vars/all, mentioned above. If you're using
a VM, the "vagrant" user is necessary for initially provisioning your server,
but once the server is provisioned, and user accounts have been created, you
may use one of those sysadmin accounts for consistency with usage in
For example, say I created an account named "alice" on each server (via
ansible/group_vars/all), I could run this command to execute all
of the user-management plays in the
dev_all.yml playbook, with the
$ cd /path/to/automation/ansible # contains "ansible.cfg" $ ansible-playbook -u alice -i development dev_all.yml -t users
Please note that
ansible.cfg needs to be in the working directory of the
ansible-playbook command in order for it to read the path to the
directory, for that to be available to all of the playbooks in
You can SSH directly into your new VMs as if they were servers on your own private network. For example:
$ ssh me@webapp1
Branching and Release Model
Please see our Branching And Release Model wiki page.
Application build directories
There are a number of
clean_* variables for redeploying various
applications. If you are deploying earlier versions of the applications and
you run into errors when they are being checked out into their build
directories, you may need to set the appropriate variable with the
ansible-playbook command, such as
-e 'clean_api=true'. This can also be
useful when a switch between branches would result in a merge conflict with a
file that has been added or removed in one version or the other.
Using local sources on your system, not from a repository
Please note that, if you're using
*_use_local_source: true for any
application, you'll be responsible for managing the state of some of your
configuration files in your local directories, which will get deployed to the
VM. Check the roles'
tasks files for details, or search the files for
"use_local_source" to see where this applies.
Roles, inventory, and variables have been laid out to allow, as much as possible, the use of the same roles and tasks in all stages of deployment -- development, staging, and production.
We have considered using Docker, with its lower memory requirements. It is important to us, however, that we be able to simulate our networking interfaces and their configuration, and it was not obvious that we could do that with Docker.
We will keep watching the progress of Docker support in Vagrant, and this situation may change. We would also like to hear from anyone who can suggest ways in which we might use Docker, keeping in mind our need to represent our network setup.
- The first time you create a VM, it has to do an extensive package upgrade of the the software in the base image, after which you should restart the VM with "vagrant reload".
- If you notice that your VirtualBox processes on your host computer are using a
lot of CPU after creating new VMs, this is probably becuase the Munin resource
monitoring tool's statistics gatherer (
munin-node) is running endlessly. We have not diagnosed what cuases this, but the easy solution is to make sure you have run
vagrant reloadas suggested above in Installation > Steps. Restarting the
munin-nodeservice on each host or rebooting the VMs seems to fix the problem.
- You probably want to exclude
$HOME/VirtualBox VMs/from any backup jobs that you have going on. The VMs can be recreated at any time, as long as you aren't storing data that can't be regenerated.
- If you destroy and re-create a VM, you should delete the old public key
$HOME/.ssh/known_hoststo avoid getting an error when you run ansible-playbook again. Look for the hostname in addition to its IP address.
- See the note in group_vars/all.dist about using SSH private keys for deploying from private GitHub repositories, if you intend to use private resources.