README.deploy

Configuration
=============


Introduction
------------

'deploy' is a simple script to deploy configurations to a collection
of hosts, with the paranoid sysadmin in mind.

'deploy' is feature oriented, i.e. you configure for a specific
feature and then deploy the configuration files to a set of hosts.

'deploy' will also do absolutely nothing without giving the sysadmin a
change to double check that the resulting configuration is sane.  This
is done by copying all the new configuration files to the hosts,
allowing the sysadmin to then unpack them, do a diff against current
configuration files before putting the new files in their intended
location, and do a manual post-deployment step as well.


Structure
---------

A basic feature (let's name it {feature} for the sake of example) is
a local subdirectory named after it, with the following minimum
content:

{feature}
  |
  +-- HOSTS
  +-- README
  +-- src/

The file HOSTS is a list of hosts where this feature is to be
present.  See 'HOSTS' below for the syntax.

The file README is instructions on how to handle this feature on the
hosts.  It will be present on the hosts as /READM.{feature}, and
should therefore be instructions for the admins that take care of the
host.  At a minimum, it's recommended that it contain information on
this configuration structure and how to access it, and then whatever
an admin needs to know to handle the files post-deployment (see
'Deployment' below).

The directory src/ contains copies of the files that are to appear on
the hosts, from the root (/).  So for example, if you want to have the
file /etc/blorp.conf, you'd have it as {feature}/src/etc/blorp.conf in
the configuration (specifically, if the feature is named "blorp", the
path would be blorp/src/etc/blorp.conf).

Additionally, if there are some files that are unique to just one
host, there may be a subdirectory named after that host present as
well.  If we wanted to have some unique files for a host that we name
{host1} here, the directory structure would look like this:

{feature}
  |
  +-- HOSTS
  +-- README
  +-- src/
  +-- {host1}/

Note that only hosts that are mentioned in HOSTS will be cared for.
Any other subdirectory (apart from src/) will be silently ignored by
deploy (but may be used freely by admin provided scripts and
templates, see below).


Feature names
-------------

Features can be named anything you want, EXCEPT:

* the name may not start with an underscore (_).  Those directories
  are reserved for deployment use.
* the name MUST NOT contain a slash.  It will fail if it does.

Other than that, keep in mind that the feature name will appear as
part of a file name (the tarball that's being transferred, see
'Deployment' below) and is transferred with scp, so if there's a
character in the feature name that might affect the operations of
deployment, simply do not use it, name your feature something better.


HOSTS
-----

In the explanation here, "host name" SHALL be understood as the name
of the A or AAAA (or possibly a CNAME ultimately leading to a A or
AAAA) record for the host in DNS.

In its simplest form, HOSTS is a list of hosts, one per line.

However, there are cases when a host has an intended final name, but
currently has a different name, or maybe no name at all.

There are also cases when several names are served on the same server,
and should therefore be treated as a group of names that are processed
all at the same time and on the same machine.

In all these cases, the actual server where the final host is served
can be seen as a staging server.  To specify a staging server, use
this syntax:

    {finalname}:{stagingserver}

In the case of several names being served on the same server, simply
use several lines with the same staging server name:

    {finalname1}:{stagingserver}
    {finalname2}:{stagingserver}
    {finalname3}:{stagingserver}

For the sake of simplicity, one can say that when no staging server is
specified, the final name is seen as both the final name and the staging
server.


Deployment
----------

Features are deployed with the script 'deploy', at the top of the
configuration file tree, i.e. the parent directory of the feature
directories, as follows:

    deploy [ -n ] {feature} [ {host1} ... ]

or possibly, if you have the scripts in place (copied or symlinked):

    ./deploy [ -n ] {feature} [ {host1} ... ]

NOTE: if hosts are given on the command line, they will only be
accepted if they are staging servers in {feature}/HOSTS.  If not,
'deploy' will fail.

It does the following for each host (called {host} below):

  - set up two staging directories, let's call them {data} and
    {control}.  {data} is used as temporary storage for the deployed
    files.  {control} is used to store the deployment scripts and the
    tarball with the deployed files.
  - if there is an executable script {feature}/pre-copy, execute it
    with {data} as current working directory.
  - copy {feature}/README to {data}/README.{feature}
  - copy the contents of {feature}/src/ recursively to {data}/
  - copy the contents of {feature}/{host}/ recursively to {data}/.
    Note that in this specific case, {host} is both the final host
    name and the staging server name.
  - copy the contents of {feature}/{host}@{staging}/ recursively to
    {data}/. Note that in this specific case, {host} is the final host
    name and {staging} is the staging server name.
  - look for any template file (files with names ending with '.tt2')
    in {data} and process each of them resulting in a files with the
    same name but without the '.tt2' ending, then remove the template
    files.  See 'Template files' below for more information.
  - if there is an executable script {feature}/post-copy, execute it
    with {data} as current working directory.
  - if there is a template {feature}/pre-unpack.tt2 then process it,
    resulting in {control}/pre-unpack.{feature}.
  - if there is a template {feature}/post-unpack.tt2 then process it,
    resulting in {control}/post-unpack.{feature}.
  - if there is a template {feature}/post-deploy.tt2 then process it,
    resulting in {control}/post-deploy.{feature}.
  - create a tarball from the contents of {data}/ (with {data}/ being
    the current working directory).
    The result is {control}/deploy.{feature}.tar.gz
  - create the main deployment scripts from _host_helpers/*.tt2,
    the result ending up in {control}
  - if there is a file {feature}/REMOVE, copy it to a list of files
    to be removed, {control}/deploy-remove.{feature}.txt.
  - unless '-n' was given, upload everything from {control}/ to
    /tmp/ on the remote hosts.

If '-n' is given, there is no uploading, and you will find the
tarballs locally.  'deploy' will tell you the full path to each
tarball.  This can be used to verify that 'deploy' does it's job
correctly.

If '-n' wasn't given, the tarball is /tmp/deploy.{feature}.tar.gz on
all hosts, and the helper scripts from _host_helpers/ end up as
/tmp/deploy-unpack.{feature}, /tmp/deploy-diff.{feature},
/tmp/deploy.{feature}, /tmp/deploy-post.{feature}, and
{feature}/pre-unpack.tt2 or {feature}/pre-unpack end up as
/tmp/pre-unpack.{feature}, {feature}/post-unpack.tt2 or
{feature}/post-unpack end up as /tmp/post-unpack.{feature}, and
{feature}/post-deploy.tt2 or {feature}/post-deploy as
/tmp/post-deploy.{feature}, and finally {feature}/REMOVE ends up as
/tmp/deploy-remove.{feature}.txt.  'deploy' ends the whole thing with
an explanation of which hosts were deployed to and what to do next.


Template files
--------------

The template files mentioned in 'Deployment' are processed with
Template Toolkit, a powerful perl based templating system.  If you're
not aquainted with that tookit, you may want to look here first:

    http://www.template-toolkit.org/

NOTE: you don't really need to know perl to make use of this
templating system.

As mentioned previously, template files have file names ending with
'.tt2', and in addition to the template files in {feature}/src/ and
{feature}/{host}/, there may be a few other useful ones as well:

- preamble.tt2 (in the top directory), is processed before the actual
  template file, and may contain stuff that is common for more than
  one feature.  It typically contains variable assignments.
- {feature}/preamble.tt2 is also processed before the actual template
  file, and may contain stuff that are needed to process the actual
  template file, for example assignment of variables with host
  specific values.
- {feature}/postamble.tt2 is processed after the actual template file.
- postamble.tt2 is also processed after the actual template file.

The exact process looks like this

    tpage --define hosts={finalhosts} --define staginghost={staging} \
          --define feature={feature} \
          --define data={data} --define control={control} \
          --pre_process preamble.tt2 --pre_process {feature}/preamble.tt2 \
          --post_process {feature}/postamble.tt2 --post_process postamble.tt2 \
          {data}/.../{file}.tt2 > {data}/.../{file}

Of course, each --pre_process and --post_process will only be there if
the corresponding file exists.


Execution on the host
---------------------

On each staging host, the admin is expected to run four commands:

-   /tmp/deploy-unpack.{feature}

    This unpacks all the data files into a staging directory.
    The staging directory is /var/tmp/deploy.{feature} by default,
    but can be overriden with the environment variable $DEPLOY_STAGING.

    If there is an executable file /tmp/pre-unpack.{feature}, it will
    be executed first, with the environment variable DEPLOY_STAGING
    assigned the staging directory.  If it fails, unpacking doesn't
    happen.

    If there is an executable file /tmp/post-unpack.{feature}, it will
    be executed as well, with the environment variable DEPLOY_STAGING
    assigned the staging directory.

-   /tmp/deploy-diff.{feature}

    This compares all the files in the staging directory with the
    corresponding live files, and display the differences.  It also
    takes files removals indicated by /tmp/deploy-remove.{feature}.txt
    into account.

    This allows the admin to see exactly what's going to happen and to
    see that everything is all right.  If something looks wrong at this
    stage, the admin should run /tmp/deploy-abort.{feature}, which will
    simply clean everything away and leave the live files untouched.

-   /tmp/deploy.{feature}

    This copies all the files from the staging directory to the live
    directory.  This also removes the files that are indicated by
    /tmp/deploy-remove.{feature}.txt from the live directory.

    All updated or removed files are backed up to a file with the same
    name with the added suffix '~'.

    When this is done, there's no return.

-   /tmp/deploy-post.{feature}

    If there is an executable file /tmp/post-deploy.{feature}, it will
    be executed first, with the environment variable DEPLOY_STAGING
    assigned the staging directory.

    This ends by cleaning away the deployment scripts and the staging
    directory.