Docker container to support backup and restore of Consul key/value data on AWS (ECS)
Python Shell
Latest commit f7054e6 Oct 30, 2015 @ehenneken ehenneken Merge pull request #23 from ehenneken/backup_file
Make backupfile name and version configurable
Permalink
Failed to load latest commit information.
tests Update of tests for Travis Oct 28, 2015
.gitignore First commit Oct 2, 2015
.travis.yml Trying different approach for Travis Oct 26, 2015
AUTHORS.md First commit Oct 2, 2015
CHANGES.md
CONTRIBUTING.md
Dockerfile Make backupfile name and version configurable Oct 30, 2015
LICENSE.md First commit Oct 2, 2015
README.md
backup.py Make backupfile name and version configurable Oct 30, 2015
requirements.txt we don't need to require json either ;-) Oct 26, 2015
utils.py

README.md

Build Status

Usage

Assuming that the Dockerfile has been used to create an image consul_backup on the adsabs Docker Hub account, a backup would be created as follows

docker run adsabs/consul_backup

In the case of restoring an existing backup, we need info to identify the backup file. In this case we would set environment variables like

ACTION="restore"
RESTORE_ID="restore_id"
OVERWRITE=True (Optional, if you want to overwrite existing values)

where the restore ID is determined by the naming scheme used for creating backup files. Assuming these variables are set in a file env, you would execute

docker run --env-file ./env adsabs/consul_backup

So, restoring an existing backup would involve going into the S3 bucket and visually inspect which backup you want restored.

In terms of executing the backup in an ECS container, we define both actions in terms of Task Definitions. The Task Definition for the backup of the Consul key/value store would then be

    {
      "containerDefinitions": [
        {
          "name": "consul-backup",
          "image": "adsabs/consul-backup",
          "cpu": 384,
          "memory": 384,
          "essential": true,
          "environment": []
        }
      ],
      "family": "consul-backup"
    }

and for restoring a backup with identifier restore_ID the Task Definition would be

{
  "containerDefinitions": [
    {
      "name": "consul-backup",
      "image": "adsabs/consul-backup",
      "cpu": 384,
      "memory": 384,
      "essential": true,
      "environment": [
        { "name": "ACTION", "value": "restore" },
        { "name": "RESTORE_ID", "value": "restore_ID" },
      ]
    }
  ],
  "family": "consul-backup"
}

and using Mission Control, assuming the Task Definition is stored in the variable task_def, we would do

python mc/manage.py register_task_def --task "$task_def"

followed by

python mc/manage.py update_service --cluster staging --service consul-backup --desiredCount 1 --taskDefinition consul-backup

to execute the task. By default logs and temporary files are created in /tmp; this can be overwritten by setting the environment variable TMD_DIR appropriately.

Also, by default, key/value pairs are restored if they are not present in the current store. In order to restore the entire backup the environment variable OVERWRITE has to be set to True.