Skip to content

house9/wal-e-cookbook

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

wal-e-cookbook

chef cookbook for installing wal-e (postgres continuous archiving to s3)

See sample usage: here

Requirements

This cookbook has been developed and tested on ubuntu servers only

packages

  • postgresql - optionally use databox-cookbook which includes postgresql
  • python - wal-e is installed using python pip (package management system)
  • git - required to install wal-e using pip over git

Attributes

wal_e::master

  • ['wal_e']['aws_access_key_id'] : AWS Access Key
  • ['wal_e']['aws_secret_access_key'] : AWS Secret
  • ['wal_e']['wale_s3_prefix'] : S3 url for the bucket where postgres WAL and backups will be stored
    • i.e. s3://your-lower-case-bucket-name/whatever/wal-e/
  • ['wal_e']['wale_git_install_reference'] : Specify which commit to install wal-e from
    • Default is HEAD, which will install from the HEAD of master
    • Optionally set this to a git commit SHA or a git TAG to install a specific version
    • NOTE: the recipe uses a marker file with the value from this configuration to determine if it should install a different version of WAL-e, so re-running it when set to HEAD will only run the first time, it will not pull newer commits unless you update this to a value besides HEAD, i.e. a SHA or TAG

wal_e::recover

Usage

wal_e::master

Include wal_e in your node's run_list:

{
  "name":"my_node",
  "run_list": [
    "recipe[wal_e::master]"
  ]
}

and specify all 3 of the S3 attributes needed by wal-e

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2"
}

wal_e::recover

Include wal_e in your node's run_list:

{
  "name":"my_node",
  "run_list": [
    "recipe[wal_e::recover]"
  ]
}

and specify all 3 of the S3 attributes needed by wal-e

optionally add the recover options

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2",
  "recover": {
    "recovery_target_timeline": "latest",
    "recovery_target_time": null
  }
}

Bringing a 'Recover' server online

WARNING: the recover recipe will delete all of the pg data directory before pulling WAL-e files from S3, you may want to recover to a new server instead of running recover on your master

  • take the master offline
    • ensure that it will no longer write to S3 with WAL-e
  • run the recover recipe to bring up a new server in recovery mode
  • once the chef script has completed, you will want to manually verify the server is up and has recovered all data
    • ssh onto box and tail postgres log
      • sudo su - postgres
      • tail -f /var/log/postgresql/postgresql-9.1-main.log
    • once server is online verify data is present
      • psql -c "\l"
      • psql app1 -c "select * from sample_data_1;"
  • Once you have verified the server is up and running and all data is present you will want to 'promote' this server to be the new master
    • change your chef configuration for this node
    • it is recommended to change the S3 endpoint to a new directory
      • see https://groups.google.com/d/msg/wal-e/Mh0zeczwBCw/xGZ7KSEvYiwJ for more information on maintaining separate prefixes for each server that becomes a primary (master) server
      • Example wale_s3_prefix
        • Original Master: s3://my-org/pg_cluster_1/wal-e-1
        • Time passes Original Master goes down, bring up Master2 and change the prefix
        • Master2: s3://my-org/pg_cluster_1/wal-e-2
        • Time passes and Master2 goes down, bring up Master3 and change the prefix
        • Master3: s3://my-org/pg_cluster_1/wal-e-3
    • provision it using chef again
    • and then verify that it is now writing WAL files to S3 on the new prefix directory

WAL-e Cookbook TODO List

  • add recipe for standby server
  • move attributes to encrypted data bag
  • allow configuration of cron settings for backup creation

Contributing

  • Fork the repository on Github
  • Create a named feature branch (like add_component_x)
  • Write your changes
  • Submit a Pull Request using Github

License

wal-e cookbook is released under the MIT License.

See the LICENSE file

Random Notes

# if you want to continually install latest HEAD version of WAL-e 
# run this before the wal_e::common recipe executes - (don't do this in production)
file "/installs/wal-e-marker" do
  content " "
end

psql -c "show data_directory"
# /var/lib/postgresql/9.1/main

psql -c "show config_file"
# /etc/postgresql/9.1/main/postgresql.conf

/var/log/postgresql/postgresql-9.1-main.log 

sudo pip install -e git+git://github.com/wal-e/wal-e.git@#egg=wal-e
sudo pip install -e git+git://github.com/wal-e/wal-e.git@v0.6.2#egg=wal-e
sudo pip install -e git+git://github.com/wal-e/wal-e.git@17417b9552d96881ce4a86cbe6bae24f8a5ef241#egg=wal-e

pip list
sudo pip uninstall --yes --quiet wal-e

/usr/local/bin/wal-e version
# => 0.7.2.dev

About

chef cookbook for installing wal-e (postgres continuous archiving of WAL files to s3)

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages