Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


chef cookbook for installing wal-e (postgres continuous archiving to s3)

See sample usage: here


This cookbook has been developed and tested on ubuntu servers only


  • postgresql - optionally use databox-cookbook which includes postgresql
  • python - wal-e is installed using python pip (package management system)
  • git - required to install wal-e using pip over git



  • ['wal_e']['aws_access_key_id'] : AWS Access Key
  • ['wal_e']['aws_secret_access_key'] : AWS Secret
  • ['wal_e']['wale_s3_prefix'] : S3 url for the bucket where postgres WAL and backups will be stored
    • i.e. s3://your-lower-case-bucket-name/whatever/wal-e/
  • ['wal_e']['wale_git_install_reference'] : Specify which commit to install wal-e from
    • Default is HEAD, which will install from the HEAD of master
    • Optionally set this to a git commit SHA or a git TAG to install a specific version
    • NOTE: the recipe uses a marker file with the value from this configuration to determine if it should install a different version of WAL-e, so re-running it when set to HEAD will only run the first time, it will not pull newer commits unless you update this to a value besides HEAD, i.e. a SHA or TAG




Include wal_e in your node's run_list:

  "run_list": [

and specify all 3 of the S3 attributes needed by wal-e

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2"


Include wal_e in your node's run_list:

  "run_list": [

and specify all 3 of the S3 attributes needed by wal-e

optionally add the recover options

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2",
  "recover": {
    "recovery_target_timeline": "latest",
    "recovery_target_time": null

Bringing a 'Recover' server online

WARNING: the recover recipe will delete all of the pg data directory before pulling WAL-e files from S3, you may want to recover to a new server instead of running recover on your master

  • take the master offline
    • ensure that it will no longer write to S3 with WAL-e
  • run the recover recipe to bring up a new server in recovery mode
  • once the chef script has completed, you will want to manually verify the server is up and has recovered all data
    • ssh onto box and tail postgres log
      • sudo su - postgres
      • tail -f /var/log/postgresql/postgresql-9.1-main.log
    • once server is online verify data is present
      • psql -c "\l"
      • psql app1 -c "select * from sample_data_1;"
  • Once you have verified the server is up and running and all data is present you will want to 'promote' this server to be the new master
    • change your chef configuration for this node
    • it is recommended to change the S3 endpoint to a new directory
      • see for more information on maintaining separate prefixes for each server that becomes a primary (master) server
      • Example wale_s3_prefix
        • Original Master: s3://my-org/pg_cluster_1/wal-e-1
        • Time passes Original Master goes down, bring up Master2 and change the prefix
        • Master2: s3://my-org/pg_cluster_1/wal-e-2
        • Time passes and Master2 goes down, bring up Master3 and change the prefix
        • Master3: s3://my-org/pg_cluster_1/wal-e-3
    • provision it using chef again
    • and then verify that it is now writing WAL files to S3 on the new prefix directory

WAL-e Cookbook TODO List

  • add recipe for standby server
  • move attributes to encrypted data bag
  • allow configuration of cron settings for backup creation


  • Fork the repository on Github
  • Create a named feature branch (like add_component_x)
  • Write your changes
  • Submit a Pull Request using Github


wal-e cookbook is released under the MIT License.

See the LICENSE file

Random Notes

# if you want to continually install latest HEAD version of WAL-e 
# run this before the wal_e::common recipe executes - (don't do this in production)
file "/installs/wal-e-marker" do
  content " "

psql -c "show data_directory"
# /var/lib/postgresql/9.1/main

psql -c "show config_file"
# /etc/postgresql/9.1/main/postgresql.conf


sudo pip install -e git+git://
sudo pip install -e git+git://
sudo pip install -e git+git://

pip list
sudo pip uninstall --yes --quiet wal-e

/usr/local/bin/wal-e version
# =>


chef cookbook for installing wal-e (postgres continuous archiving of WAL files to s3)







No packages published