Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


chef cookbook for installing wal-e (postgres continuous archiving to s3)

See sample usage: here


This cookbook has been developed and tested on ubuntu servers only


  • postgresql - optionally use databox-cookbook which includes postgresql
  • python - wal-e is installed using python pip (package management system)
  • git - required to install wal-e using pip over git



  • ['wal_e']['aws_access_key_id'] : AWS Access Key
  • ['wal_e']['aws_secret_access_key'] : AWS Secret
  • ['wal_e']['wale_s3_prefix'] : S3 url for the bucket where postgres WAL and backups will be stored
    • i.e. s3://your-lower-case-bucket-name/whatever/wal-e/
  • ['wal_e']['wale_git_install_reference'] : Specify which commit to install wal-e from
    • Default is HEAD, which will install from the HEAD of master
    • Optionally set this to a git commit SHA or a git TAG to install a specific version
    • NOTE: the recipe uses a marker file with the value from this configuration to determine if it should install a different version of WAL-e, so re-running it when set to HEAD will only run the first time, it will not pull newer commits unless you update this to a value besides HEAD, i.e. a SHA or TAG




Include wal_e in your node's run_list:

  "run_list": [

and specify all 3 of the S3 attributes needed by wal-e

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2"


Include wal_e in your node's run_list:

  "run_list": [

and specify all 3 of the S3 attributes needed by wal-e

optionally add the recover options

"wal_e": {
  "aws_secret_access_key": "SECRET",
  "aws_access_key_id": "ACCESS_KEY",
  "wale_s3_prefix": "s3://your-lower-case-bucket-name/wal-e/or_whatever",
  "wale_git_install_reference": "v0.6.2",
  "recover": {
    "recovery_target_timeline": "latest",
    "recovery_target_time": null

Bringing a 'Recover' server online

WARNING: the recover recipe will delete all of the pg data directory before pulling WAL-e files from S3, you may want to recover to a new server instead of running recover on your master

  • take the master offline
    • ensure that it will no longer write to S3 with WAL-e
  • run the recover recipe to bring up a new server in recovery mode
  • once the chef script has completed, you will want to manually verify the server is up and has recovered all data
    • ssh onto box and tail postgres log
      • sudo su - postgres
      • tail -f /var/log/postgresql/postgresql-9.1-main.log
    • once server is online verify data is present
      • psql -c "\l"
      • psql app1 -c "select * from sample_data_1;"
  • Once you have verified the server is up and running and all data is present you will want to 'promote' this server to be the new master
    • change your chef configuration for this node
    • it is recommended to change the S3 endpoint to a new directory
      • see for more information on maintaining separate prefixes for each server that becomes a primary (master) server
      • Example wale_s3_prefix
        • Original Master: s3://my-org/pg_cluster_1/wal-e-1
        • Time passes Original Master goes down, bring up Master2 and change the prefix
        • Master2: s3://my-org/pg_cluster_1/wal-e-2
        • Time passes and Master2 goes down, bring up Master3 and change the prefix
        • Master3: s3://my-org/pg_cluster_1/wal-e-3
    • provision it using chef again
    • and then verify that it is now writing WAL files to S3 on the new prefix directory

WAL-e Cookbook TODO List

  • add recipe for standby server
  • move attributes to encrypted data bag
  • allow configuration of cron settings for backup creation


  • Fork the repository on Github
  • Create a named feature branch (like add_component_x)
  • Write your changes
  • Submit a Pull Request using Github


wal-e cookbook is released under the MIT License.

See the LICENSE file

Random Notes

# if you want to continually install latest HEAD version of WAL-e 
# run this before the wal_e::common recipe executes - (don't do this in production)
file "/installs/wal-e-marker" do
  content " "

psql -c "show data_directory"
# /var/lib/postgresql/9.1/main

psql -c "show config_file"
# /etc/postgresql/9.1/main/postgresql.conf


sudo pip install -e git+git://
sudo pip install -e git+git://
sudo pip install -e git+git://

pip list
sudo pip uninstall --yes --quiet wal-e

/usr/local/bin/wal-e version
# =>


chef cookbook for installing wal-e (postgres continuous archiving of WAL files to s3)




You can’t perform that action at this time.