Go Makefile
Switch branches/tags
Permalink
Failed to load latest commit information.
cmd/wal-g Fix region handling Aug 18, 2017
test_tools Test tools edit Aug 18, 2017
vendor Remove self-duplicative vendor Aug 16, 2017
.travis.yml Add OS, arch information to release tarball name Aug 17, 2017
CONTRIBUTORS Add Patrick Figel as a contributor Aug 31, 2017
LICENSE.md License Aug 17, 2017
Makefile Produce wal-g binary with `make test` Aug 17, 2017
README.md Add S3 storage class configuration option Aug 31, 2017
backup.go Upload concurrency stream variable Aug 10, 2017
backup_test.go travis Aug 17, 2017
compress.go Add comments for testing tools. Aug 9, 2017
compress_test.go README badges Aug 18, 2017
connect.go Support extra PostgreSQL environment variables Aug 18, 2017
connect_test.go Rename katie31 to wal-g org Aug 16, 2017
decompress.go Add help subcommands Aug 16, 2017
decompress_test.go Rename katie31 to wal-g org Aug 16, 2017
error.go Fix comments, write to stderr for errors, change error handling for c… Aug 8, 2017
extract.go Random exponential backoff Aug 17, 2017
extract_test.go travis Aug 17, 2017
make.go Fix comments, write to stderr for errors, change error handling for c… Aug 8, 2017
structs.go Add S3 storage class configuration option Aug 31, 2017
structs_test.go travis Aug 17, 2017
tar.go golint Aug 8, 2017
timer.go README badges Aug 18, 2017
upload.go Add S3 storage class configuration option Aug 31, 2017
upload_storage_class_test.go Add S3 storage class configuration option Aug 31, 2017
upload_test.go Add S3 storage class configuration option Aug 31, 2017
utility.go Fix comments, write to stderr for errors, change error handling for c… Aug 8, 2017
utility_test.go Rename katie31 to wal-g org Aug 16, 2017
walk.go Add help subcommands Aug 16, 2017
walk_test.go travis Aug 17, 2017

README.md

WAL-G

Build Status Go Report Card

WAL-G is an archival restoration tool for Postgres.

WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4 compression, multiple processors and non-exclusive base backups for Postgres. More information on the design and implementation of WAL-G can be found on the Citus Data blog post "Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres".

Table of Contents

Installation

A precompiled binary for Linux AMD 64 of the latest version of WAL-G can be obtained under the Releases tab.

To decompress the binary, use:

tar -zxvf wal-g.linux-amd64.tar.gz

For other incompatible systems, please consult the Development section for more information.

Configuration

Required

To connect to Amazon S3, WAL-G requires that these variables be set:

  • WALE_S3_PREFIX (eg. s3://bucket/path/to/folder)

WAL-G determines AWS credentials like other AWS tools. You can set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (optionally with AWS_SECURITY_TOKEN), or ~/.aws/credentials (optionally with AWS_PROFILE), or you can set nothing to automatically fetch credentials from the EC2 metadata service.

WAL-G uses the usual PostgreSQL environment variables to configure its connection, especially including PGHOST, PGPORT, PGUSER, and PGPASSWORD/PGPASSFILE/~/.pgpass.

Optional

WAL-G can automatically determine the S3 bucket's region using s3:GetBucketLocation, but if you wish to avoid this API call or forbid it from the applicable IAM policy, specify:

  • AWS_REGION(eg. us-west-2)

Concurrency values can be configured using:

  • WALG_DOWNLOAD_CONCURRENCY

To configure how many goroutines to use during extraction, use WALG_DOWNLOAD_CONCURRENCY. By default, WAL-G uses the minimum of the number of files to extract and 10.

  • WALG_UPLOAD_CONCURRENCY

To configure how many concurrency streams to use during backup uploading, use WALG_UPLOAD_CONCURRENCY. By default, WAL-G uses 10 streams.

  • WALG_S3_STORAGE_CLASS

To configure the S3 storage class used for backup files, use WALG_S3_STORAGE_CLASS. By default, WAL-G uses the "STANDARD" storage class. Other supported values include "STANDARD_IA" for Infrequent Access and "REDUCED_REDUNDANCY" for Reduced Redundancy.

Usage

WAL-G currently supports these commands:

  • backup-fetch

When fetching base backups, the user should pass in the name of the backup and a path to a directory to extract to. If this directory does not exist, WAL-G will create it and any dependent subdirectories.

wal-g backup-fetch ~/extract/to/here example-backup

WAL-G can also fetch the latest backup using:

wal-g backup-fetch ~/extract/to/here LATEST
  • backup-push

When uploading backups to S3, the user should pass in the path containing the backup started by Postgres as in:

wal-g backup-push /backup/directory/path
  • wal-fetch

When fetching WAL archives from S3, the user should pass in the archive name and the name of the file to download to. This file should not exist as WAL-G will create it for you.

wal-g wal-fetch example-archive new-file-name
  • wal-push

When uploading WAL archives to S3, the user should pass in the absolute path to where the archive is located.

wal-g wal-push /path/to/archive

Development

Installing

To compile and build the binary:

go get github.com/wal-g/wal-g
make all

Users can also install WAL-G by using make install. Specifying the GOBIN environment variable before installing allows the user to specify the installation location. On default, make install puts the compiled binary in go/bin.

export GOBIN=/usr/local/bin
make install

Testing

WAL-G relies heavily on unit tests. These tests do not require S3 configuration as the upload/download parts are tested using mocked objects. For more information on testing, please consult test_tools.

WAL-G will perform a round-trip compression/decompression test that generates a directory for data (eg. data...), compressed files (eg. compressed), and extracted files (eg. extracted). These directories will only get cleaned up if the files in the original data directory match the files in the extracted one.

Test coverage can be obtained using:

go test -v -coverprofile=coverage.out
go tool cover -html=coverage.out

Authors

See also the list of contributors who participated in this project.

License

This project is licensed under the Apache License, Version 2.0, but the lzo support is licensed under GPL 3.0+. Please refer to the LICENSE.md file for more details.

Acknowledgements

WAL-G would not have happened without the support of Citus Data

WAL-G came into existence as a result of the collaboration between a summer engineering intern at Citus, Katie Li, and Daniel Farina, the original author of WAL-E who currently serves as a principal engineer on the Citus Cloud team. Citus Data also has an open source extension to Postgres that distributes database queries horizontally to deliver scale and performance.