Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

expose zstd compression #1326

Merged
merged 1 commit into from
Aug 29, 2022
Merged

expose zstd compression #1326

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4, LZMA, or Brotli compression, multiple processors, and non-exclusive base backups for Postgres. More information on the original design and implementation of WAL-G can be found on the Citus Data blog post ["Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres"](https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/).
WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4, LZMA, ZSTD, or Brotli compression, multiple processors, and non-exclusive base backups for Postgres. More information on the original design and implementation of WAL-G can be found on the Citus Data blog post ["Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres"](https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-faster-restores-for-postgres/).

**Table of Contents**
- [Installation](#installation)
Expand Down Expand Up @@ -55,8 +55,8 @@ To configure where WAL-G stores backups, please consult the [Storages](STORAGES.
### Compression
* `WALG_COMPRESSION_METHOD`

To configure the compression method used for backups. Possible options are: `lz4`, `lzma`, `brotli`. The default method is `lz4`. LZ4 is the fastest method, but the compression ratio is bad.
LZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli is a good trade-off between speed and compression ratio, which is about 3 times better than LZ4.
To configure the compression method used for backups. Possible options are: `lz4`, `lzma`, `zstd`, `brotli`. The default method is `lz4`. LZ4 is the fastest method, but the compression ratio is bad.
LZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli and zstd are a good trade-off between speed and compression ratio, which is about 3 times better than LZ4.

### Encryption

Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.4.1
github.com/Azure/go-autorest/autorest v0.11.21
github.com/DATA-DOG/godog v0.7.14-0.20190529133509-96731eaefa46
github.com/DataDog/zstd v1.4.4
github.com/DataDog/zstd v1.5.3-0.20220606203749-fd035e54e312
github.com/RoaringBitmap/roaring v0.4.21
github.com/aws/aws-sdk-go v1.44.7
github.com/blang/semver v3.5.1+incompatible
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ github.com/DATA-DOG/go-sqlmock v1.3.3 h1:CWUqKXe0s8A2z6qCgkP4Kru7wC11YoAnoupUKFD
github.com/DATA-DOG/go-sqlmock v1.3.3/go.mod h1:f/Ixk793poVmq4qj/V1dPUg2JEAKC73Q5eFN3EC/SaM=
github.com/DATA-DOG/godog v0.7.14-0.20190529133509-96731eaefa46 h1:Y4moA9AVGfQbZ4f+LzrTnfk9mHXj/HseIFu1E4wvZPo=
github.com/DATA-DOG/godog v0.7.14-0.20190529133509-96731eaefa46/go.mod h1:GQyRe8MROPiE5oQeEdXWTlfOmQRyMSXFkNYIORsW4xk=
github.com/DataDog/zstd v1.4.4 h1:+IawcoXhCBylN7ccwdwf8LOH2jKq7NavGpEPanrlTzE=
github.com/DataDog/zstd v1.4.4/go.mod h1:1jcaCB/ufaK+sKp1NBhlGmpz41jOoPQ35bpF36t7BBo=
github.com/DataDog/zstd v1.5.3-0.20220606203749-fd035e54e312 h1:MB4OhDISHzUx+9VIhkCinijzgGAsyrFqko4PjPjJmbw=
github.com/DataDog/zstd v1.5.3-0.20220606203749-fd035e54e312/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
github.com/Microsoft/go-winio v0.4.14 h1:+hMXMk01us9KgxGb7ftKQt2Xpf5hH/yky+TDA+qxleU=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
Expand Down
3 changes: 2 additions & 1 deletion internal/compression/compression_default.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@ import (
"github.com/wal-g/wal-g/internal/compression/zstd"
)

var CompressingAlgorithms = []string{lz4.AlgorithmName, lzma.AlgorithmName}
var CompressingAlgorithms = []string{lz4.AlgorithmName, lzma.AlgorithmName, zstd.AlgorithmName}

var Compressors = map[string]Compressor{
lz4.AlgorithmName: lz4.Compressor{},
lzma.AlgorithmName: lzma.Compressor{},
zstd.AlgorithmName: zstd.Compressor{},
}

var Decompressors = []Decompressor{
Expand Down