expose zstd compression #1326
Merged
expose zstd compression #1326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* expose upgraded zstd compressor * mention zstd in docs * zstd 1.5.2 + patches * remove unused lines in go.sum
serprex
approved these changes
Aug 24, 2022
|
FWIW I saw one report of brotli corruption: a file properly decrypted by gpg, but unexctractable. Probably, due to cosmic rays or something. @heinerstilz do you validate you backups? Will you use Zstd? We need someone who will warn us if something is still wrong with Zstd. |
|
Implementing our backup solution for PostgreSQL, the idea is to go with zstd unless we find an issue with it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Database name
PostgreSQL
Pull request description
Describe what this PR fix
The zstd compression library is already supported internally in WAL-G. It seems to have been disabled because of a data corruption bug in zstd that has meanwhile been resolved (DataDog/zstd#39).
It has been mentioned that it might be good to bring zstd back (see discussion on #300).
This PR
In our measurements with PostgreSQL and WAL-G on a ~270 GB dataset, zstd was significantly less CPU intensive than brotli:
1.6x less real time and 1.25x less user time backing up/compressing with WALG_UPLOAD_CONCURRENCY=8
1.6x less real time and 2.8x less user time restoring/decompressing.
Compression ratio was around 2.4x for both.
Another measurement by a different party that seems to support zstd's performance advantage over brotli: https://peazip.github.io/fast-compression-benchmark-brotli-zstandard.html#:~:text=Comparing%20Brotli%20and%20Zstandard%20extraction,twice%20as%20fast%20as%20Brotli.
Please provide steps to test this PR
Which of the tests makes most sense to be adapted for zstd?