-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallelize compression/decompression during backup and restore #9856
Parallelize compression/decompression during backup and restore #9856
Conversation
Drop-in of pgzip for gzip
@pliu thanks for the contribution. Whilst the code changes in this PR are trivial, there may be significant changes to the behaviour of the database. In order to accept this contribution it would be good to see some benchmarks or performance tests. Further, the core team would probably need to look into the third part library and check it's suitable for inclusion. /cc @aanthony1243 |
@e-dard Do you have internal benchmark/performance tests that you can run against the change or is this something I will need to perform? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @pliu this is looking fine. Thanks for this. We have recently started using dep
as our dependency management tool. Do you mind if I add a commit to the PR with the correct dep
update for pgzip
?
agreed, once we update for dep, this is ok to merge. |
@e-dard Feel free to make the necessary changes :) Thank you! |
b334800
to
4035591
Compare
@e-dard Done |
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Parallelize compression/decompression during backup and restore
Drop-in of pgzip for gzip
Compression for the backup and restore commands of influxd were able to download the data from the database quickly, but subsequent compression and decompression was done in a single-threaded manner that is extremely slow.
Required for all non-trivial PRs