compress boltdb files to gzip while uploading from shipper #2507
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds supports for compressing boltdb files before uploading which is expected to reduce file sizes by 1/3rd.
The files which are compressed would have
.gz
extension and until we build a compactor we would have both compressed and uncompressed files in the storage which is taken care of in the read path.While it is not necessary but it would be good to merge PR #2487 before merging this to avoid having some files in both compressed and uncompressed form.
It uses gzip pool that we already have built for reducing allocations while compressing/decompressing chunks.
Checklist