compress boltdb files to gzip while uploading from shipper #2507
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2507 +/- ##
==========================================
+ Coverage 63.45% 63.60% +0.14%
==========================================
Files 163 163
Lines 14263 14273 +10
==========================================
+ Hits 9051 9078 +27
+ Misses 4499 4481 -18
- Partials 713 714 +1
|
owen-d
approved these changes
Aug 20, 2020
| @@ -142,7 +147,28 @@ func SetupDBTablesAtPath(t *testing.T, tableName, path string, dbs map[string]DB | |||
|
|
|||
| for name, dbRecords := range dbs { | |||
| AddRecordsToDB(t, filepath.Join(tablePath, name), boltIndexClient, dbRecords.Start, dbRecords.NumRecords) | |||
| if compressRandomFiles && rand.Intn(2) == 0 { | |||
There was a problem hiding this comment.
In general, I think adding randomness like this to tests is bad practice because it creates flakey test cases. Instead, we can run the test case twice, once against compressed files and once against uncompressed files, or you can make this "randomness" deterministic by basing it off of the index like
var i int
for name, dbRecords := range dbs {
if compress && i % 2 == 0 {
// compress it
}
i++
}There was a problem hiding this comment.
Fixed it. Thanks for the review!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Adds supports for compressing boltdb files before uploading which is expected to reduce file sizes by 1/3rd.
The files which are compressed would have
.gzextension and until we build a compactor we would have both compressed and uncompressed files in the storage which is taken care of in the read path.While it is not necessary but it would be good to merge PR #2487 before merging this to avoid having some files in both compressed and uncompressed form.
It uses gzip pool that we already have built for reducing allocations while compressing/decompressing chunks.
Checklist