Skip to content

Commit b13b927

Browse files
author
Ibrahim Jarif
authored
Compress/Encrypt Blocks in the background (#1227)
This commit adds support for compression/encryption in the background. All incoming writes are saved to a common buffer and then worker go routine pick up each block (a block is just a start and end point in the main buffer). The worker goroutines also fix the table start and end points. We fix the interleaving space (which might be caused because the block was compressed) in the builder.Finish() call. All the blocks are copied over to their correct position. Master branch vs This commit (new) Benchmarks. ``` benchstat master.txt new.txt name old time/op new time/op delta (-ve is better. Reduction in time) no_compression-16 177ms ± 1% 177ms ± 2% ~ (p=0.690 n=5+5) encryption-16 313ms ± 4% 245ms ± 3% -21.69% (p=0.008 n=5+5) zstd_compression/level_1-16 330ms ± 1% 227ms ± 2% -31.22% (p=0.008 n=5+5) zstd_compression/level_3-16 345ms ± 2% 227ms ± 3% -34.39% (p=0.008 n=5+5) zstd_compression/level_15-16 10.7s ± 1% 1.2s ± 0% -88.96% (p=0.008 n=5+5) name old speed new speed delta (+ve is better. Speed improvement) no_compression-16 471MB/s ± 1% 471MB/s ± 2% ~ (p=0.690 n=5+5) encryption-16 266MB/s ± 4% 340MB/s ± 2% +27.67% (p=0.008 n=5+5) zstd_compression/level_1-16 252MB/s ± 1% 367MB/s ± 2% +45.40% (p=0.008 n=5+5) zstd_compression/level_3-16 241MB/s ± 2% 367MB/s ± 3% +52.44% (p=0.008 n=5+5) zstd_compression/level_15-16 7.80MB/s ± 1% 70.62MB/s ± 0% +805.33% (p=0.008 n=5+5) ``` Please look at #1227 (comment) for the code used for benchmarking.
1 parent bdb2b13 commit b13b927

File tree

8 files changed

+293
-116
lines changed

8 files changed

+293
-116
lines changed

db2_test.go

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -669,16 +669,13 @@ func TestL0GCBug(t *testing.T) {
669669
// Simulate a crash by not closing db1 but releasing the locks.
670670
if db1.dirLockGuard != nil {
671671
require.NoError(t, db1.dirLockGuard.release())
672+
db1.dirLockGuard = nil
672673
}
673674
if db1.valueDirGuard != nil {
674675
require.NoError(t, db1.valueDirGuard.release())
676+
db1.valueDirGuard = nil
675677
}
676-
for _, f := range db1.vlog.filesMap {
677-
require.NoError(t, f.fd.Close())
678-
}
679-
require.NoError(t, db1.registry.Close())
680-
require.NoError(t, db1.lc.close())
681-
require.NoError(t, db1.manifest.close())
678+
require.NoError(t, db1.Close())
682679

683680
db2, err := Open(opts)
684681
require.NoError(t, err)

options.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,7 @@ func DefaultOptions(path string) Options {
155155

156156
func buildTableOptions(opt Options) table.Options {
157157
return table.Options{
158+
TableSize: uint64(opt.MaxTableSize),
158159
BlockSize: opt.BlockSize,
159160
BloomFalsePositive: opt.BloomFalsePositive,
160161
LoadingMode: opt.TableLoadingMode,

0 commit comments

Comments
 (0)