Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(bench): add compression benchmark #3090

Merged
merged 2 commits into from
Jun 9, 2022
Merged

test(bench): add compression benchmark #3090

merged 2 commits into from
Jun 9, 2022

Conversation

MrCroxx
Copy link
Contributor

@MrCroxx MrCroxx commented Jun 9, 2022

What's changed and what's your intention?

Introduce compression benchmark to compare the encode speed between block compression and stream compression.

Stream compression is helpful to better align block size in SSTs, which benefits the file cache system by reducing internal fragmentation.

Results

The uncompressed data is consists of (tables * pairs) encoded kv pairs:

Format:
key: b't' + table id (u32) + user key (u64) + cell id (u32)
value: [random slice] (size = `vsize`)
algorithm tables pairs vsize uc size c size c rate time (us) time diff
lz4 block 10 100 8 25000 13943 0.558 4.35 + 147.45 -
lz4 stream 10 100 8 25000 13943 0.558 187.51 +23.5%
lz4 block 10 100 16 33000 22931 0.695 5.08 + 216.46 -
lz4 stream 10 100 16 33000 22931 0.695 256.97 +16.0%
lz4 block 10 100 32 49000 38942 0.795 5.75 + 407.91 -
lz4 stream 10 100 32 49000 38942 0.795 469.22 +13.4%
lz4 block 10 100 64 81000 70959 0.876 8.22 + 946.09 -
lz4 stream 10 100 64 81000 70959 0.876 1005.70 +5.4%
  • For the value is generated randomly, the larger vsize is, the lower compression rate is. The compression rate should be higher with real-world workload. And the speed will be also faster.

In conclusion, The larger each uncompressed entry size is, the smaller the speed difference is. And there is no obvious difference between compression rates.

Because L0 doesn't enable any compression algorithms, IMO, the extra compress time is bearable for compection only.

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

#198

@MrCroxx MrCroxx added the component/test Test related issue. label Jun 9, 2022
@MrCroxx MrCroxx self-assigned this Jun 9, 2022
@github-actions github-actions bot added Invalid PR Title and removed component/test Test related issue. labels Jun 9, 2022
@MrCroxx MrCroxx changed the title bench(compression): add compression benchmark test(bench): add compression benchmark Jun 9, 2022
@github-actions github-actions bot added component/test Test related issue. and removed Invalid PR Title labels Jun 9, 2022
@codecov
Copy link

codecov bot commented Jun 9, 2022

Codecov Report

Merging #3090 (8fb9821) into main (4e9090d) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #3090      +/-   ##
==========================================
- Coverage   73.48%   73.47%   -0.01%     
==========================================
  Files         734      734              
  Lines      100338   100338              
==========================================
- Hits        73730    73728       -2     
- Misses      26608    26610       +2     
Flag Coverage Δ
rust 73.47% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/storage/src/hummock/sstable/block.rs 97.22% <100.00%> (ø)
src/meta/src/barrier/mod.rs 69.13% <0.00%> (-0.33%) ⬇️
src/common/src/types/ordered_float.rs 24.70% <0.00%> (-0.20%) ⬇️

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@MrCroxx MrCroxx merged commit c8c4acf into main Jun 9, 2022
@MrCroxx MrCroxx deleted the xx/bench-compression branch June 9, 2022 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/test Test related issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants