Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sst_dump recompress show #blocks compressed and not compressed #4835

Closed
wants to merge 8 commits into from
Closed

sst_dump recompress show #blocks compressed and not compressed #4835

wants to merge 8 commits into from

Conversation

thatsafunnyname
Copy link
Contributor

@thatsafunnyname thatsafunnyname commented Dec 31, 2018 •
edited by ajkr
Loading

Closes #1474
Helps show when the 12.5% threshold for GoodCompressionRatio (originally from ldb) is hit.

Example output:

> ./sst_dump --file=/tmp/test.sst --command=recompress
from [] to []
Process /tmp/test.sst
Sst file format: block-based
Block Size: 16384
Compression: kNoCompression           Size:  122579836 Blocks:   2300 Compressed:      0 (  0.0%) Not compressed (ratio):   2300 (100.0%) Not compressed (abort):      0 (  0.0%)
Compression: kSnappyCompression       Size:   46289962 Blocks:   2300 Compressed:   2119 ( 92.1%) Not compressed (ratio):    181 (  7.9%) Not compressed (abort):      0 (  0.0%)
Compression: kZlibCompression         Size:   29689825 Blocks:   2300 Compressed:   2301 (100.0%) Not compressed (ratio):      0 (  0.0%) Not compressed (abort):      0 (  0.0%)
Unsupported compression type: kBZip2Compression.
Compression: kLZ4Compression          Size:   44785490 Blocks:   2300 Compressed:   1950 ( 84.8%) Not compressed (ratio):    350 ( 15.2%) Not compressed (abort):      0 (  0.0%)
Compression: kLZ4HCCompression        Size:   37498895 Blocks:   2300 Compressed:   2301 (100.0%) Not compressed (ratio):      0 (  0.0%) Not compressed (abort):      0 (  0.0%)
Unsupported compression type: kXpressCompression.
Compression: kZSTD                    Size:   32208707 Blocks:   2300 Compressed:   2301 (100.0%) Not compressed (ratio):      0 (  0.0%) Not compressed (abort):      0 (  0.0%)

@thatsafunnyname
Show #blocks compressed and not compressed
5188a5e 
For: facebook#1474
Helps show when the 12.5% threshold for GoodCompressionRatio (originally from ldb) is hit.
@thatsafunnyname
Show #blocks compressed and not compressed
97df471 
For: facebook#1474
Helps show when the 12.5% threshold for GoodCompressionRatio (originally from ldb) is hit.
@thatsafunnyname thatsafunnyname mentioned this pull request Jan 2, 2019
Closed
siying
siying reviewed Jan 2, 2019
View reviewed changes
Copy link
Contributor

@siying siying left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great! Only one comment.

tools/sst_dump_tool.cc Outdated
@@ -158,7 +158,8 @@ Status SstFileDumper::DumpTable(const std::string& out_filename) {
}

uint64_t SstFileDumper::CalculateCompressedTableSize(
const TableBuilderOptions& tb_options, size_t block_size) {
const TableBuilderOptions& tb_options, size_t block_size,
uint64_t& num_data_blocks) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We follow Google C++ Style and it bans using reference as output argument: https://google.github.io/styleguide/cppguide.html#Reference_Arguments

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I changed it to a pointer.

riversand963
riversand963 reviewed Jan 8, 2019
View reviewed changes
Copy link
Contributor

@riversand963 riversand963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @thatsafunnyname for the contribution. I've left some comments.

tools/sst_dump_tool.cc
@@ -188,6 +189,7 @@ uint64_t SstFileDumper::CalculateCompressedTableSize(
exit(1);
}
uint64_t size = table_builder->FileSize();
*num_data_blocks = table_builder->GetTableProperties().num_data_blocks;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add assert(num_data_blocks != nullptr) to make static checker happy?

tools/sst_dump_tool.cc
const uint64_t not_compressed_blocks =
opts.statistics->getAndResetTickerCount(NUMBER_BLOCK_NOT_COMPRESSED);
if( (compressed_blocks + not_compressed_blocks) > num_data_blocks ){
num_data_blocks = compressed_blocks + not_compressed_blocks;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment on when (compressed_blocks + not_compressed_blocks) > num_data_blocks is true? Thanks

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a comment, it is when the option enable_index_compression is true.

@sagar0
Copy link
Contributor

sagar0 commented Jan 8, 2019

I restarted the failing travis-lite build. This should be fixed, hopefully.

facebook-github-bot
facebook-github-bot reviewed Sep 9, 2019
View reviewed changes
Copy link
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@siying is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@siying siying self-assigned this Sep 10, 2019
@siying
Copy link
Contributor

siying commented Sep 10, 2019

@thatsafunnyname Some errors when building it:

internal_repo_rocksdb/repo/tools/sst_dump_tool.cc:206:20: error: 'stats_level_' is a private member of 'rocksdb::Statistics'
  opts.statistics->stats_level_ = StatsLevel::kAll;
                   ^
stderr: internal_repo_rocksdb/repo/tools/sst_dump_tool.cc:206:20: error: 'stats_level_' is a private member of 'rocksdb::Statistics'
  opts.statistics->stats_level_ = StatsLevel::kAll;
                   ^

I guess master has changed, so this needs to be accessed through a setter. If you fix it, I can land it.

@facebook-github-bot
Copy link
Contributor

@thatsafunnyname has updated the pull request. Re-import the pull request

@thatsafunnyname
Copy link
Contributor Author

Closing in favour of #5791 which has been updated against master and is using a setter to set the stats level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@siying siying siying left review comments

@facebook-github-bot facebook-github-bot facebook-github-bot left review comments

@riversand963 riversand963 riversand963 left review comments

Assignees

@siying siying

Labels
CLA Signed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Show number of blocks compressed and not compressed when using "sst_dump --show_compression_sizes".
5 participants
@thatsafunnyname @sagar0 @siying @facebook-github-bot @riversand963