[largeNbDicts] Fix decompression segfault; Add additional benchmark metrics #3205
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Benchmarking decompression results in a segfault in `createCompressInstructions` because `cctxParams` is NULL. Skip running that function if we are not benching compression.
Summary: Add column headers and data for whether it's a compression or a decompression run, compression level, nbDicts and dictAttachPref in additional to compr/decompr speed. Test Plan: Example output: ``` ./largeNbDicts Compression/Decompression,Level,nbDicts,dictAttachPref,Speed Compression,1,1,0,300.9 Compression,1,1,1,296.4 Compression,1,1,2,307.8 Compression,1,10,0,292.3 Compression,1,100,0,293.3 Compression,3,110,0,106.0 Decompression,-1,110,-1,155.6 Decompression,-1,110,-1,709.4 Decompression,-1,120,-1,709.1 Decompression,-1,120,-1,734.6 ```
zhuhan0
changed the title
Summary: Added an option -p# where -p0 (default) sets the aggregation method to fastest speed while -p1 sets the aggregation method to median. Also added a new column in the csv file to report this option's value. Test Plan: `` $ ./largeNbDicts -1 --nbDicts=1 -D ~/benchmarks/html/html_8_16K.32K.dict ~/benchmarks/html/html_8_16K/* loading 7450 files... created src buffer of size 83.4 MB split input into 7450 blocks loading dictionary /home/zhuhan/benchmarks/html/html_8_16K.32K.dict compressing at level 1 without dictionary : Ratio=3.03 (28827863 bytes) compressed using a 32768 bytes dictionary : Ratio=4.28 (20410262 bytes) generating 1 dictionaries, using 0.1 MB of memory Compression Speed : 306.0 MB/s Fastest Speed : 310.6 MB/s $ ./largeNbDicts -1 --nbDicts=1 -p1 -D ~/benchmarks/html/html_8_16K.32K.dict ~/benchmarks/html/html_8_16K/* loading 7450 files... created src buffer of size 83.4 MB split input into 7450 blocks loading dictionary /home/zhuhan/benchmarks/html/html_8_16K.32K.dict compressing at level 1 without dictionary : Ratio=3.03 (28827863 bytes) compressed using a 32768 bytes dictionary : Ratio=4.28 (20410262 bytes) generating 1 dictionaries, using 0.1 MB of memory Compression Speed : 306.9 MB/s Median Speed : 298.4 MB/s ```
|
Added two more commits:
|
embg
approved these changes
Jul 20, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Benchmarking decompression results in a segfault in
createCompressInstructionsbecause
cctxParamsis NULL.Skip running that function if we are not benching
compression.