feat(cli): added decompression benchmark #3773
Conversation
jkowalski
force-pushed
the
decompression-benchmark
branch
from
e2dae06
to
193f09e
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3773 +/- ##
==========================================
+ Coverage 75.86% 76.93% +1.07%
==========================================
Files 470 476 +6
Lines 37301 29017 -8284
==========================================
- Hits 28299 22325 -5974
+ Misses 7071 4749 -2322
- Partials 1931 1943 +12 ☔ View full report in Codecov by Sentry. |
| }() | ||
| } | ||
|
|
||
| // run one on the main goroutine and N-1 in parallel. | ||
| v := run() | ||
| v := run(args[len(args)-1]) |
There was a problem hiding this comment.
What's the rationale for running one in the "caller" go routine? It seems the code could be simplified slightly if this case was removed.
It is surprising that only the result of one of the executions is returned.
There was a problem hiding this comment.
premature optimization :)
| @@ -155,11 +212,86 @@ func (c *commandBenchmarkCompression) run(ctx context.Context) error { | |||
| return compressedSize | |||
| } | |||
|
|
|||
| outputBuffers := makeOutputBuffers(c.parallel, defaultCompressedDataByMethod) | |||
There was a problem hiding this comment.
What prevents GC from running while benchmarking and affecting the results? Is GC turned off? would the last method to be benchmarked be likely to experience more memory pressure?
There was a problem hiding this comment.
we actually call runtime.GC() before each algorithm.
cli/command_benchmark_compression.go
Outdated
| func (c *commandBenchmarkCompression) runCompression(ctx context.Context, data []byte, repeatCount int, algorithms map[compression.Name]compression.Compressor) error { | ||
| var results []compressionBechmarkResult | ||
|
|
||
| log(ctx).Infof("Compressing input file %q (%v) using all compression methods.", c.dataFile, units.BytesString(int64(len(data)))) |
There was a problem hiding this comment.
is it still the case that all compression methods are used here?
There was a problem hiding this comment.
replaced "all" with a number.
| } | ||
|
|
||
| for _, a := range strings.Split(c.algorithms, ",") { | ||
| if strings.HasPrefix(string(name), a) { |
There was a problem hiding this comment.
for my understanding: why HasPrefix?
There was a problem hiding this comment.
so you can test all "zstd" in one shot
| } | ||
|
|
||
| if benchmarkDecompression { | ||
| if err := c.runDecompression(ctx, data, repeatCount, algorithms); err != nil { |
There was a problem hiding this comment.
How does this work when multiple algorithms are specified and only decompression is benchmarked? essentially, what's the input for multiple decompressors when there is no prior compression.
There was a problem hiding this comment.
we compress once per compressor, then decompress N times.
There was a problem hiding this comment.
I see. I missed that. Thanks.
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
jkowalski
force-pushed
the
decompression-benchmark
branch
from
d4d6a3f
to
a3281ad
Compare
No description provided.