Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
N-way comparison with plots #8
This is the big feature I've been working on. I would link you to the branch, but my modem is broken and I forgot to bring the code/laptop to work. Which also means that I cannot currently show you pretty pictures to help make the case for this feature.
What I can do is give a quick overview of the design decisions and effects of this feature.
Hmmmmmmmm. I'm not as excited about this one, honestly. Not because I think it isn't useful, but because it's a large increase in functionality that I'm not sure I'll be able to maintain. The dependency on
I'm also not keen on an N-way comparison. I feel like a much simpler approach is to keep to a pairwise comparison, but permit the inputs to contain more than one sample of each benchmark. (This is how I envision extracting p-values, for example.)
With that said, plotting stuff is cool, and I know it's something I'm going to want to do eventually. I would be more in favor of adding something simpler to
Another possibility is pushing the benchmark parsing/comparison code into a library, and letting others develop tools on top of that. But that feel likes a monstrous amount of overkill...
Yeah, I don't like the dependency on gnuplot either. But at least there are windows binaries for gnuplot.
The reason I even came up with was because I needed an overview of how different implementation techniques compared for some benchmarks. Although pair-wise comparison is fine to find the best implementation for one benchmark, it's easier to click through some images to get an impression of performance on a benchmark suite.
I now know a little bit more about plotting with
Just generating the gnuplot scripts and leaving the calls to gnuplot to the user is certainly a possibility. But I'm not sure the design challenges will be moot, that has to do with details of gnuplot. I'll write more on that tonight or tomorrow, this is taking too much time/concentration and I should be working ^^'
I don't know exactly how much work it would be to split off a library, but that was one of the things that came to mind as I wrote this issue. It really all depends on how far we want to take this. I think splitting off a library is something we can always do later when it's a more obvious choice though. When you have good enough testability, things will be naturally organised in way that facilitates splitting up.
Ok, so more info on the current implementation. I pushed the working code, so you can try out this branch. Usage:
along with warnings about tests that are only in one module. Here is one of the generated plots:
The generated plot script looks something like this:
set terminal png noenchanced set output 'target/benchcmp/ac_ten_one_prefix_byte_every_match' set title 'ac_ten_one_prefix_byte_every_match' set ylabel 'ns/iter' set boxwidth 0.9 set style data histograms set style fill solid 1.0 set bars fullwidth set style fill solid border -1 set style histogram errorbars gap 2 lw 1 unset xtics set xrange [0.33:0.83] set ytics border mirror norotate set yrange [0:1544127e-1] plot '-' binary endian=little record=1 format='%uint64' using 1:2 title 'dense', '-' binary endian=little record=1 format='%uint64' using 1:2 title 'dense_boxed', '-' binary endian=little record=1 format='%uint64' using 1:2 title 'full', '-' binary endian=little record=1 format='%uint64' using 1:2 title 'full_overlap', '-' binary endian=little record=1 format='%uint64' using 1:2 title 'sparse'
After that comes the data (as raw
I think I can figure something out with a gnuplot script that can be