Consistent color for frameworks in the benchmark plots

[KristofferC]

In https://chriselrod.github.io/LoopVectorization.jl/latest/examples/dot_product/, OpenBLAS is green but in https://chriselrod.github.io/LoopVectorization.jl/latest/examples/matrix_vector_ops/, LoopVectorization is green. Creates a bit of confusion when you flip between them.

[chriselrod]

VegaLite sorts all the categories alphabetically, and then uses the color scale with respect to that alphabetical ordering.
As a workaround, I renamed them "01. LoopVectorization", "02. Julia", etc, so that they're assigned consistent colors. Docs should rebuild soon.

Think that's a reasonable solution, or would it be better to try and find out how to lose the numbers?

[KristofferC]

Think that's a reasonable solution, or would it be better to try and find out how to lose the numbers?

Sounds reasonable to me. I guess assigning a color to each framework and giving that explicitly to the plots is also possible but might be more work. As long as the colors are consistent Im happy at least heh :)

[KristofferC]

Speaking of colors though, the orange ones are pretty hard to tell apart

[chriselrod]

I agree. I've had trouble, too.

Here is the function I used for plotting benchmarks. I was using category20 to define the colors.

Maybe one of those is better than category20. I just tried set3, but the second yellow and last yellow are very hard for me to tell apart. paired's paired orange colors look the same as category20.
I need at least 12 categories (otherwise, it'll start recycling). But it looks like it's hard to get more than 10 easy-to-tell-apart colors, since all of them longer than that seem to have collisions.

I think at this point I should switch plotting libraries. I believe Gadfly should be able to work with this:

julia> using Colors

julia> distinguishable_colors(12, [RGB(1,1,1), RGB(0,0,0)], dropseed=true)
12-element Array{RGB{N0f8},1} with eltype RGB{FixedPointNumbers.Normed{UInt8,8}}:
 RGB{N0f8}(0.78,0.129,0.867)
 RGB{N0f8}(0.82,0.29,0.0)
 RGB{N0f8}(0.0,0.549,0.0)
 RGB{N0f8}(0.0,0.498,0.694)
 RGB{N0f8}(0.82,0.675,0.0)
 RGB{N0f8}(0.529,0.0,0.212)
 RGB{N0f8}(1.0,0.561,0.631)
 RGB{N0f8}(0.0,0.0,0.545)
 RGB{N0f8}(0.18,1.0,0.443)
 RGB{N0f8}(0.404,0.322,0.0)
 RGB{N0f8}(0.0,0.816,0.851)
 RGB{N0f8}(0.0,0.314,0.263)

Or perhaps:

julia> distinguishable_colors(13, [RGB(0,0,0), RGB(1,1,1)])[2:end]
12-element Array{RGB{N0f8},1} with eltype RGB{FixedPointNumbers.Normed{UInt8,8}}:
 RGB{N0f8}(1.0,1.0,1.0)
 RGB{N0f8}(0.78,0.129,0.867)
 RGB{N0f8}(0.82,0.29,0.0)
 RGB{N0f8}(0.0,0.549,0.0)
 RGB{N0f8}(0.0,0.498,0.694)
 RGB{N0f8}(0.82,0.675,0.0)
 RGB{N0f8}(0.529,0.0,0.212)
 RGB{N0f8}(1.0,0.561,0.631)
 RGB{N0f8}(0.0,0.0,0.545)
 RGB{N0f8}(0.18,1.0,0.443)
 RGB{N0f8}(0.404,0.322,0.0)
 RGB{N0f8}(0.0,0.816,0.851)

I'll try this and see how it looks.

[chriselrod]

Thoughts?

I think Julia and ifort may look a little similar.

[KristofferC]

Yeah, maybe I would even say the earlier was a bit better? It is a hard problem though since there are quite many plots in each figure and each plot is quite "noisy".

One possibility is to use something like PlotlyJS where you can enable disable plots interactively, would make it easier to compare e.g. LoopVectorization vs vanilla Julia by turning off the other plots.

[chriselrod]

One possibility is to use something like PlotlyJS where you can enable disable plots interactively, would make it easier to compare e.g. LoopVectorization vs vanilla Julia by turning off the other plots.

Can I place these plots in the assets folder, and then have them appear in the documentation in a manner users can interact with them?
If so, that would be amazing.

[KristofferC]

I think saving them as JSON (http://spencerlyon.com/PlotlyJS.jl/manipulating_plots/#saving-figures), putting them in the asset folders and then bundle the plotly js library should work....? I am many steps removed from Web development :p

[chriselrod]

I am many steps removed from Web development :p

Me too. For now, I'd prefer a simpler solution. Or at least something that lets the most interesting contenders stand out.

Currently, my plots (locally, haven't pushed yet) look like this:
A * b

a' * b

A' * B'

(Also, a definite regression compared to what's on the docs now. Even though the assembly looks better: with the matmul, it xor-ed 25 registers to zero them out, did a check to see if the loop will execute at least once, and then if it will, it xor-ed those 25 registers a second time despite not having touched them. It doesn't do that anymore, but the code is randomly slower anyway. I guess in relative terms, xoring registers an extra O(M*N) times isn't really that important. =/)

The idea here was to use a perceptually accurate color map, and then add LoopVectorization as black, and pick a second color that seemed to stand out from the rest for base Julia (I chose red). The scheme I chose was ColorSchemes.cyclic_mygbm_30_95_c78_n256_s25; the remaining 12 colors I used in my plots were evenly spaced through that "perceptually accurate" color map, which I think should mean they're relatively easy to distinguish..
But I have a hard time telling all those greens and yellows and yellow-oranges apart. Seems like you could pick 14 colors that stand out a lot more from each other than these. But this is a lot of lines, so there's a limit before they blur together.

This was with an official 1.4.1 binary, but I'll probably use 1.5 next time to address OpenBLAS's bad performance.

[tbeason]

The problem is really not the colors. It is the number of lines. My suggestion: plot everything as a ratio to LoopVectorization. The other lines are just there as a performance comparison for this package, we don't really care about them relative to each other or in isolation. Then if we see all the lines below 1, LoopVectorization is good. You could put the actual GFLOPS of LoopVectorization on the right y axis if you like.

[chriselrod]

Hmm. That's a good idea.

I think the current plots are a lot interesting to me personally, because I'm more interested in how their performance changes in absolute terms (e.g., spikes/troughs, improvement or degradation as a function of size). But that would make relative performance more clear.
I think I'd include both sets of plots.