RFC: Benchmark tweaks

[pygy]

• Added pi_sum_vec to Matlab, Julia, Python and R (disabled)
• Tweaked parseint in perf.R
• Added perf.lua by Francesco Abatte (Needs the GSL Shell)

[ViralBShah]

These seem quite reasonable. I don't think that most R users write cmpfun in their code, but we can leave it in here for the benchmark.

@StefanKarpinski We will need to refresh the benchmarks post 0.1, and we may even want to include our graph. Anyways, we should move towards running these benchmarks on julia.mit.edu going forward.

[StefanKarpinski]

Agreed. We should use Gadfly to generate a pretty graph too!

[pygy]

The behaviour is now correct AFAICT, but I've yet to run the code (Can't get GSL Shell to compile on this machine). I've also tweaked parseint to mirror the julia version exactly.

Note that the pull request by @franko (#2286) has the same error.

[JeffBezanson]

Yes, the two pull requests is a bit confusing. Should we just use this one?

[pygy]

This stems from an email conversation between Viral, Francesco and me. Viral requested a pull request, and we both did it, independently.

I'd keep this one, since there are other changes.

[ViralBShah]

This one also has the vectorized versions.

[pygy]

It should be good now. @franko, could you implement pi_sum_vec in lua (already in perf.jl, .m, .py and .R)?

[franko]

@pygy gsl shell does not support coding in vectorized form so I cannot implement pi_sum_vec without implementing some amount of boilerplate code.

This was a design choice (good or bad) for gsl shell: it doesn't aim to be like matlab.

Talking about the benchmark I believe it should mandate which problem should be solved and not how it should be solved.

[StefanKarpinski]

If the problem is to compute fib(20) the best code is 6765 in every language. No one actually computes Fibonacci numbers using double recursion – it's obviously a terrible algorithm. The point of that benchmark is to see how good each language is at recursion, not to see how fast you can compute a known constant. The point of the pi summation benchmark is also not to compute pi – I'm pretty sure that's known to a fair number of digits – the point is to see how fast each language is at executing tight numerical loops. Why is this so difficult to understand?

[pygy]

I guess @franko didn't read the home page...

[franko]

@StefanKarpinski I'm not going to argue about that. From the practical point of view gsl shell lacks support for operations in vector form so it is probably better to omit it.

[dcjones]

using Compose, Gadfly, DataFrames

benchmarks = DataFrame(readcsv("benchmarks.csv"),
                       ["Language", "Benchmark", "Time"])

p = plot(benchmarks,
         {:y => "Benchmark", :x => "Time", :color => "Language"},
         Geom.point, Scale.x_log10, Guide.XLabel("Time (Log10 Seconds)"))

draw(SVG("benchmarks.svg", 800px, 400px), p)

It doesn't work great as point plot, there are too many colors. I should do this as a bar chart, but I still need to implement colors in the bar geometry.

[pao]

This might be easier to comprehend as a "relative to C" metric to get rid of the baseline shift for the various algorithms.

[franko]

@dcjones Really nice plots! I really have to learn Julia and its plotting system :-)

Otherwise I think that these data are very difficult to plot in an effective way. One of the best way is probably to plot side by side one language with only another one.

[StefanKarpinski]

Very pretty! Unfortunately, I'm having a bit of trouble running this against master (or 0.1). Now that 0.1 is out hopefully people can stabilize packages against that.

[pygy]

@franko, at last, I got it to compile. I get two errors, though, because math.min and sting.format choke on cdata (it also does in LuaJIT, but I thought that you had tweaked it in the GSL Shell... apparently not).

Wrapping them in tonumber() does the trick.

I get better results in parse_int and quicksort using a pure LuaJIT implementation (replacing cdata numbers with Lua numbers and the iter.ilist(...) with a double[?] array).

gsl_shell,parse_int,0.651
gsl_shell,parse_int2,0.456
gsl_shell,quicksort,0.991
gsl_shell,quicksort2,0.682

Do you mind if I use them instead of yours ?

[dcjones]

@StefanKarpinski I just updated, I think I'm caught up to 0.1 now. Not to drag this off-topic, but would this be a good time to start tagging packages?

[StefanKarpinski]

I was just thinking about that and I'm not sure. Lemme consider it a bit more.

[dcjones]

@pao It's not much better.

using Compose, Gadfly, DataFrames

benchmarks = DataFrame(readcsv("benchmarks.csv"),
                       ["Language", "Benchmark", "Time"])

benchmarks = merge(benchmarks, subset(benchmarks, :(Language .== "c")),
                   "Benchmark", "outer")
within!(benchmarks, :(Time ./= Time_1))

p = plot(benchmarks,
         {:y => "Benchmark", :x => "Time", :color => "Language"},
         Geom.point, Scale.x_log2, Guide.XLabel("Time (Log2 Relative to C)"))

draw(SVG("benchmarks.svg", 800px, 400px), p)

@franko Yeah, I think I'll do a bar chart with a bar for each language grouped by benchmark, or something of the sort. A good opportunity to force me to actually implement that. :)

[pao]

I think it's a bit clearer, anyways, even if it's hard to pick out a particular performer. Picking good colors is an old problem, but see e.g. http://colorbrewer2.org/. Or perhaps port @timholy's MATLAB code up on File Exchange http://www.mathworks.us/matlabcentral/fileexchange/29702

[dcjones]

That's interesting. I have a lot of color spaces implemented, but haven't spent a lot of time on actually selecting colors. I'm just choosing equidistant hues in LAB space. Maybe this would be clear enough if I took "C" out (since it's always 1) and experimented with color scales.

[timholy]

@pao, you don't miss anything!

@dcjones, your algorithm sounds quite similar to the one I developed. The main difference possibly being that you can ask it to avoid the background (or multiple other "reserved" colors).

[dcjones]

You're also not fixing chroma and lightness like I am. It makes sense to vary only hue (or only lightness) to show quantitative data, but qualitative scales like this might benefit by choosing from a wider range of colors.

I might try adding colorbrewer's scales, but I tend to prefer algorithms to anything "hand curated". If anyone knowns of any papers on maximizing distinguishability with constraints for colorblindness (and maybe printability), I'd be interested.

[pao]

I didn't do a deep search to see what ColorBrewer is doing behind the scenes--I can't imagine those were all created by hand, but maybe they were? Penn State is mentioned in the footer, so there might be some publications.

Also, should we move this particular discussion to the Gadfly tracker?

[catawbasam]

Stephen D. Gardner, 2005, Evaluation of the ColorBrewer Color Schemes for Accommodation of Map Readers with Impaired Color Vision (6.1MB PDF)

Available from http://www.personal.psu.edu/cab38/

[ViralBShah]

Is this ok to merge now?

[pygy]

Not yet, I have a few tweaks to make.

[pygy]

It's ready.

[StefanKarpinski]

My machine is no longer a reasonable system to use for the official Julia benchmarks – it makes sounds like a dying animal. We should start using julia.mit.edu. Among other things, that means we're going to need to get a license to run Matlab on there.

[ViralBShah]

MIT has a site license for matlab, I believe.

[diegozea]

Would be great update the benchmark http://julialang.org/ before April 25 ( Ubuntu release ) and use Julia 0.1.2 on it.

P.D.: Comparison 0.1.2 ( julia.0.1 ) with actual master 0.2 ( julia-m )
printfd looks a little slow now ( 1.27 times slower )

dzea@deepthought:~/DNA2Seq_dev$ julia.0.1 ~/bin/julia/test/perf.jl 
julia,fib,0.05698204040527344
julia,parse_int,0.1780986785888672
julia,mandel,0.1628398895263672
julia,quicksort,0.3437995910644531
julia,pi_sum,33.48278999328613
julia,rand_mat_stat,9.737014770507812
julia,rand_mat_mul,25.419950485229492
julia,printfd,19.804954528808594

dzea@deepthought:~/DNA2Seq_dev$ julia-m ~/bin/julia-master/test/perf.jl 
julia,fib,0.054846000000000006
julia,parse_int,0.178239
julia,mandel,0.159812
julia,quicksort,0.343546
julia,pi_sum,33.498786
julia,rand_mat_stat,11.099239
julia,rand_mat_mul,25.27908
julia,printfd,25.161358000000003

[ViralBShah]

We should probably do it right away. Could you file as an issue on the julialang.github.com repo?

[diegozea]

@ViralBShah there is already an issue on that repo: JuliaLang/www_old.julialang.org#23

[ViralBShah]

Thanks.