add convert benchmarks

[stevengj]

Add convert benchmarks based on JuliaLang/julia#18954

[jrevels]

Looks like Travis is failing to build Blosc due to a cert error, but only when testing v0.6 (happened in #33 as well)? Any idea what's going on there? EDIT: Ah, I see it's described in JuliaIO/Blosc.jl#28

[stevengj]

Probably safe to merge this and #33, since the 0.6 failure is due to JuliaLang/julia#18997.

[jrevels]

Can we make this 100 by 100 instead 2000 by 2000 (thus significantly increasing the number of samples taken within the time limit)?

[stevengj]

ok

[stevengj]

Decreasing the benchmark size shouldn't actually improve the statistics for the mean, it will just increase the variance, right? i.e. if it is the choice between benchmarking an operation 10x on a size-10000 array or 1000x on a size-100 array, in either case you have executed the underlying loop body the same number of times and should expect the same statistics for the mean, no? However, by doing the timing with finer granularity, you will get a larger variance.

[stevengj]

Or does nanosoldier use the median rather than the mean?

[jrevels]

Nanosoldier uses the minimum (a lot more detail on can be found in this paper). Furthermore, once I get time to fully implement hypothesis testing in BenchmarkTools, it will be important that the sample sizes are large enough to characterize an underlying timing distribution.

A less rigorous reason: it seems, from my experience, that benchmarks that touch/use less memory are less prone to the mysterious "erroneous" timing offsets we've seen on Nanosoldier in practice (which I believe are memory-related in nature, though I haven't pinned the exact cause down yet).

[stevengj]

That's good. (FFTW's benchmarks have always used the minimum as well, for much the same reason, but we never formally studied the statistics.)

[stevengj]

See also here on FFTW's benchmarking methodology. Actually, we take the minimum of several averages, so it's not quite the same.

[stevengj]

LMbench also used a minimum to avoid noise. There is probably some other literature as well that you and @jiahao should take a look at for any future version of your work, although probably you guys did the most detailed statistical study.

[vchuravy]

Maybe worth having scalar version of these? I was originally thinking that these would be to fast, but since JuliaCI/BenchmarkTools.jl#23 is merged that shouldn't be a problem anymore.

[stevengj]

A scalar version seems somewhat redundant; it seems like we just need one or the other.