benchmark suite

[HarlanH]

It'd be nice to know things like how much overhead does the NA masking have over a raw vector, and how much memory does a PooledDataVec save, etc. The core Julia team is thinking about this too, and also about storing benchmarks over time as the language matures. JuliaLang/julia#1073

[tshort]

Here are some comparisons between DataVecs and Vectors. DataVecs are a bit faster with nafilter(x). For everything else, Vectors are faster, often by quite a bit.

This is a little unfair in that I tweaked naFilter ahead of time to run better. Also, the code needed for NA support in vectors is here:

https://github.com/tshort/JuliaData/blob/floatNA/src/alternate_NA.jl

N = 10000000
v= randn(N)
datavec = DataVec(v)
idx = randi(N, 10000)
vna = copy(v)
vna[idx] = NA
datavecna = copy(datavec)
datavecna[idx] = NA

# Results without NA's in the data:

julia> @time sum(v)
elapsed time: 0.06444692611694336 seconds
-3206.747631560774

julia> @time sum(datavec)
elapsed time: 2.0190951824188232 seconds
-3206.7476315604167

julia> @time sum(nafilter(v))
elapsed time: 0.265045166015625 seconds
-3206.747631560774

julia> @time sum(nafilter(datavec))
elapsed time: 0.19718289375305176 seconds
-3206.747631560774

julia> @time sum(naFilter(v))
elapsed time: 0.046386003494262695 seconds
-3206.7476315604167

julia> @time sum(naFilter(datavec))
elapsed time: 4.131659030914307 seconds
-3206.7476315604167


# Results with NA's in the data:

julia> @time sum(vna)
elapsed time: 0.05554509162902832 seconds
NaN

julia> @time sum(datavecna)
no method +(Float64,NAtype)
 in method_missing at base.jl:60
 in sum at reduce.jl:63

julia> @time sum(nafilter(vna))
elapsed time: 0.23963308334350586 seconds
-3061.658194729886

julia> @time sum(nafilter(datavecna))
elapsed time: 0.18987488746643066 seconds
-3061.658194729886

julia> @time sum(naFilter(vna))
elapsed time: 0.04549598693847656 seconds
-3061.658194729652

julia> @time sum(naFilter(datavecna))
elapsed time: 4.118844985961914 seconds
-3061.658194729652

[tshort]

With a DataVec-specific method for sum, one should be able to make these as fast as the Vector versions.

[johnmyleswhite]

We should build upon this benchmark suite once we've settled a bit more upon the basic behaviors that DataVec's and DataFrame's should exhibit. One question I have: where will we store results as they accumulate over time? In a CSV that gets updated and held on GitHub? Having this benchmark suite run repeatedly is very important if we want to make sure that new changes actually improve the global performance of DataFrames rather than improve one component at the expense of others.

[johnmyleswhite]

Closed by 319eab6

Please add new benchmarks using the Benchmark package. Please append results to benchmarks/results.csv so that we can track performance over time.