performance benchmarking

[dcherian]

bincount-ed sum is 6x slower than sum, and bincount-ed count is 2x slower than count so there are definitely cases where it makes sense to split the dataset early instead of using flox. I'm seeing this on Pangeo Cloud using the GODAS dataset.

array = np.zeros((10 ** 5,), dtype=int)
by = array

%timeit np.bincount(by)  # count
%timeit np.bincount(by, weights=array)  # sum
%timeit np.sum(~np.isnan(by)) # count
%timeit array.sum() # sum

264 µs ± 2.76 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
441 µs ± 1.33 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
117 µs ± 2.68 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
65.9 µs ± 1.17 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

[dcherian]

With engine="flox"

%timeit b, a = flox.core._prepare_for_flox(by, array); flox.aggregate_flox.nanlen(b, a, fill_value=0)
%timeit b, a = flox.core._prepare_for_flox(by, array); flox.aggregate_flox.sum(b, a, fill_value=0)

237 µs ± 33.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
180 µs ± 6.06 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)