Feature idea: Check which function is faster

[PallHaraldsson]

I use @btime (and hypertime) to time, looking for the minimum time.

They both take some time, longer than @time which isn't reliable.

I was thinking would it be valuable to run task A, task B, A, B, etc. to see which is faster? Or because of cache, maybe A, A, .. A, B, B, ...B, and maybe again A..., B...?

Often I don't really care about the exact minimum value, I just care which is faster since I'm optimizing, and figure it might need way fewer runs, still statistically significant.

[PallHaraldsson]

One more idea for this. If both timings seem to be statistically the same, then look at the number and amount of allocations. They would usually be the same then too, but if for some reason different, it might be useful to point it out.

Conversely, if the two timings are different, and it's because the GC kicked in, then may not be meaningful. This would though likely not happen for the min. time.

[Moelf]

https://juliaci.github.io/BenchmarkTools.jl/dev/manual/#TrialRatio-and-TrialJudgement

[PallHaraldsson]

Ok, I think you're saying already possible (just complicated). Or at least these are the building blocks to make my proposed macro happen.

For the example here:
https://discourse.julialang.org/t/why-is-dot-plus-faster/81830

I would do:

no_dot_timing = @benchmark no_dot(A, B)
with_dot_timing = @benchmark with_dot(A, B)

then something. I would rather want something like:

@bfaster no_dot(A, B) with_dot(A, B)

The benefit could also be early exit, not possible for the other way. Often very obvious which is faster after just few trials. First implementation could do as above, reusing code.

[gdalle]

Same issue as #239