Minimal benchmark setup

[tkf]

See #737. Please comment on how we organize benchmark/* here.

[coveralls]

Coverage remained the same at 81.975% when pulling c511e52 on tkf/pkgbenchmark into 7a78efd on master.

[tkf]

What do you think about this strategy? I use this for all my packages and it works well for me. This is useful for running each file separately like this: run(include("benchmark/bench_qr.jl")).

[c42f]

This sounds completely fine to me!

---

PS: My fantasy interface might be more akin to @testset where the groups are assembled "magically" using dynamic scope

@benchmark_group "all" begin
    @benchmark_group "QR $K" for K=1:22
        a = rand(SMatrix{K,K,Float64,K*K})
        m = Matrix(a)
        @benchmarkable qr($a)
        @benchmarkable qr($m)
     end
     # other groups  or includes()...
end

but unlike Test, to allow running these selectively. But no doubt that's an issue for BenchmarkTools ;-)

[KristofferC]

PS: My fantasy interface might be more akin to @testset where the groups are assembled "magically" using dynamic scope

Ref JuliaCI/PkgBenchmark.jl#9 (comment).

[c42f]

Thanks for the crossref 👍

I do strongly disagree with Jarret on that one. I think dynamic scoping (not global scope mind you!) can be fairly clear and non-magical for this kind of use as long as it's clearly documented.

[c42f]

I think this is good; simple enough and flexible.

[c42f]

This sounds completely fine to me!

---

PS: My fantasy interface might be more akin to @testset where the groups are assembled "magically" using dynamic scope

@benchmark_group "all" begin
    @benchmark_group "QR $K" for K=1:22
        a = rand(SMatrix{K,K,Float64,K*K})
        m = Matrix(a)
        @benchmarkable qr($a)
        @benchmarkable qr($m)
     end
     # other groups  or includes()...
end

but unlike Test, to allow running these selectively. But no doubt that's an issue for BenchmarkTools ;-)

[KristofferC]

could be const I guess?

[tkf]

OK, initially I didn't want const to avoid seeing the redefinition warning but it's not a good motivation anymore since I use modules.

[KristofferC]

could be const I guess?