Make part of BenchmarkTools.jl?

[ViralBShah]

In conversation with @shashi we were wondering if this package should just become part of BenchmarkTools.jl?

[ViralBShah]

cc @jrevels

[shashi]

Just checked - this won't add any extra dependencies to BenchmarkTools on Julia 0.5+ 👍

[shashi]

I think this package definitely belongs inside BenchmarkTools in some form or as it is. The functions and macros in this package fall squarely in the category of "benchmark tools", and the current split doesn't feel very satisfying. I can make a PR to BenchmarkTools if @jrevels has no objections.

[jrevels]

I'm fine with merging the functionality here into BenchmarkTools (sorry for the late response).

[jrevels]

I'm fine with merging the functionality here into BenchmarkTools (sorry for the late response).

Note that I'm referring to the benchmarkpkg functionality. I'd prefer not to add the additional macros here to BenchmarkTools. The similarity of @benchmark, @bench, @benchmarkable and @benchgroup could easily be a point of confusion for new users. The original BenchmarkTrackers package had basically the exact same macros, but after spending a lot of time working with that interface, I ended up rewriting the BenchmarkGroup API to move away from macros.

Compare the example in the README to how one would write it in native BenchmarkTools:

module MyBenchmarks

# instantiate global state unbeknownst to naive users
using BenchmarkTools 

# magically generate/push group to global state
@benchgroup "utf8" ["string", "unicode"] begin
    teststr = UTF8String(join(rand(MersenneTwister(1), 'a':'d', 10^4)))
    @bench "replace" replace($teststr, "a", "b")
    @bench "join" join($teststr, $teststr)
end

end # module

...versus...

module MyBenchmarks

using BenchmarkTools 

# just tell users that `benchmarkpkg` will look for and execute MyBenchmarks.SUITE
const SUITE = BenchmarkGroup()

# hook up group explicitly, no magic
let g = addgroup!(SUITE, "utf8", ["string", "unicode"])
    teststr = UTF8String(join(rand(MersenneTwister(1), 'a':'d', 10^4)))
    g["replace"] = @benchmarkable replace($teststr, "a", "b")
    g["join"] = @benchmarkable join($teststr, $teststr)
end

end # module

It's true that the latter is slightly more verbose, but at the huge benefit (IMO) of transparency/explicitness/similarity to working with Dicts.

[juliohm]

No matter where this functionality goes, it is good to have a larger group of people looking into benchmark tools for Julia.

[KristofferC]

@jrevels, FYI, the benchmark macros are now removed, the way to use this pacakge now is by using the official BenchmarkToosl dict interface.

[goerz]

Is the plan to move everything (specifically the benchmarkpgk method) into BenchmarkTools, making this package obsolete towards the release of Julia 0.7?

[KristofferC]

Probably not.

[KristofferC]

Don't think this will happen.