add @benchmarkset & @case for defining benchmark suite

[Roger-luo]

This PR adds 2 macros to provide a simpler syntax for defining large benchmark suite.

I found it can be a bit annoying if there are multiple benchmark suite like this:

https://github.com/QuantumBFS/YaoArrayRegister.jl/blob/master/benchmark/benchmarks.jl

it is not necessary to manually control things like SUITE["matrices"]["contiguous"], thus I implement these two macro to make defining benchmark suite in a similar syntax as the testset in Test.

now you can just write things like

@suite "suite 1" begin
     @case k for k in 1:n
            rand($k,  $k) * rand($k, $k)
      end
end

instead of

SUITE = BenchmarkGroup()
SUITE["suite 1"] = BenchmarkGroup()

for k in 1:n
    SUITE["suite 1"][k] = @benchmarkable rand($k, $k) * rand($k, $k)
end

PS. currently it will generate a constant SUITE, which is compatible with PkgBenchmark

Update: I find I need some more modification on this new syntax, the current one still fails to deal with some cases.

[Roger-luo]

I changed the name to @benchmarkset and @case, this will provide a similar way to define benchmark suite to @testset and @test

[vchuravy]

Can you rebase onto master to get CI?

[codecov-io]

Codecov Report

Merging #147 (9fb6e11) into master (5df4174) will decrease coverage by 0.23%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##           master     #147      +/-   ##
==========================================
- Coverage   86.37%   86.14%   -0.24%     
==========================================
  Files           6        6              
  Lines         624      635      +11     
==========================================
+ Hits          539      547       +8     
- Misses         85       88       +3     

Impacted Files	Coverage Δ
src/BenchmarkTools.jl	100.00% <ø> (ø)
src/groups.jl	94.67% <81.81%> (-1.53%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5df4174...9fb6e11. Read the comment docs.

[Roger-luo]

bump just fixed the tests.

[Roger-luo]

hi @vchuravy mind releasing a patch version for this?

[KristofferC]

Just pointing out, this package was intentionally designed to not use macros since they were considered confusing and unclear:

JuliaCI/PkgBenchmark.jl#9 (comment)

[Roger-luo]

Just pointing out, this package was intentionally designed to not use macros since they were considered confusing and unclear:

@KristofferC Ah that makes sense, I didn't know one can use this package with addgroup! , it seems not being mentioned in manual at all, only in https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/reference.md#misc-functions

I feel need to polish the documentation a bit for this if we want to revert this PR. I'm OK with both, not sure what other people thinks.

[KristofferC]

Well, too late now.

[awadell1]

Is the equivalent using addgroup! something like:

# Root Suite
SUITE = BenchmarkGroup()

# BenchmarkGroup generated by @benchmarkset
s = BenchmarkGroup()

# For loop for each @case 
for k in 1:n
    s[k] = @benchmarkable rand($k, $k) * rand($k, $k)
end

addgroup!(SUITE, s)