Remember keyword parameters for tune!

[wouterwln]

When defining a benchmark suite, it might occur that, for in-place operations, we want to limit the amount of evals to 1. However, for other benchmarks in the same suite, we might want to tune these benchmarks, so it makes sense to call tune! on the entire suite. However, this tries to overwrite evals in the benchmarks for which we have defined it by hand. A MWE would be:

using BenchmarkTools

mutable struct foo
    data
end


function bar!(foo)
    foo.data .= 1
    foo.data = map(x -> (), foo.data)
end

suite = BenchmarkGroup()
suite["bar!"] = @benchmarkable bar!(f) setup=(f=foo(ones(10))) evals=1
tune!(suite)

If we call bar! on the same foo instance twice, this will throw an error because we convert the elements of foo.data to tuples, which we cannot set to 1 again. tune! tries to apply the bar! function twice to the same instance of foo, even though evals=1 is explicitly passed to the @benchmarkable. Hence, this suite can never be tune!d, which is kind of a shame. Is there a workaround around this?

[gdalle]

I think this is fixed by #318, can you check if the problem still occurs on master?

[wouterwln]

You're right, and I now also see the duplicate issues ;). It's fixed on master. Thanks!

[gdalle]

I hope I can tag a release in the coming weeks but I'm a little short on bandwidth