NUnit integration

[mijay]

I personally very like BenchmarkDotNet. But when you compare it with other micro-benchmarking tools for .NET you can note that it misses one quite useful feature - declarative definitions of benchmarks, like in pUnit.

To address this issue I wrote a small class CompetitionBase which now allows me to define benchmark competitions as follows:

public class FirstCompetition: CompetitionBase
{
    [Benchmark]
    public Action MethodA()
    {
        // smth
    }

    [Benchmark]
    public Action MethodB()
    {
        // smth
    }
}

This benchmark competition will be recognized by ReSharper as one NUnit test and can be executed directly from VisualStudio with one keystroke.

So my question is, whether you are interested to merge such "NUnit integration" feature to your library? If yes, I can create an appropriate pull-request with this code.

One problem that I can foresee for my implementation is that it will add an additional dependency to NUnit + it cannot be ported to MSTest due to its internal bug/limitations.

[AndreyAkinshin]

@mijay, I like you idea, it will be useful feature. I'm waiting for pull-request from you. After it I will thinking about MSTest support.

[AndreyAkinshin]

@mijay, I accepted your pull-request and performed full refactoring of the project. Now you can use NUnit integration. Also you can define new BenchmarkCompetition via attributes (see BenchmarkDotNet.Samples).

[mijay]

@AndreyAkinshin, thank you.

The only issue I want to mention is that in refactored code you are creating BenchmarkCompetitionTask.Action as follows:

Action action = () => methodInfo.Invoke(this, new object[0]);

That means, that on every iteration of a benchmark the super-slow MethodInfo.Invoke is used. And I am afraid that this can decrease the clarity of benchmarks' results — the overhead introduced by reflection-based call will blur the difference between "slow" and "fast" times.

I would suggest to either use Expression.Call or Delegate.CreateDelegate to create fast delegate instead. Code using expressions looks like the following:

Action action = Expression
    .Lambda<Action>(
        Expression.Call(
            Expression.Constant(this),
            methodInfo))
    .Compile();

Unfortunately, I do not know how to implement the approach based on Delegate.CreateDelegate it in your case — when signature of a benchmark function is not fixed. If it would be fixed, I would write:

var action = (Action) Delegate.CreateDelegate(typeof(Action), this, methodInfo);

but in case with unknown signature, the only way to create a delegate is

Type delegateType = methodInfo.ReturnType == typeof (void)
    ? typeof (Action)
    : typeof (Func<>).MakeGenericType(methodInfo.ReturnType);
Delegate action = Delegate.CreateDelegate(delegateType, this, methodInfo);

but such untyped delegates can be called only through DynamicInvoke method, which is even slower than MethodInfo.Invoke.

Just to mention: function call emitted by Delegate.CreateDelegate is 90x faster than MethodInfo.Invoke; function call created from expressions is 9x faster than MethodInfo.Invoke. I didn't try the approach based on reflection emit, but I think it will give the same performance as Delegate.CreateDelegate and won't have its limitations.

[AndreyAkinshin]

@mijay, Thank you for your advices. I'll think about this issue. But I think that the problem is not critical: MethodInfo.Invoke is called only once time per benchmark.