Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit Benchmark.NET benchmarks programmatically #4

Open
Aaronontheweb opened this issue Jan 12, 2024 · 0 comments
Open

Emit Benchmark.NET benchmarks programmatically #4

Aaronontheweb opened this issue Jan 12, 2024 · 0 comments

Comments

@Aaronontheweb
Copy link
Owner

Is your feature request related to a problem? Please describe.

Ideally, we should try to leverage Benchmark.NET for the actual execution and running of these benchmarks. The IActorBenchmark type is practically set up for this automatically:

public interface IActorBenchmark
{
    IBenchmarkConfiguration Configuration { get; }
    
    Task<(int totalActors, int messagesPerActor)> Setup(CancellationToken ct);
    
    Task Run(CancellationToken ct);
    
    Task Teardown(CancellationToken ct);
}

Describe the solution you'd like

I would like to do the following:

  1. 1 Benchmark should be computed per IActorBenchmarkDefintion;
  2. 1 BenchmarkCase should be computed per unique IBenchmarkConfiguration; and
  3. The InvocationsPerOperation number for each BenchmarkCase should be set programmatically via the `IActorBenchmark.Setup~ computed value, if that's possible.

I believe we should be able to do this by tapping into the Benchmark.NET APIs directly and passing them into the BenchmarkSwitcher - but it will take a bit of meta-programming magic to make this work.

Essentially, we need to make sure this function returns true for whatever wrapping we do around Benchmark.NET:

https://github.com/dotnet/BenchmarkDotNet/blob/570eeefa20c8692662722cd4bf8a1b663bd361b7/src/BenchmarkDotNet/Running/TypeFilter.cs#L12-L45

Describe alternatives you've considered

It might actually be a lot easier to instrument this with NBench, as it allows for dynamic computation of operation invocations in a much more simplified manner than Benchmark.NET.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant