Generate one executable per runtime settings

[adamsitnik]

Today we generate, build and execute new project per every benchmark.

What I want to do is: take all the benchmarks that have the same runtime settings (GC, Jit, .NET framework) and create a single executable for all of them. Then run executable per benchmark (but just tell it which benchmark to run). (In the future: run them in parallel if user wants to == optional feature)

Why: I want to run thousands of benchmarks as part of the CI and spending few seconds per benchmark for build is very expensive.

Moreover, I want to implement local coreclr toolchain which needs a lot of time to compile (I will be using dotnet restore --packages folder) so the gain will be even bigger.

[AndreyAkinshin]

In the future: run them in parallel if user wants to

Great idea!

[adamsitnik]

@AndreyAkinshin done!

Pseudo code:

1. Take all benchmarks, group them by runtime settings (jit/gc/runtime etc) into partitions
2. Build single .exe for each partition
3. Each bechmarks gets a unique benchmark Id per partition
4. When running bechmark tell the process which type to load by providing the id.

Time to build all (more than 650) benchmarks from our Samples project: 13s.

[AndreyAkinshin]

Awesome! I guess it's the most impressive perf improvement in BenchmarkDotNet ever!

However, I should ask: do you a have a setting in Config which disables this behavior? (It can be useful for debug).

[adamsitnik]

Currently what I do is:

1. [KeepBenchmarkFiles]
2. Run single benchmark
3. Find the .exe file and run it with -0 argument

@AndreyAkinshin is that enough? It's just a matter of passing the right Id

[adamsitnik]

@AndreyAkinshin I will be now working on #700 so there will be a lot of build artifacts related debugging.
If my current approach will not be enough I will improve it.

All builds are green, it works so now I am closing this one.