Support benchmarking startup performance

[cwensley]

I am attempting to determine the startup performance of my library (Portable.Xaml), but it seems there's no option to only run the benchmark a single time.

I've tried using Job.Dry.With(RunStrategy.ColdStart) in my Config, but it still is running the benchmark twice.

Investigating the issue, it appears that this line in the generated benchmark calls the Engine.Jitting() method, which calls the benchmark, before running the benchmark.

Is there a way to get around this that I'm not seeing? If not, I think a config option to omit the Jitting() phase would be extremely useful.

Thanks for any help!

[adamsitnik]

Hi @cwensley

I like your idea, I had the same problem few times.

@AndreyAkinshin maybe we should just ommit the jitting for ColdStart? What do you think? Or rather a new strategy? Once ?

[AndreyAkinshin]

@cwensley, thanks for the report. The bug will be fixed in the next version of BenchmarkDotNet.
@adamsitnik, I will fix it. I almost finished a new RunStrategy (which solves one-time Setup/Cleanup) problem. So, I will also disable jitting for ColdStart and the new strategy. Hopefully, will be done on holidays.

[cwensley]

Hey guys!

Thanks for the quick response, this will be really useful. I look forward to the next release to test it out!

Cheers,
Curtis.

[AndreyAkinshin]

@cwensley, fixed, will be released soon. Also see IntroColdStart.