Memory leak and crash with [Setup]

[JamesNK]

I have this file: https://github.com/JamesNK/Newtonsoft.Json/blob/9b9c3da624cfe92da402049fce9344d06f451da6/Src/Newtonsoft.Json.Tests/Benchmarks/LargeJArrayBenchmarks.cs

When the benchmarks are run memory usage slowly increases until the runner crashes with an OutOfMemoryException.

// AfterSetup
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.Program.Main(String[] args)

[AndreyAkinshin]

Hey @JamesNK.

In BenchmarkDotNet, we assume that a benchmark method shouldn't produce any side-effects. In your benchmark class, you have the following method:

[Benchmark]
public void AddPerformance()
{
  _largeJArraySample.Add(1);
}

Be default, our benchmark engine could run your method as many times as it required.
In general case, the Add method doesn't have a steady state with a definite time of execution.
For example, Add can require internal array resizing which takes much more time than usual Add.
BenchmarkDotNet are trying to warmup your method and find a steady state.
But the current implementation doesn't have such state!

Possible solution:

1. Make sure that all your benchmark methods have a steady state. For example, you can measure the Add-Remove pair to avoid side-effects. Also it makes sense to benchmark two cases: with array resizing and without it.
2. Introduce a Job and specify values for run characteristics (like LaunchCount, WarmupCount, and so on). In this case, you will get the reproducable results. Otherwise, BenchmarkDotNet will use the default autotuned strategies which fails to find a steady state.

[JamesNK]

Hmmm, ok. I didn't realize the framework did that, but I thought that the Setup was run before each iteration to keep the state steady. That was the impression I got when reading the documentation.

http://benchmarkdotnet.org/Advanced/Setup.htm

If you have some data that you want to initialize, the [Setup] method is the best place for this. It will be invoked only once before each iteration.

[AndreyAkinshin]

Probably, we have to update the documentation.
It's just impossible to perform a nanobenchmark with a heavy setup before each iteration.

[mattwarren]

It might be because our terminology around 'operations' and 'iterations' is confusing, from How it works:

BenchmarkDotNet follows the following steps to run your benchmarks:

1. BenchmarkRunner generates an isolated project per each benchmark method/job/params and builds it in Release mode.
2. Next, we take each method/job/params combination and try to measure its performance by launching benchmark process several times (LaunchCount).
3. An invocation of the target method is an operation. A bunch of operations is an iteration. If you have a Setup method, it will be invoked before each iteration, but not between operations. We have the following type of iterations:
   * Pilot: The best operation count will be chosen.
   * IdleWarmup, IdleTarget: BenchmarkDotNet overhead will be evaluated.
   * MainWarmup: Warmup of the main method.
   * MainTarget: Main measurements.
   * Result = MainTarget - <AverageOverhead>
4. After all of the measurements, BenchmarkDotNet creates:
   * An instance of the Summary class that contains all information about benchmark runs.
   * A set of files that contains summary in human-readable and machine-readable formats.
   * A set of plots.

So an invocation of the target method is an operation.
A bunch of operations is an iteration.
If you have a Setup method, it will be invoked before each iteration, but not between operations.

[AndreyAkinshin]

@mattwarren, thanks for the clarification. I definitely have to improve our documentation.

[adamsitnik]

We have quite good docs about setup now. +cleanup and global/interation/targeted versions.

I am closing this one.