UnrollFactor setting not working

[colgreen]

Example using 0.12.1 (runtime is .NET 5)

[InvocationCount(3, 1)]
public class Test
{
    public static void Main() => BenchmarkRunner.Run<Test>();

    [IterationSetup]
    public void MySetup() => Console.WriteLine("MySetup");

    [Benchmark]
    public void MyBenchmark()
    {
        Console.WriteLine("MyBenchmark");
        Thread.Sleep(100);
    }
}

(noting that the second argument to InvocationCount is unrollFactor, and actually defaults to 1 - but shown here set explicitly to 1).

Output...

MySetup
MyBenchmark
MyBenchmark
MyBenchmark
WorkloadWarmup   1: 3 op, 338580900.00 ns, 112.8603 ms/op
MySetup
MyBenchmark
MyBenchmark
MyBenchmark
WorkloadWarmup   2: 3 op, 327990300.00 ns, 109.3301 ms/op
MySetup
MyBenchmark
MyBenchmark
MyBenchmark
WorkloadWarmup   3: 3 op, 327250400.00 ns, 109.0835 ms/op
...

Also see #730

This affects some of the benchmarks in the dotnetcore repo, hence those affected benchmarks will currently be reporting potentially very misleading results. E.g. The Array.Sort() benchmarks are mostly running against already sorted arrays. See:

https://github.com/dotnet/performance/blob/master/src/benchmarks/micro/libraries/System.Collections/Sort.cs

[adamsitnik]

[InvocationCount(3, 1)]

WorkloadWarmup 1: 3 op,

You have set the invocation count to 3 and unroll factor to 1, hence the code gets executed 3 times per iteration. This works as expected. I am not sure if I understand where the problem is?

[colgreen]

Hi. Perhaps I have misunderstood then. The scenario is performance testing of Array.Sort, where we pre-allocate a large array at global startup time, fill it with random (unsorted) values as part of [IterationSetup]. Thus, we want to re-run [IterationSetup] once before each [Benchmark], otherwise the benchmark is sorting already sorted data.

My reading was that the Unroll=n would cause the Benchmark to run n times for each call to IterationSetup], hence I set Unroll=1, as discussed in #730.

So maybe I just need to use [InvocationCount(1, 1)] ?

[adamsitnik]

My reading was that the Unroll=n would cause the Benchmark to run n times for each call to IterationSetup

I see. What you are looking for is InvocationCount, which sets the number of invocations per iteration. Unroll factor controls the manual loop unrolling which you most probably should not care about when using IterationSetup. This is why it's set to 1 by default.

You might find the following doc useful: https://benchmarkdotnet.org/articles/guides/how-it-works.html