New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Benchmarking a method doesn't run global setup when filter is applied #564

atifaziz opened this Issue Oct 6, 2017 · 1 comment


None yet
2 participants

atifaziz commented Oct 6, 2017

Steps to Reproduce

  1. Clone the repo and checkout 9c194c6
  2. cd into the clone directory
  3. Build the project by running build.bat
  4. Run benchmarks in class IL_Loops by executing: dotnet run -c Release -f netcoreapp2.0 --project samples\BenchmarkDotNet.Samples -- class=IL_Loops
  5. Run a benchmark for the method ForEachArray in class IL_Loops by executing: dotnet run -c Release -f netcoreapp2.0 --project samples\BenchmarkDotNet.Samples -- method=ForEachArray

Expected Result

Expected all requested benchmarks in steps 4 and 5 to run to completion.

Actual Result

Benchmarks in step 4 run fine but the benchmark for the single method ForEachArray in step 5 fails with a NullReferenceException:

Target type: IL_Loops
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   IL_Loops.ForEachArray: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: IL_Loops.ForEachArray: DefaultJob
// *** Generate *** 
// Result = Success
// BinariesDirectoryPath = A:\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\netcoreapp2.0\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa\bin\Release\netcoreapp2.0

// *** Build ***
// dotnet restore took 5.54s
// dotnet build took 11.31s
// Result = Success

// *** Execute ***
// Launch: 1 / 1

// Benchmark Process Environment Information:
// Runtime=.NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
// GC=Concurrent Workstation
// Job: DefaultJob

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at BenchmarkDotNet.Samples.IL.IL_Loops.ForEachArray() in A:\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\IL\IL_Loops.cs:line 33
   at BenchmarkDotNet.Autogenerated.Runnable.MainMultiAction(Int64 invokeCount) in A:\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\netcoreapp2.0\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa.notcs:line 400
   at BenchmarkDotNet.Engines.Engine.Jitting() in A:\BenchmarkDotNet\src\BenchmarkDotNet.Core\Engines\Engine.cs:line 93
   at BenchmarkDotNet.Autogenerated.Runnable.Run(IHost host) in A:\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\netcoreapp2.0\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa.notcs:line 122
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at BenchmarkDotNet.Autogenerated.Program.AfterAssemblyLoadingAttached(String[] args) in A:\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\bin\Release\netcoreapp2.0\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa\ac9c1caf-53d4-4b45-8b62-7bfbd28528fa.notcs:line 56
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

// ***** BenchmarkRunner: Finish  *****

// * Export *

// * Detailed results *
IL_Loops.ForEachArray: DefaultJob
Runtime = ; GC = 
There are no any results runs

Total time: 00:00:17 (17.72 sec)

// * Summary *

BenchmarkDotNet=v0.10.9.20171006-develop, OS=Windows 10 Threshold 2 [1511, November Update] (10.0.10586)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742187 Hz, Resolution=364.6724 ns, Timer=TSC
.NET Core SDK=2.0.0
  [Host] : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT

       Method | Mean | Error |
------------- |-----:|------:|
 ForEachArray |   NA |    NA |

Benchmarks with issues:
  IL_Loops.ForEachArray: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *

Global total time: 00:00:19 (19.79 sec)

This happens because initialValuesArray is uninitialzed in the benchmark:

        public int ForEachArray()
            var counter = 0;
            foreach (var i in initialValuesArray)
                counter += i;
            return counter;

This is in turn due to the global setup in GlobalSetupData of that class never getting called. This can be verified by looking at the generated code for the benchmark.

Additional Information

The tip of master (9076a69) was broken at the time of reporting so its parent 9c194c6 was used instead.

The same problem can be reproduced against the last released version of 0.10.9.

While steps 4 & 5 specify netcoreapp2.0 as the framework, the bug can be observed for any of the targets supported by BenchmarkDotNet.Samples.csproj.

@adamsitnik adamsitnik changed the title from [Bug] Benchmarking a method doesn't run global setup to [Bug] Benchmarking a method doesn't run global setup when filter is applied Oct 15, 2017

@adamsitnik adamsitnik self-assigned this Oct 15, 2017

@adamsitnik adamsitnik added this to the v0.10.10 milestone Oct 15, 2017


This comment has been minimized.

Show comment
Hide comment

adamsitnik Oct 15, 2017


@atifaziz thanks a lot for a great bug report! fixed ;)


adamsitnik commented Oct 15, 2017

@atifaziz thanks a lot for a great bug report! fixed ;)

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment