New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IterationSetup not run in Job.InProcess #513

Closed
pakrym opened this Issue Aug 2, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@pakrym

pakrym commented Aug 2, 2017

Repro:

    [Config(typeof(CoreConfig))]
    public class TimeToFirstServiceBenchmark1
    {
        private object _a;
        
        [IterationSetup(Target = nameof(Transient))]
        public void SetupTransientIteration()
        {
            _a = new object();
        }

        [Benchmark]
        public void Transient()
        {
            _a.ToString();
        }
    }

    public class CoreConfig : ManualConfig
    {
        public CoreConfig()
        {
            Add(JitOptimizationsValidator.FailOnError);
            Add(MemoryDiagnoser.Default);
            Add(StatisticColumn.OperationsPerSecond);
            Add(Job.InProcess);
        }
    }

Expected:

Successful run.

Actual:

Target type: TimeToFirstServiceBenchmark1
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   TimeToFirstServiceBenchmark1.Transient: InProcess(Toolchain=InProcessToolchain)

// Validating benchmarks:
// **************************
// Benchmark: TimeToFirstServiceBenchmark1.Transient: InProcess(Toolchain=InProcessToolchain)
// *** Generate ***
// Result = Success
// BinariesDirectoryPath =

// *** Build ***
// Result = Success

// *** Execute ***
// Launch: 1 / 1

// Benchmark Process Environment Information:
// Runtime=.NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
// GC=Concurrent Workstation
// Job: InProcess(Toolchain=InProcessToolchain)

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Extensions.DependencyInjection.Performance.TimeToFirstServiceBenchmark1.Transient() in D:\github\aspnet\DependencyInjection\test\DI.Performance\TimeToFirstServiceBenchmark - Copy.cs:line 24
   at InvokeMultipleEmitted(BenchmarkActionVoid , Int64 )
   at BenchmarkDotNet.Engines.Engine.Jitting()
   at BenchmarkDotNet.Toolchains.InProcess.InProcessRunner.Runnable.RunCore(IHost host, Benchmark benchmark, BenchmarkActionCodegen codegenMode)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at BenchmarkDotNet.Toolchains.InProcess.InProcessRunner.Run(IHost host, Benchmark benchmark, BenchmarkActionCodegen codegenMode)
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

// Run, Diagnostic
// BeforeAnythingElse

// Benchmark Process Environment Information:
// Runtime=.NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
// GC=Concurrent Workstation
// Job: InProcess(Toolchain=InProcessToolchain)

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Extensions.DependencyInjection.Performance.TimeToFirstServiceBenchmark1.Transient() in D:\github\aspnet\DependencyInjection\test\DI.Performance\TimeToFirstServiceBenchmark - Copy.cs:line 24
   at InvokeMultipleEmitted(BenchmarkActionVoid , Int64 )
   at BenchmarkDotNet.Engines.Engine.Jitting()
   at BenchmarkDotNet.Toolchains.InProcess.InProcessRunner.Runnable.RunCore(IHost host, Benchmark benchmark, BenchmarkActionCodegen codegenMode)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at BenchmarkDotNet.Toolchains.InProcess.InProcessRunner.Run(IHost host, Benchmark benchmark, BenchmarkActionCodegen codegenMode)
// Exception: System.InvalidOperationException: Sequence contains no elements
   at System.Linq.Enumerable.Last[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver, List`1 artifactsToCleanup)

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\TimeToFirstServiceBenchmark1-report.csv
  BenchmarkDotNet.Artifacts\results\TimeToFirstServiceBenchmark1-report-github.md
  BenchmarkDotNet.Artifacts\results\TimeToFirstServiceBenchmark1-report.html

// * Detailed results *
TimeToFirstServiceBenchmark1.Transient: InProcess(Toolchain=InProcessToolchain)
Runtime = ; GC =
There are no any results runs

Total time: 00:00:00 (0.12 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Xeon CPU E5-1650 v3 3.50GHz, ProcessorCount=12
Frequency=3410075 Hz, Resolution=293.2487 ns, Timer=TSC
.NET Core SDK=2.0.0-preview3-006908
  [Host] : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT

Job=InProcess  Toolchain=InProcessToolchain

    Method | Mean | Error | Op/s | Allocated |
---------- |-----:|------:|-----:|----------:|
 Transient |   NA |    NA |   NA |       N/A |

Benchmarks with issues:
  TimeToFirstServiceBenchmark1.Transient: InProcess(Toolchain=InProcessToolchain)

// * Warnings *
IterationSetupCleanup
  TimeToFirstServiceBenchmark1.Transient: InProcess -> MemoryDiagnoser could provide inaccurate results because of the IterationSetup

// * Legends *
  Mean      : Arithmetic mean of all measurements
  Error     : Half of 99.9% confidence interval
  Op/s      : Operation per second
  Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
  1 ns      : 1 Nanosecond (0.000000001 sec)

// * Diagnostic Output - MemoryDiagnoser *


// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *

Global total time: 00:00:01 (1.62 sec)
@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin
Member

AndreyAkinshin commented Aug 2, 2017

@ig-sinicyn

This comment has been minimized.

Show comment
Hide comment
@ig-sinicyn

ig-sinicyn Aug 2, 2017

Collaborator

@AndreyAkinshin
Will look sometime later today. I'm quite busy right now, sorry:)

Collaborator

ig-sinicyn commented Aug 2, 2017

@AndreyAkinshin
Will look sometime later today. I'm quite busy right now, sorry:)

@ig-sinicyn

This comment has been minimized.

Show comment
Hide comment
@ig-sinicyn

ig-sinicyn Aug 2, 2017

Collaborator

Done. BTW, why BenchmarkProgram.txt uses null-conditional operators to call the setup/cleanup methods?as example:

            Runnable instance = new Runnable();
            $ParamsContent$
            ...
            instance?.globalSetupAction();
            instance?.iterationSetupAction();
Collaborator

ig-sinicyn commented Aug 2, 2017

Done. BTW, why BenchmarkProgram.txt uses null-conditional operators to call the setup/cleanup methods?as example:

            Runnable instance = new Runnable();
            $ParamsContent$
            ...
            instance?.globalSetupAction();
            instance?.iterationSetupAction();
@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Aug 3, 2017

Member

@ig-sinicyn I guess setup/cleanup actions were null in case of the absence of the corresponded attributes (in early versions of the setup/cleanup stuff). Probably, we can safely remove ? here (it should be checked first).

Member

AndreyAkinshin commented Aug 3, 2017

@ig-sinicyn I guess setup/cleanup actions were null in case of the absence of the corresponded attributes (in early versions of the setup/cleanup stuff). Probably, we can safely remove ? here (it should be checked first).

@AndreyAkinshin AndreyAkinshin added this to the v0.10.10 milestone Aug 3, 2017

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment