New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warn the users when running Benchmarks from xUnit with shadow copy enabled #558

Closed
pardahlman opened this Issue Sep 27, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@pardahlman

pardahlman commented Sep 27, 2017

Thanks for great work on BenchmarkDotNet 馃帀 .

I'm probably doing something obviously wrong, but I can't figure out what.

I'm trying to run a very simple benchmark on .NETFramework 4.6.1. In fact, at this point it doesn't benchmark anything

    [Benchmark]
    public void Do_Nothing()
    {
      
    }

The code is executed from within an xunit test through the BenchmarkRunner without any custom configuration

BenchmarkRunner.Run<OperationTrackerBenchmarks>();

When running the benchmark I get the following output

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
  Discovering: RaySearch.Core.Benchmarks
  Discovered:  RaySearch.Core.Benchmarks
  Starting:    RaySearch.Core.Benchmarks
Calling global setup
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   OperationTrackerBenchmarks.Do_Nothing: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: OperationTrackerBenchmarks.Do_Nothing: DefaultJob
// *** Generate ***
// Result = Success
// BinariesDirectoryPath = C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301

// *** Build ***
BuildScript: C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301\fdf9d873-38a8-41e8-aee0-18db9c17dcce.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\pardah\AppData\Local\Temp\5ffab0a6-52bc-47e5-a280-b18ca7220111\5ffab0a6-52bc-47e5-a280-b18ca7220111\assembly\dl3\b912a5ae\67e43fd3_ae37d301\fdf9d873-38a8-41e8-aee0-18db9c17dcce.exe
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!


// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report.csv
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report-github.md
  BenchmarkDotNet.Artifacts\results\OperationTrackerBenchmarks-report.html

// * Detailed results *
OperationTrackerBenchmarks.Do_Nothing: DefaultJob
Runtime = ; GC =
There are no any results runs

Total time: 00:00:11 (11.11 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Core i7-7820HQ CPU 2.90GHz (Kaby Lake), ProcessorCount=8
Frequency=2835937 Hz, Resolution=352.6171 ns, Timer=TSC
  [Host] : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2110.0


     Method | Mean | Error |
----------- |-----:|------:|
 Do_Nothing |   NA |    NA |

Benchmarks with issues:
  OperationTrackerBenchmarks.Do_Nothing: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
  Finished:    RaySearch.Core.Benchmarks
=== TEST EXECUTION SUMMARY ===
   RaySearch.Core.Benchmarks  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 13,927s

The way I interpret this is that I can't execute the benchmark (Benchmarks with issues: OperationTrackerBenchmarks.Do_Nothing: DefaultJob). I've tried to troubleshoot of find a way to get a more specific error message, without luck. So my question is: how can I troubleshoot this further?

@pardahlman

This comment has been minimized.

Show comment
Hide comment
@pardahlman

pardahlman Sep 27, 2017

Hello again,

I've tried a few things to isolate this further. First of I switched computer 馃槃 . Further more I created a new solution and referenced BenchmarkDotNet and xUnit (and the console runner for the xUnit.) I have packaged this into a zip file that can be built and run. See attached.

To reproduce this issue:

  1. Extract zip file, restore and build solution
  2. In the solution folder run the xUnit runner
> .\packages\xunit.runner.console.2.2.0\tools\xunit.console.exe .\BasicBenchmark\bin\Release\BasicBenchmark.dll

This is my exact output from this test project

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
  Discovering: BasicBenchmark
  Discovered:  BasicBenchmark
  Starting:    BasicBenchmark
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   VoidBenchmark.Do_Nothing_At_All: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: VoidBenchmark.Do_Nothing_At_All: DefaultJob
// *** Generate ***
// Result = Success
// BinariesDirectoryPath = C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301

// *** Build ***
BuildScript: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.exe
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!


// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.csv
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report-github.md
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.html

// * Detailed results *
VoidBenchmark.Do_Nothing_At_All: DefaultJob
Runtime = ; GC =
There are no any results runs

Total time: 00:00:07 (7.19 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Xeon CPU E3-1545M v5 2.90GHz, ProcessorCount=8
Frequency=2835937 Hz, Resolution=352.6171 ns, Timer=TSC
  [Host] : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2110.0


            Method | Mean | Error |
------------------ |-----:|------:|
 Do_Nothing_At_All |   NA |    NA |

Benchmarks with issues:
  VoidBenchmark.Do_Nothing_At_All: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
  Finished:    BasicBenchmark
=== TEST EXECUTION SUMMARY ===
   BasicBenchmark  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 8.969s

BasicBenchmark.zip

pardahlman commented Sep 27, 2017

Hello again,

I've tried a few things to isolate this further. First of I switched computer 馃槃 . Further more I created a new solution and referenced BenchmarkDotNet and xUnit (and the console runner for the xUnit.) I have packaged this into a zip file that can be built and run. See attached.

To reproduce this issue:

  1. Extract zip file, restore and build solution
  2. In the solution folder run the xUnit runner
> .\packages\xunit.runner.console.2.2.0\tools\xunit.console.exe .\BasicBenchmark\bin\Release\BasicBenchmark.dll

This is my exact output from this test project

xUnit.net Console Runner (64-bit .NET 4.0.30319.42000)
  Discovering: BasicBenchmark
  Discovered:  BasicBenchmark
  Starting:    BasicBenchmark
// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   VoidBenchmark.Do_Nothing_At_All: DefaultJob

// Validating benchmarks:
// **************************
// Benchmark: VoidBenchmark.Do_Nothing_At_All: DefaultJob
// *** Generate ***
// Result = Success
// BinariesDirectoryPath = C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301

// *** Build ***
BuildScript: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\par.dahlman\AppData\Local\Temp\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\5e157c4c-8903-4e6a-b7f3-c2fd5d874378\assembly\dl3\27353197\3f806127_c837d301\a18dd1ce-ceec-460f-8b57-e9852b859533.exe
ExitCode != 0
No more Benchmark runs will be launched as NO measurements were obtained from the previous run!


// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.csv
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report-github.md
  BenchmarkDotNet.Artifacts\results\VoidBenchmark-report.html

// * Detailed results *
VoidBenchmark.Do_Nothing_At_All: DefaultJob
Runtime = ; GC =
There are no any results runs

Total time: 00:00:07 (7.19 sec)

// * Summary *

BenchmarkDotNet=v0.10.9, OS=Windows 10 Redstone 2 (10.0.15063)
Processor=Intel Xeon CPU E3-1545M v5 2.90GHz, ProcessorCount=8
Frequency=2835937 Hz, Resolution=352.6171 ns, Timer=TSC
  [Host] : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2110.0


            Method | Mean | Error |
------------------ |-----:|------:|
 Do_Nothing_At_All |   NA |    NA |

Benchmarks with issues:
  VoidBenchmark.Do_Nothing_At_All: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
  Finished:    BasicBenchmark
=== TEST EXECUTION SUMMARY ===
   BasicBenchmark  Total: 1, Errors: 0, Failed: 0, Skipped: 0, Time: 8.969s

BasicBenchmark.zip

@pardahlman

This comment has been minimized.

Show comment
Hide comment
@pardahlman

pardahlman Sep 27, 2017

Hang on, looks like this works if I run it as a from a console app 馃 . Any ideas why that is?

pardahlman commented Sep 27, 2017

Hang on, looks like this works if I run it as a from a console app 馃 . Any ideas why that is?

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 27, 2017

Member

Hello @pardahlman

You need to disable shadow copy. We are currently not supporting it and not planning to change it (due to many consequences, not because we don't like it)

Here you can find the right setting for xunit

Member

adamsitnik commented Sep 27, 2017

Hello @pardahlman

You need to disable shadow copy. We are currently not supporting it and not planning to change it (due to many consequences, not because we don't like it)

Here you can find the right setting for xunit

@adamsitnik adamsitnik self-assigned this Sep 27, 2017

@pardahlman

This comment has been minimized.

Show comment
Hide comment
@pardahlman

pardahlman Sep 28, 2017

Hello @adamsitnik - thanks for pointing me in the right direction - I wouldn't have figured this out myself. Everything works as expected now.

Do you think there is any way to detect if shadow copy is performed and warn the user? I'd be glad to help out if you think this is possible or desirable. If not, you can close this issue and keep on providing first class open source! 馃憣

pardahlman commented Sep 28, 2017

Hello @adamsitnik - thanks for pointing me in the right direction - I wouldn't have figured this out myself. Everything works as expected now.

Do you think there is any way to detect if shadow copy is performed and warn the user? I'd be glad to help out if you think this is possible or desirable. If not, you can close this issue and keep on providing first class open source! 馃憣

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 30, 2017

Member

@pardahlman you are right, we should have provided nice warning. I hope that you did not waste too much time on this.

The users are now going to get following warning:

Assembly Test, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.
Member

adamsitnik commented Sep 30, 2017

@pardahlman you are right, we should have provided nice warning. I hope that you did not waste too much time on this.

The users are now going to get following warning:

Assembly Test, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.

@adamsitnik adamsitnik added this to the v0.10.10 milestone Sep 30, 2017

@adamsitnik adamsitnik changed the title from Question: troubleshooting benchmarks with issues to Warn the users when running Benchmarks from xUnit with shadow copy enabled Sep 30, 2017

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment