New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Visual Basic project files (.vbroj) targeting .NET Core #626

Closed
eschneider999 opened this Issue Jan 16, 2018 · 15 comments

Comments

Projects
None yet
3 participants
@eschneider999

eschneider999 commented Jan 16, 2018

Trying to use benchmark dot net and get the following issue?

// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   BenchmarkTests.SimpleQuery: DefaultJob
//   BenchmarkTests.LoadItems: DefaultJob
//   BenchmarkTests.LoadItemsReadTable: DefaultJob
//   BenchmarkTests.LoadItemsWriteTable: DefaultJob
//   BenchmarkTests.LoadItemById: DefaultJob
//   BenchmarkTests.CreateGuid: DefaultJob
//   BenchmarkTests.DeleteGuid: DefaultJob
//   BenchmarkTests.LoadGuidByGuidId: DefaultJob
//   BenchmarkTests.LoadGuidItems: DefaultJob
//   BenchmarkTests.CreateGuidSequence: DefaultJob
//   BenchmarkTests.CreateUpdateGuid: DefaultJob
//   BenchmarkTests.Update: DefaultJob
//   BenchmarkTests.Delete: DefaultJob
//   BenchmarkTests.InsertUpdate: DefaultJob
//   BenchmarkTests.InsertUpdateItems: DefaultJob

// Validating benchmarks:
// ***** Building 15 benchmark(s) in Parallel: Start   *****
// ***** Done, took 00:00:00 (0.16 sec)   *****
// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkTests-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkTests-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkTests-report.html

// * Detailed results *
BenchmarkTests.SimpleQuery: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadItems: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadItemsReadTable: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadItemsWriteTable: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadItemById: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.CreateGuid: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.DeleteGuid: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadGuidByGuidId: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.LoadGuidItems: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.CreateGuidSequence: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.CreateUpdateGuid: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.Update: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.Delete: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.InsertUpdate: DefaultJob
Runtime = ; GC = 
There are not any results runs

BenchmarkTests.InsertUpdateItems: DefaultJob
Runtime = ; GC = 
There are not any results runs

Total time: 00:00:00 (0.18 sec)

// * Summary *

BenchmarkDotNet=v0.10.12, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Intel Core i7-6700HQ CPU 2.60GHz (Skylake), 1 CPU, 8 logical cores and 4 physical cores
Frequency=2531247 Hz, Resolution=395.0622 ns, Timer=TSC
.NET Core SDK=2.1.4
  [Host] : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT


              Method | Mean | Error |
-------------------- |-----:|------:|
         SimpleQuery |   NA |    NA |
           LoadItems |   NA |    NA |
  LoadItemsReadTable |   NA |    NA |
 LoadItemsWriteTable |   NA |    NA |
        LoadItemById |   NA |    NA |
          CreateGuid |   NA |    NA |
          DeleteGuid |   NA |    NA |
    LoadGuidByGuidId |   NA |    NA |
       LoadGuidItems |   NA |    NA |
  CreateGuidSequence |   NA |    NA |
    CreateUpdateGuid |   NA |    NA |
              Update |   NA |    NA |
              Delete |   NA |    NA |
        InsertUpdate |   NA |    NA |
   InsertUpdateItems |   NA |    NA |

Benchmarks with issues:
  BenchmarkTests.SimpleQuery: DefaultJob
  BenchmarkTests.LoadItems: DefaultJob
  BenchmarkTests.LoadItemsReadTable: DefaultJob
  BenchmarkTests.LoadItemsWriteTable: DefaultJob
  BenchmarkTests.LoadItemById: DefaultJob
  BenchmarkTests.CreateGuid: DefaultJob
  BenchmarkTests.DeleteGuid: DefaultJob
  BenchmarkTests.LoadGuidByGuidId: DefaultJob
  BenchmarkTests.LoadGuidItems: DefaultJob
  BenchmarkTests.CreateGuidSequence: DefaultJob
  BenchmarkTests.CreateUpdateGuid: DefaultJob
  BenchmarkTests.Update: DefaultJob
  BenchmarkTests.Delete: DefaultJob
  BenchmarkTests.InsertUpdate: DefaultJob
  BenchmarkTests.InsertUpdateItems: DefaultJob

// * Legends *
  Mean  : Arithmetic mean of all measurements
  Error : Half of 99.9% confidence interval
  1 ns  : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *
@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 16, 2018

Hello @eschneider999

Before we run the benchmarks, we build a dedicated project first. The project needs to reference the .csproj file which defines the benchmark.

To find the .csproj we search for file with name {yourAssemblyName}.csproj

The assembly name is defined in the properites of your project file:

image

Did you configure some custom assembly name for your benchmark project?

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 16, 2018

ok, what about Benchmarks.vbproj ?

@adamsitnik adamsitnik changed the title from Generate Exception: Unable to find BenchmarkTests in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj to Support Visual Basic project files (.vbroj) targeting .NET Core Jan 16, 2018

@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 16, 2018

@eschneider999 this was not handled properly ;)

When this build finishes you should be able to download our latest CI package with VB support from our CI feed:

<packageSources>
  <add key="bdn-CI" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>
@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 16, 2018

@eschneider999 and ofc please feel free to reopen if it does not solve the issue

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 16, 2018

thanks

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 16, 2018

why do you need the project? does that mean you can't do benchmarks with-out the project and compilation?

@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 16, 2018

to make the results stable across the runs we always generate, build and run new process for every benchmark

You can use InProcessToolchain to avoid that

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 16, 2018

It would be nice if I could benchmark my unit tests inside my project?

@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 16, 2018

you can run your benchmarks from your unit test project but we don't provide any infrastructure that allows to compare the result with previous ones or store them in any way. So we just run the benchmarks and give you the results

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 16, 2018

I couln't get it working, but maybe it was that bug you just fixed.

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 17, 2018

Still have issues? using 0.10.12.411
Also my nuget refs have yellow triangles on them?

// ***** BenchmarkRunner: Start *****
// Found benchmarks:
// BenchmarkTests.SimpleQuery: DefaultJob
// BenchmarkTests.LoadItems: DefaultJob
// BenchmarkTests.LoadItemsReadTable: DefaultJob
// BenchmarkTests.LoadItemsWriteTable: DefaultJob
// BenchmarkTests.LoadItemById: DefaultJob
// BenchmarkTests.CreateGuid: DefaultJob
// BenchmarkTests.DeleteGuid: DefaultJob
// BenchmarkTests.LoadGuidByGuidId: DefaultJob
// BenchmarkTests.LoadGuidItems: DefaultJob
// BenchmarkTests.CreateGuidSequence: DefaultJob
// BenchmarkTests.CreateUpdateGuid: DefaultJob
// BenchmarkTests.Update: DefaultJob
// BenchmarkTests.Delete: DefaultJob
// BenchmarkTests.InsertUpdate: DefaultJob
// BenchmarkTests.InsertUpdateItems: DefaultJob

// Validating benchmarks:
// ***** Building 15 benchmark(s) in Parallel: Start *****
// ***** Done, took 00:00:00 (0.16 sec) *****
// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// Generate Exception: Unable to find Benchmarks in D:\Dev\FrozenElephant\SymbioticORM\Benchmarks and its subfolders. Most probably the name of output exe is different than the name of the .(c/f)sproj

// ***** BenchmarkRunner: Finish *****

// * Export *
BenchmarkDotNet.Artifacts\results\BenchmarkTests-report.csv
BenchmarkDotNet.Artifacts\results\BenchmarkTests-report-github.md
BenchmarkDotNet.Artifacts\results\BenchmarkTests-report.html

// * Detailed results *
BenchmarkTests.SimpleQuery: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadItems: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadItemsReadTable: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadItemsWriteTable: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadItemById: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.CreateGuid: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.DeleteGuid: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadGuidByGuidId: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.LoadGuidItems: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.CreateGuidSequence: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.CreateUpdateGuid: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.Update: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.Delete: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.InsertUpdate: DefaultJob
Runtime = ; GC =
There are not any results runs

BenchmarkTests.InsertUpdateItems: DefaultJob
Runtime = ; GC =
There are not any results runs

Total time: 00:00:00 (0.19 sec)

// * Summary *

BenchmarkDotNet=v0.10.12, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)
Intel Core i7-6700HQ CPU 2.60GHz (Skylake), 1 CPU, 8 logical cores and 4 physical cores
Frequency=2531247 Hz, Resolution=395.0622 ns, Timer=TSC
.NET Core SDK=2.1.4
[Host] : .NET Core 2.0.5 (Framework 4.6.26020.03), 64bit RyuJIT

          Method | Mean | Error |

-------------------- |-----:|------:|
SimpleQuery | NA | NA |
LoadItems | NA | NA |
LoadItemsReadTable | NA | NA |
LoadItemsWriteTable | NA | NA |
LoadItemById | NA | NA |
CreateGuid | NA | NA |
DeleteGuid | NA | NA |
LoadGuidByGuidId | NA | NA |
LoadGuidItems | NA | NA |
CreateGuidSequence | NA | NA |
CreateUpdateGuid | NA | NA |
Update | NA | NA |
Delete | NA | NA |
InsertUpdate | NA | NA |
InsertUpdateItems | NA | NA |

Benchmarks with issues:
BenchmarkTests.SimpleQuery: DefaultJob
BenchmarkTests.LoadItems: DefaultJob
BenchmarkTests.LoadItemsReadTable: DefaultJob
BenchmarkTests.LoadItemsWriteTable: DefaultJob
BenchmarkTests.LoadItemById: DefaultJob
BenchmarkTests.CreateGuid: DefaultJob
BenchmarkTests.DeleteGuid: DefaultJob
BenchmarkTests.LoadGuidByGuidId: DefaultJob
BenchmarkTests.LoadGuidItems: DefaultJob
BenchmarkTests.CreateGuidSequence: DefaultJob
BenchmarkTests.CreateUpdateGuid: DefaultJob
BenchmarkTests.Update: DefaultJob
BenchmarkTests.Delete: DefaultJob
BenchmarkTests.InsertUpdate: DefaultJob
BenchmarkTests.InsertUpdateItems: DefaultJob

// * Legends *
Mean : Arithmetic mean of all measurements
Error : Half of 99.9% confidence interval
1 ns : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
// * Artifacts cleanup *

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 17, 2018

also could you output the full path and file name in the exception? if you can't find something, it should be explicit in what it's looking for.

@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 17, 2018

Also my nuget refs have yellow triangles on them?
BenchmarkDotNet=v0.10.12, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.192)

Your IDE did not download the version with the fix. Link to help

@eschneider999

This comment has been minimized.

eschneider999 commented Jan 17, 2018

using version 0.10.12.411

@AndreyAkinshin AndreyAkinshin added this to the v0.10.13 milestone Jan 17, 2018

@adamsitnik

This comment has been minimized.

Member

adamsitnik commented Jan 17, 2018

@eschneider999 your log tells me that you are using 0.10.12. Please run dotnet restore for the project with benchmarks

GeorgePlotnikov added a commit to GeorgePlotnikov/BenchmarkDotNet that referenced this issue Feb 25, 2018

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment