BenchmarkDotNet should respect LangVersion project setting

[rpokrovskij]

There is VS 2017 solution developed using C# 7.2 (BenchmarkDotNet 0.10.12)
Benchmark Core project created. Using C# 7.2 instruction added to proj file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
     <TargetFrameworks>netcoreapp2.0;net47</TargetFrameworks>
	 <LangVersion>latest</LangVersion>
  </PropertyGroup>

The solution can be compiled. But after starting benchmark with
dotnet run -c Release -f netcoreapp2.0 -p "$BenchmarkProjectPath"
those errors produced:

D:\cot\DashboardCode\Routines\AdminkaV1\Injected\AdminkaRoutineHandler.cs(128,27): error CS8107: Feature 'private protected' is not available in C# 7.0. Please use language version 7.2 or greater. [D:\cot\DashboardCod
e\Routines\AdminkaV1\Injected.NETStandard\AdminkaV1.Injected.NETStandard.csproj]
D:\cot\DashboardCode\Routines\AdminkaV1\Injected\AdminkaRoutineHandler.cs(128,27): error CS8107: Feature 'private protected' is not available in C# 7.0. Please use language version 7.2 or greater. [D:\cot\DashboardCod
e\Routines\AdminkaV1\Injected.NETFramework\AdminkaV1.Injected.NETFramework.csproj]

Build FAILED.

D:\cot\DashboardCode\Routines\AdminkaV1\Injected\AdminkaRoutineHandler.cs(128,27): error CS8107: Feature 'private protected' is not available in C# 7.0. Please use language version 7.2 or greater. [D:\cot\DashboardCod
e\Routines\AdminkaV1\Injected.NETStandard\AdminkaV1.Injected.NETStandard.csproj]
D:\cot\DashboardCode\Routines\AdminkaV1\Injected\AdminkaRoutineHandler.cs(128,27): error CS8107: Feature 'private protected' is not available in C# 7.0. Please use language version 7.2 or greater. [D:\cot\DashboardCod
e\Routines\AdminkaV1\Injected.NETFramework\AdminkaV1.Injected.NETFramework.csproj]

[adamsitnik]

hi @rpokrovskij

thanks for the bug report.

I have fixed the issue, you need to wait until this build is green and download 0.10.12.425 package from our CI NuGet feed:

<packageSources>
  <add key="bdn-nightly" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[TenCoKaciStromy]

@adamsitnik Thanks for fix!
I came across the same behaviour when I want run benchmarks with referenced projects built in debug mode. The issue is still happening.

I tried create PR with fix but when I found the commit with this fix, I didn't able to resolve how to fix the behaviour with debug mode.

The solution/workaround is - change the configuration from "Debug" to "Release" and change on all projects the the language version to 7.x (same like in debug mode). It will write this settings into *.csproj.
After that, you can change configuration back to "Debug" and all runs as expected.

I don't know if cases with debug mode is important for scope of this project. Maybe the workaround is good enough, so I didn't open new issue. I hope this will help others when they trying resolve same issue as me.

[adamsitnik]

hi @TenCoKaciStromy

Even if you run the benchmark process in Debug, BDN is going to rebuild it and run in Release. Why would you like to benchmark the Debug build?

[TenCoKaciStromy]

Hi,
it's not the case where I need run in debug necessarily. I don't know workflow of other devs but in my case (in current project) it looks like this:

• stage 1
• write tests
• implement the feature
• fix feature
• write performance tests
• stage 2
• run performance tests
• improve performance of feature (sometimes there is work on others related features to allow performance improvement)
• fix bugs created during performance improvement
• repeat stage 2 until I'm ok with performance

In release mode I can't debug my code, so I need to switch between debug/release configuration. It involves two main inconveniences.

1. I forgot switching configurations all the time.
2. I need sync debug & release configuration setup for every project in solution (compilation symbols, pre/post-build events, etc...).
   And the third thing is there is bug in Test Explorer. It sometime stucks after switching between debug/release configuration, the VS restart is needed. I know it's totally out of scope from this project. But remaining in debug configuration solves all these things.

There is one more scenario I know about (but doesn't bother me now). I know environments where application in production is compiled in debug mode because the debug information is much more valuable than performance gain from release build. In this case, the measuring of performance is still valueable even if performance is not top priority.

Maybe I'm missing something. I would love to know more about and improve myself

[adamsitnik]

Hi @TenCoKaciStromy

Unit tests should test the correctness, benchmarks the performance. There is a HUGE difference for the performance for Debug vs Release builds (in some cases it can be more than x100) and we don't allow to benchmark Debug code on purpose because it does not make any sense (it would be something similar to testing security with anonymous authentication enabled for entire web app).

However, I understand you IDE frustration ;)

What I can recommend:

1. Use VS for writing test and benchmarks, debugging and running tests
2. Open a new console window and use MSBuild to build your benchmarks project in Release, run the benchmarks from the console.

[rpokrovskij]

I use shared project to have the same "test code" for unit tests and benchmark projects when they have common code and direct referencing doesn't work because of platform specific.

[nietras]

@adamsitnik I'm trying to use BDN with a project where I am using the latest Compiler toolset i.e.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>net5.0;net48</TargetFrameworks>
    <OutputType>Exe</OutputType>
    <LangVersion>preview</LangVersion>
  </PropertyGroup>
  
  <PropertyGroup>
    <PlatformTarget>AnyCPU</PlatformTarget>
    <DebugType>pdbonly</DebugType>
    <DebugSymbols>true</DebugSymbols>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <Optimize>true</Optimize>
    <Configuration>Release</Configuration>
    <IsPackable>false</IsPackable>
  </PropertyGroup>
  
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.12.1" />
    <PackageReference Include="BenchmarkDotNet.Diagnostics.Windows" Version="0.12.1" />
    <PackageReference Include="Microsoft.Net.Compilers.Toolset" Version="4.0.0-2.21310.45">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>
</Project>

However, this fails with the LangVersion as can be seen below:

BenchmarkDotNet.Autogenerated.csproj(20,18): error MSB4086: A numeric comparison was attempted on "$(LangVersion)" that evaluates to "preview" instead of a number, in condition "'$(LangVersion)' == '' Or '$(LangVersion)' < '7.3'".

PS: This seemed related to this issue. Problem is the last check it appears.

[adamsitnik]

@nietras please update to 0.13.0, the issue should be fixed there

[nietras]

@adamsitnik yeah just discovered that in #1537 thanks! 😀