#if Blocks don't support all scenarios

[RealDotNetDave]

I am trying to use the code below using BenchmarkDotNet. Whenever I use a #if block in my benchmarks, I get errors like the one below.

#if !NETFRAMEWORK
        [Benchmark(Description = "LOOPING: for() & Span")]
        [BenchmarkCategory(Categories.New, Categories.GenericCollections, Categories.Span)]
        public void TestFor04()
        {
            var collection = new System.Span<PersonFixed>(personFixedCollection.ToArray());

            for (int count = 0; count < collection.Length; count++)
            {
                ProcessPerson(collection[count]);
            }
        }
#endif

Standard error:
Time Elapsed 00:00:06.01
2 Error(s)
0 Warning(s)
c3bcb847-7512-47e8-b975-1bd66eb19257.notcs(4005,17): error CS0103: The name 'TestFor04' does not exist in the current context [C:\src\Books\RockYourCode\CodePerformance\src\OutputBin\netcoreapp3.0\c3bcb847-7512-47e8-b975-1bd66eb19257\BenchmarkDotNet.Autogenerated.csproj]
c3bcb847-7512-47e8-b975-1bd66eb19257.notcs(3431,32): error CS0103: The name 'TestFor04' does not exist in the current context [C:\src\Books\RockYourCode\CodePerformance\src\OutputBin\netcoreapp3.0\c3bcb847-7512-47e8-b975-1bd66eb19257\BenchmarkDotNet.Autogenerated.csproj]
Build FAILED.
c3bcb847-7512-47e8-b975-1bd66eb19257.notcs(4005,17): error CS0103: The name 'TestFor04' does not exist in the current context [C:\src\Books\RockYourCode\CodePerformance\src\OutputBin\netcoreapp3.0\c3bcb847-7512-47e8-b975-1bd66eb19257\BenchmarkDotNet.Autogenerated.csproj]
c3bcb847-7512-47e8-b975-1bd66eb19257.notcs(3431,32): error CS0103: The name 'TestFor04' does not exist in the current context [C:\src\Books\RockYourCode\CodePerformance\src\OutputBin\netcoreapp3.0\c3bcb847-7512-47e8-b975-1bd66eb19257\BenchmarkDotNet.Autogenerated.csproj]

[AndreyAkinshin]

@RealDotNetDave thanks for the report! Currently, we do not support such use cases: all defined benchmarks should be available for all kinds of jobs (for all kinds of defined target frameworks in your case). I guess, your problem can be resolved with the help of attributes which allows excluding specific benchmarks from run under specific conditions. @adamsitnik already wrote a prototype for such kinds of filtering in #1097 I'm going to review this PR in the nearest future and merge it into master with the help of @adamsitnik. Once it merged, we will be able to easily introduce filters for target frameworks.

[RealDotNetDave]

The reason I was looking at the #if is that Microsoft uses them in their benchmarking. Anyway, I looked at the thread and don't see an attribute for methods, only an entire class. Is the TargetFrameworkJobAttribute the way to do it?

[AndreyAkinshin]

@RealDotNetDave

The reason I was looking at the #if is that Microsoft uses them in their benchmarking.

Could you show an example?

Anyway, I looked at the thread and don't see an attribute for methods, only an entire class. Is the TargetFrameworkJobAttribute the way to do it?

It seems that you read a wrong thread: there are no mentions of TargetFrameworkJobAttribute in #1097 (we dicuss TargetFrameworkJobAttribute in some other issues/PRs).

[RealDotNetDave]

The code below is from the MicroBenchmarks solution from the Microsoft repository. I haven't run that solution.

#if !NETFRAMEWORK // API added in .NET Core 2.0
        [Benchmark]
        public HashSet<T> HashSet()
        {
            var collection = new HashSet<T>(Size);
            var uniqueValues = _uniqueValues;
            for(int i = 0; i < uniqueValues.Length; i++)
                collection.Add(uniqueValues[i]);
            return collection;
        }
#endif

[RealDotNetDave]

Are you talking about the OperatingSystemsFilter attribute? It does not seem to be in the latest BDN package.

[AndreyAkinshin]

The code below is from the MicroBenchmarks solution from the Microsoft repository. I haven't run that solution.

It should work correctly only if you run a full framework job from a full framework host process or .NET Core job from a .NET Core host process. However, if you will try to run both jobs from the same process, you will get an inconsistent set of benchmarks.

Are you talking about the OperatingSystemsFilter attribute?

Yes.

It does not seem to be in the latest BDN package.

This PR is still unmerged. I hope to find some time for review in the nearest future.

[adamsitnik]

This is by design.

When running the host process targetting XYZ framework, BDN is using the reflection to obtain the list of available methods (benchmarks). If you are using #if defines, then the list of benchmarks is going to be different per host process target framework.

Performance repo docs describe how to compare multiple runtimes performance here:
https://github.com/dotnet/performance/blob/master/docs/benchmarkdotnet.md#multiple-runtimes

The host process needs to be the lowest common API denominator of the runtimes you want to compare!