Using BenchmarkDotNet with Unity

[AndreasObel]

Hello!
I'm working with a semester project at my university where i would like to benchmark some of my code.

I have an issue with using BenchmarkDotNet with VS2017 code which i have created through Unity.
Ive been trying to find several guides and other stuff which did not help me. first of I've changed the Scripting Runtime Version to .NET 4.x Equivelant and the API Compatibility Level to .NET 4.x in Unity.

Next up i tried changing the Target Framework in Visual Studio to .NET Framework 4.6 since it would seem that is what i needed to do since that is what is said on the NuGet Website.
When i try to install the BenchmarkDotNet from the NuGet Manager in VS i get the error:

"Could not install package 'BenchmarkDotNet 0.10.14'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.6', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author."

I've tried to install the BenchmarkDotNet.Core and BenchmarkDotNet.Toolchains.Roslyn without any problems to my knowledge.

This is about as far as I've gotten and i dont know what to do about it. Maybe I'm missing something obvious but I'm completely new to the .NET scene and i dont know what to do!

Have a nice day!

[adamsitnik]

Hi @AndreasObel

I don't have Unity installed now, but perhaps @xoofx has some experience with using BenchmarkDotNet, Unity and Visual Studio all together?

[xoofx]

@AndreasObel the projects generated by the Unity editor are not standard .NET projects so they are likely not playing nicely with NuGet, which is a bit unfortunate I agree...

I checked and you can maybe workaround this by editing csproj manually before trying to install a NuGet package, and then edit back the project once it is installed... but as you will see, that's not something I would try to do...

You need to comment the lines in your csproj

    <!-- <NoConfig>true</NoConfig>
    <NoStdLib>true</NoStdLib> -->

and all the System/Netstandard assemblies :

!--  
 <Reference Include="netstandard">
 <HintPath>C:/Work/unity/build/WindowsEditor/Data/NetStandard/ref/2.0.0/netstandard.dll</HintPath>
 </Reference>
 <Reference Include="Microsoft.Win32.Primitives">
 <HintPath>C:/Work/unity/build/WindowsEditor/Data/NetStandard/compat/2.0.0/shims/netstandard/Microsoft.Win32.Primitives.dll</HintPath>
 </Reference>
....
 <Reference Include="System.Xml.Serialization">
 <HintPath>C:/Work/unity/build/WindowsEditor/Data/NetStandard/compat/2.0.0/shims/netfx/System.Xml.Serialization.dll</HintPath>
 </Reference> -->

Then you should be able to install a NuGet package.

The problem is that it will install also a bunch System.* package reference that you will have to remove by hand into your project, then uncomment what was commented before.... and it should compile.

But not sure BenchmarkDotNet will work anyway in the end. You need to have a console program for this (and then, why bother using a Unity project to create it, as it is actually not really supported)

Bottom line is that you can only use BenchmarkDotNet with regular .NET code. If you try to use some Unity code, it will likely fail at some point on Unity API (the UnityEngine DLL requiring a custom Unity .NET runtime)

But if it is to benchmark some .NET code, then you can setup a regular C# project from VS that collect the same files used by your Unity csproj project and test this more easily from there...

[adamsitnik]

@xoofx big thanks for help!

@AndreasObel I am afraid that benchmarking Unity is not supported as of today.

@xoofx if you are ever interested in adding such support please let me know. It should not be hard to implement

[DeathArrow01]

@xoofx would it be possible to use UnityEngine.dll in a .net project? I don't need to benchmark Unity methods but I need Unity data types like Transform.

[xoofx]

@xoofx would it be possible to use UnityEngine.dll in a .net project? I don't need to benchmark Unity methods but I need Unity data types like Transform.

@DeathArrow01 you can, but if you start to use some methods on Unity objects, you can quickly hit an internal call that would not run on a .NET standard runtime (even from static initializers, so it can be impossible in that case)

[DeathArrow01]

@xoofx thank you!
It seems it's not a trivial task to use BenchmarkDotNet with Unity scripts. It will require either modifying Unity engine code, which I doubt someone from Unity will have time, either modifying BenchmarkDotNet, so it can be imported into Unity project and call Unity methods on Unity's Main Thread.

[hiradyazdan]

@xoofx is it possible to use a unity class library which only references UnityEngine.dll and UnityEngine.UI.dll, only the mode is set to a different profile than release (not debug, but a custom one with optimize tag to true)?

The methods under performance test are not unity related at all, the only reason for having a unity class library is only that I can have an output to use for benchmark.

It looks like the build compiles fine with BenchmarkDotNet, but when running with ClrJob I still get these errors:

Assembly myproject.Benchmark.Unity which defines benchmarks references non-optimized UnityEngine
        If you own this dependency, please, build it in RELEASE.
        If you don't, you can disable this policy by using 'config.With(ConfigOptions.DisableOptimizationsValidator)'.

Assembly myproject.Benchmark.Unity which defines benchmarks references non-optimized UnityEngine.UI
        If you own this dependency, please, build it in RELEASE.
        If you don't, you can disable this policy by using 'config.With(ConfigOptions.DisableOptimizationsValidator)'.