New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing reference to Microsoft.CodeAnalysis.CSharp when using BenchmarkDotNet in Linqpad #445

Closed
veleek opened this Issue May 9, 2017 · 12 comments

Comments

Projects
None yet
4 participants
@veleek

veleek commented May 9, 2017

When attempting to use BenchmarkDotNet in LinqPad, you run into a benchmark "build" error like below:

// *** Build ***
// Exception: System.MissingMethodException: Method not found: 'Void Microsoft.CodeAnalysis.CSharp.CSharpCompilationOptions..ctor(Microsoft.CodeAnalysis.OutputKind, Boolean, System.String, System.String, System.String, System.Collections.Generic.IEnumerable`1<System.String>, Microsoft.CodeAnalysis.OptimizationLevel, Boolean, Boolean, System.String, System.String, System.Collections.Immutable.ImmutableArray`1<Byte>, System.Nullable`1<Boolean>, Microsoft.CodeAnalysis.Platform, Microsoft.CodeAnalysis.ReportDiagnostic, Int32, System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.String,Microsoft.CodeAnalysis.ReportDiagnostic>>, Boolean, Boolean, Microsoft.CodeAnalysis.XmlReferenceResolver, Microsoft.CodeAnalysis.SourceReferenceResolver, Microsoft.CodeAnalysis.MetadataReferenceResolver, Microsoft.CodeAnalysis.AssemblyIdentityComparer, Microsoft.CodeAnalysis.StrongNameProvider, Boolean)'.
   at BenchmarkDotNet.Toolchains.Roslyn.Builder.Build(GenerateResult generateResult, ILogger logger, Benchmark benchmark, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(ILogger logger, IToolchain toolchain, GenerateResult generateResult, Benchmark benchmark, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)

The reference to BenchmarkDotNet was added via a NuGet package and all of the dependencies that came with it. By manually adding the Microsoft.CodeAnalysis.CSharp NuGet package I'm able to get around that error.

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik May 10, 2017

Member

@veleek thanks for reporting the bug! Could you also report it in LinqPad repo? I tried to reproduce it, but I don't have the paid version of LinqPad and I can't even install BenchmarkDotNet there.

Most probably LinqPad is using and older version of Roslyn dll (Microsoft.CodeAnalysis.CSharp) and it's ignoring ours. If so then there is little we can do (except of giving more meaningful error message ;) )

Member

adamsitnik commented May 10, 2017

@veleek thanks for reporting the bug! Could you also report it in LinqPad repo? I tried to reproduce it, but I don't have the paid version of LinqPad and I can't even install BenchmarkDotNet there.

Most probably LinqPad is using and older version of Roslyn dll (Microsoft.CodeAnalysis.CSharp) and it's ignoring ours. If so then there is little we can do (except of giving more meaningful error message ;) )

@adamsitnik adamsitnik self-assigned this May 10, 2017

@albahari

This comment has been minimized.

Show comment
Hide comment
@albahari

albahari May 11, 2017

Adam - I've sent you a LINQPad license.
I'll also look into this myself soon.

albahari commented May 11, 2017

Adam - I've sent you a LINQPad license.
I'll also look into this myself soon.

@albahari

This comment has been minimized.

Show comment
Hide comment
@albahari

albahari May 11, 2017

I've taken a look, and it appears that there's a conflict between the Roslyn assemblies in LINQPad and those referenced by BenchmarkDotNet. I've fixed this in the 5.22.05 build:
http://www.linqpad.net/download.aspx#beta

However, there's still a problem which will require a fix in BenchmarkDotNet. LINQPad normally puts the compiled query into a different folder than the referenced assemblies - this allows for optimizations to reduce file I/O, which is important in the scratchpad scenario. BenchmarkDotNet puts the target executable into the same folder as the compiled query assembly (LINQPadQuery.dll), so it can't find the referenced assemblies. However, it knows where they are, because the correct full paths appear in the list of references in the batch file that it produces that calls csc. The solution is for it to handle the AppDomain's AssemblyResolve event, so that when it can't find an assembly, it looks in the correct place. Assuming refs is a string array of referenced assemblies, something like this should do the trick:

var refsDict = refs
	.Where (Path.IsPathRooted)
	.Where (File.Exists)
	.GroupBy (Path.GetFileNameWithoutExtension, StringComparer.OrdinalIgnoreCase)
	.ToDictionary (group => group.Key, group => group.First(), StringComparer.OrdinalIgnoreCase);

AppDomain.CurrentDomain.AssemblyResolve += (sender, args) =>
{
	string path;
	return refsDict.TryGetValue (new AssemblyName (args.Name).Name, out path) ? Assembly.LoadFrom (path) : null;
};

albahari commented May 11, 2017

I've taken a look, and it appears that there's a conflict between the Roslyn assemblies in LINQPad and those referenced by BenchmarkDotNet. I've fixed this in the 5.22.05 build:
http://www.linqpad.net/download.aspx#beta

However, there's still a problem which will require a fix in BenchmarkDotNet. LINQPad normally puts the compiled query into a different folder than the referenced assemblies - this allows for optimizations to reduce file I/O, which is important in the scratchpad scenario. BenchmarkDotNet puts the target executable into the same folder as the compiled query assembly (LINQPadQuery.dll), so it can't find the referenced assemblies. However, it knows where they are, because the correct full paths appear in the list of references in the batch file that it produces that calls csc. The solution is for it to handle the AppDomain's AssemblyResolve event, so that when it can't find an assembly, it looks in the correct place. Assuming refs is a string array of referenced assemblies, something like this should do the trick:

var refsDict = refs
	.Where (Path.IsPathRooted)
	.Where (File.Exists)
	.GroupBy (Path.GetFileNameWithoutExtension, StringComparer.OrdinalIgnoreCase)
	.ToDictionary (group => group.Key, group => group.First(), StringComparer.OrdinalIgnoreCase);

AppDomain.CurrentDomain.AssemblyResolve += (sender, args) =>
{
	string path;
	return refsDict.TryGetValue (new AssemblyName (args.Name).Name, out path) ? Assembly.LoadFrom (path) : null;
};
@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 11, 2017

Member

Hey @albahari! It's awesome to see you here. =)
I already tried 5.22.05, it works, thank you very much for the quick fix! Also thanks for the advice about assembly resolving. @adamsitnik, what do you think, will it help?

By the way, we have another LinqPad problem (probably related): #66 So, it makes sense to check how lprun works after all the fixes.

Member

AndreyAkinshin commented May 11, 2017

Hey @albahari! It's awesome to see you here. =)
I already tried 5.22.05, it works, thank you very much for the quick fix! Also thanks for the advice about assembly resolving. @adamsitnik, what do you think, will it help?

By the way, we have another LinqPad problem (probably related): #66 So, it makes sense to check how lprun works after all the fixes.

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 11, 2017

Member

@albahari, by the way, I think it would be cool to color BenchmarkDotNet output in LinqPad. I already wrote a quick proof-of-concept script (it already works; can be just copy-pasted to LinqPad):

public sealed class LinqPadLogger : ILogger
{
	private const string DefaultColor = "";

	public static readonly ILogger Default = new LinqPadLogger();

	private readonly Dictionary<LogKind, string> colorScheme;

	public LinqPadLogger(Dictionary<LogKind, string> colorScheme = null)
	{
		this.colorScheme = colorScheme ?? CreateColorfulScheme();
	}

	public void Write(LogKind logKind, string text) => Write(logKind, Console.Write, text);

	public void WriteLine() => Console.WriteLine();

	public void WriteLine(LogKind logKind, string text) => Write(logKind, Console.WriteLine, text);

	private void Write(LogKind logKind, Action<object> write, string text)
	{
		write(Util.WithStyle(text, "color:" + GetColor(logKind)));
	}

	private string GetColor(LogKind logKind) =>
		colorScheme.ContainsKey(logKind) ? colorScheme[logKind] : DefaultColor;

	private static Dictionary<LogKind, string> CreateColorfulScheme() =>
		new Dictionary<LogKind, string>
		{
			{ LogKind.Default, "#808080" },
			{ LogKind.Help, "#006400" },
			{ LogKind.Header, "#FF00FF" },
			{ LogKind.Result, "#A9A9A9" },
			{ LogKind.Statistic, "#00FFFF" },
			{ LogKind.Info, "#FFFF00" },
			{ LogKind.Error, "#FF0000" },
			{ LogKind.Hint, "#008B8B" }
		};

	public static Dictionary<LogKind, ConsoleColor> CreateGrayScheme()
	{
		var colorScheme = new Dictionary<LogKind, ConsoleColor>();
		foreach (var logKind in Enum.GetValues(typeof(LogKind)).Cast<LogKind>())
			colorScheme[logKind] = ConsoleColor.Gray;
		return colorScheme;
	}
}

public class LinqPadConfig : IConfig
{
	public static readonly IConfig Instance = new LinqPadConfig();

	private LinqPadConfig()
	{
	}

	public IEnumerable<IColumnProvider> GetColumnProviders() => DefaultColumnProviders.Instance;

	public IEnumerable<IExporter> GetExporters()
	{
		// Now that we can specify exporters on the cmd line (e.g. "exporters=html,stackoverflow"), 
		// we should have less enabled by default and then users can turn on the ones they want
		yield return CsvExporter.Default;
		yield return MarkdownExporter.GitHub;
		yield return HtmlExporter.Default;
	}

	public IEnumerable<ILogger> GetLoggers()
	{
		yield return LinqPadLogger.Default;
	}

	public IEnumerable<IAnalyser> GetAnalysers()
	{
		yield return EnvironmentAnalyser.Default;
		yield return OutliersAnalyser.Default;
		yield return MinIterationTimeAnalyser.Default;
	}

	public IEnumerable<IValidator> GetValidators()
	{
		yield return BaselineValidator.FailOnError;
		yield return JitOptimizationsValidator.DontFailOnError;
		yield return UnrollFactorValidator.Default;
	}

	public IEnumerable<Job> GetJobs() => Array.Empty<Job>();

	public IOrderProvider GetOrderProvider() => null;

	public ConfigUnionRule UnionRule => ConfigUnionRule.Union;

	public bool KeepBenchmarkFiles => false;

	public ISummaryStyle GetSummaryStyle() => SummaryStyle.Default;

	public IEnumerable<IDiagnoser> GetDiagnosers() => Array.Empty<IDiagnoser>();

	public IEnumerable<HardwareCounter> GetHardwareCounters() => Array.Empty<HardwareCounter>();
}

[DryJob, KeepBenchmarkFiles]
public class MyBenchmarks
{
	[Benchmark]
	public void Foo() => Thread.Sleep(100);
}

void Main()
{	
	BenchmarkRunner.Run<MyBenchmarks>(LinqPadConfig.Instance);
}

However, I have a few questions:

  • Is it possible to understand that the code was called from LinqPad? What the best way to do it?
  • Is it possible to get the current LinqPad theme (Windows Default or Dark)? Creating two default color schemes would be great.
  • Some of the internal BenchmarkDotNet methods directly use ILogger.Write instead of ILogger.WriteLine. Is it possible to dump colored html without a line break?
  • What the best way to call Util.WithStyle without a dependency to LinqPad assemblies (I guess it should be a simple call via reflection; just want to be sure that there is no a better option).

@albahari, I will be grateful for any comments.

Member

AndreyAkinshin commented May 11, 2017

@albahari, by the way, I think it would be cool to color BenchmarkDotNet output in LinqPad. I already wrote a quick proof-of-concept script (it already works; can be just copy-pasted to LinqPad):

public sealed class LinqPadLogger : ILogger
{
	private const string DefaultColor = "";

	public static readonly ILogger Default = new LinqPadLogger();

	private readonly Dictionary<LogKind, string> colorScheme;

	public LinqPadLogger(Dictionary<LogKind, string> colorScheme = null)
	{
		this.colorScheme = colorScheme ?? CreateColorfulScheme();
	}

	public void Write(LogKind logKind, string text) => Write(logKind, Console.Write, text);

	public void WriteLine() => Console.WriteLine();

	public void WriteLine(LogKind logKind, string text) => Write(logKind, Console.WriteLine, text);

	private void Write(LogKind logKind, Action<object> write, string text)
	{
		write(Util.WithStyle(text, "color:" + GetColor(logKind)));
	}

	private string GetColor(LogKind logKind) =>
		colorScheme.ContainsKey(logKind) ? colorScheme[logKind] : DefaultColor;

	private static Dictionary<LogKind, string> CreateColorfulScheme() =>
		new Dictionary<LogKind, string>
		{
			{ LogKind.Default, "#808080" },
			{ LogKind.Help, "#006400" },
			{ LogKind.Header, "#FF00FF" },
			{ LogKind.Result, "#A9A9A9" },
			{ LogKind.Statistic, "#00FFFF" },
			{ LogKind.Info, "#FFFF00" },
			{ LogKind.Error, "#FF0000" },
			{ LogKind.Hint, "#008B8B" }
		};

	public static Dictionary<LogKind, ConsoleColor> CreateGrayScheme()
	{
		var colorScheme = new Dictionary<LogKind, ConsoleColor>();
		foreach (var logKind in Enum.GetValues(typeof(LogKind)).Cast<LogKind>())
			colorScheme[logKind] = ConsoleColor.Gray;
		return colorScheme;
	}
}

public class LinqPadConfig : IConfig
{
	public static readonly IConfig Instance = new LinqPadConfig();

	private LinqPadConfig()
	{
	}

	public IEnumerable<IColumnProvider> GetColumnProviders() => DefaultColumnProviders.Instance;

	public IEnumerable<IExporter> GetExporters()
	{
		// Now that we can specify exporters on the cmd line (e.g. "exporters=html,stackoverflow"), 
		// we should have less enabled by default and then users can turn on the ones they want
		yield return CsvExporter.Default;
		yield return MarkdownExporter.GitHub;
		yield return HtmlExporter.Default;
	}

	public IEnumerable<ILogger> GetLoggers()
	{
		yield return LinqPadLogger.Default;
	}

	public IEnumerable<IAnalyser> GetAnalysers()
	{
		yield return EnvironmentAnalyser.Default;
		yield return OutliersAnalyser.Default;
		yield return MinIterationTimeAnalyser.Default;
	}

	public IEnumerable<IValidator> GetValidators()
	{
		yield return BaselineValidator.FailOnError;
		yield return JitOptimizationsValidator.DontFailOnError;
		yield return UnrollFactorValidator.Default;
	}

	public IEnumerable<Job> GetJobs() => Array.Empty<Job>();

	public IOrderProvider GetOrderProvider() => null;

	public ConfigUnionRule UnionRule => ConfigUnionRule.Union;

	public bool KeepBenchmarkFiles => false;

	public ISummaryStyle GetSummaryStyle() => SummaryStyle.Default;

	public IEnumerable<IDiagnoser> GetDiagnosers() => Array.Empty<IDiagnoser>();

	public IEnumerable<HardwareCounter> GetHardwareCounters() => Array.Empty<HardwareCounter>();
}

[DryJob, KeepBenchmarkFiles]
public class MyBenchmarks
{
	[Benchmark]
	public void Foo() => Thread.Sleep(100);
}

void Main()
{	
	BenchmarkRunner.Run<MyBenchmarks>(LinqPadConfig.Instance);
}

However, I have a few questions:

  • Is it possible to understand that the code was called from LinqPad? What the best way to do it?
  • Is it possible to get the current LinqPad theme (Windows Default or Dark)? Creating two default color schemes would be great.
  • Some of the internal BenchmarkDotNet methods directly use ILogger.Write instead of ILogger.WriteLine. Is it possible to dump colored html without a line break?
  • What the best way to call Util.WithStyle without a dependency to LinqPad assemblies (I guess it should be a simple call via reflection; just want to be sure that there is no a better option).

@albahari, I will be grateful for any comments.

@albahari

This comment has been minimized.

Show comment
Hide comment
@albahari

albahari May 11, 2017

Is it possible to understand that the code was called from LinqPad? What the best way to do it?

AppDomain.CurrentDomain.FriendlyName will match or start with "LINQPad Query Server". This is valid for the actual query. If you need to know from the tests that run in another process, you could check whether the assembly location starts with %temp%\LINQPad...

Is it possible to get the current LinqPad theme (Windows Default or Dark)?

I'll add a bool property Util.IsDarkThemeEnabled to the next build

Some of the internal BenchmarkDotNet methods directly use ILogger.Write instead of ILogger.WriteLine. Is it possible to dump colored html without a line break?

Will fix for next build

What the best way to call Util.WithStyle without a dependency to LinqPad assemblies

Reflection is one option, or you put the call into a separate class that's accessed only once you know that the LINQPad assembly is available. LINQPad never changes its AssemblyVersion, so you won't have to worry about binding redirects when LINQPad is updated.

albahari commented May 11, 2017

Is it possible to understand that the code was called from LinqPad? What the best way to do it?

AppDomain.CurrentDomain.FriendlyName will match or start with "LINQPad Query Server". This is valid for the actual query. If you need to know from the tests that run in another process, you could check whether the assembly location starts with %temp%\LINQPad...

Is it possible to get the current LinqPad theme (Windows Default or Dark)?

I'll add a bool property Util.IsDarkThemeEnabled to the next build

Some of the internal BenchmarkDotNet methods directly use ILogger.Write instead of ILogger.WriteLine. Is it possible to dump colored html without a line break?

Will fix for next build

What the best way to call Util.WithStyle without a dependency to LinqPad assemblies

Reflection is one option, or you put the call into a separate class that's accessed only once you know that the LINQPad assembly is available. LINQPad never changes its AssemblyVersion, so you won't have to worry about binding redirects when LINQPad is updated.

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 11, 2017

Member

@albahari, awesome, thanks! Waiting for the next build!

Member

AndreyAkinshin commented May 11, 2017

@albahari, awesome, thanks! Waiting for the next build!

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin May 11, 2017

Member

@albahari, create a separarte issue for this feature: #447

Member

AndreyAkinshin commented May 11, 2017

@albahari, create a separarte issue for this feature: #447

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik May 11, 2017

Member

@albahari thank you very much for the license and for the help!

I currently got to a crazy situation:

  1. I have expanded our AssemblyHelperResolver to handle LINQPad shadow copy folders
  2. I can't use it in our auto-generated executable because it's in the path not know to me yet

I need to find a better solution.

Member

adamsitnik commented May 11, 2017

@albahari thank you very much for the license and for the help!

I currently got to a crazy situation:

  1. I have expanded our AssemblyHelperResolver to handle LINQPad shadow copy folders
  2. I can't use it in our auto-generated executable because it's in the path not know to me yet

I need to find a better solution.

@albahari

This comment has been minimized.

Show comment
Hide comment
@albahari

albahari May 12, 2017

Can you read and parse BDN.Generated.bat?

string cmdLineCompilation = File.ReadAllText (Path.ChangeExtension (Assembly.GetExecutingAssembly().Location, ".bat"));

var refs = cmdLineCompilation.Split ("/".ToCharArray())
	.Where (option => option.StartsWith ("r:", StringComparison.OrdinalIgnoreCase) || option.StartsWith ("reference:", StringComparison.OrdinalIgnoreCase))
	.SelectMany (r => r.Substring (r.IndexOf (":") + 1)
		.Split (",".ToCharArray())
		.Select (x => x.Trim())
		.Reverse()
		.Select ((x, n) => n == 0 && x.Contains ("\"") ? x.Substring (0, x.LastIndexOf ("\"")) : x)
		.Select (x => x.Replace ("\"", "")));

albahari commented May 12, 2017

Can you read and parse BDN.Generated.bat?

string cmdLineCompilation = File.ReadAllText (Path.ChangeExtension (Assembly.GetExecutingAssembly().Location, ".bat"));

var refs = cmdLineCompilation.Split ("/".ToCharArray())
	.Where (option => option.StartsWith ("r:", StringComparison.OrdinalIgnoreCase) || option.StartsWith ("reference:", StringComparison.OrdinalIgnoreCase))
	.SelectMany (r => r.Substring (r.IndexOf (":") + 1)
		.Split (",".ToCharArray())
		.Select (x => x.Trim())
		.Reverse()
		.Select ((x, n) => n == 0 && x.Contains ("\"") ? x.Substring (0, x.LastIndexOf ("\"")) : x)
		.Select (x => x.Replace ("\"", "")));
@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik May 13, 2017

Member

Ok, I finally got it running.

I tried other apporach, setting the privatePath

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="Addins" />
    </assemblyBinding>
  </runtime>
</configuration>

But it does not support absolute paths, only sub directories. Which was no go because the shadow copy folder which LINQPad creates is not a sub directory of the folder where output .dll is. (I don't blame LINQPad for anything, privatePath seemed to be very promising but turned out to be useless due to limitations)

Then I decided to come back to the AssemblyLoad event. To overcome all possible problems, I had put the code of the helper in our auto-generated code file, so it has no dependencies now.

Both LINQPad and lprun works now. Once again big thanks to @albahari for help!

@veleek you should be able to download our 0.10.6.185 package from our CI feed in 20 minutes. More info

Member

adamsitnik commented May 13, 2017

Ok, I finally got it running.

I tried other apporach, setting the privatePath

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <probing privatePath="Addins" />
    </assemblyBinding>
  </runtime>
</configuration>

But it does not support absolute paths, only sub directories. Which was no go because the shadow copy folder which LINQPad creates is not a sub directory of the folder where output .dll is. (I don't blame LINQPad for anything, privatePath seemed to be very promising but turned out to be useless due to limitations)

Then I decided to come back to the AssemblyLoad event. To overcome all possible problems, I had put the code of the helper in our auto-generated code file, so it has no dependencies now.

Both LINQPad and lprun works now. Once again big thanks to @albahari for help!

@veleek you should be able to download our 0.10.6.185 package from our CI feed in 20 minutes. More info

@albahari

This comment has been minimized.

Show comment
Hide comment
@albahari

albahari May 14, 2017

I've now released LINQPad 5.22.06 beta which exposes Util.IsDarkThemeEnabled and includes the fix when calling Console.Write with an empty string.

A couple more points:

  • You can customize how LINQPad dumps the Summary by writing a ToDump method: https://www.linqpad.net/CustomizingDump.aspx. This doesn't require taking a dependency on LINQPad.exe, and even allows custom HTML.

  • You can include LINQPad samples in the NuGet package to get people started - just include them in a folder called linqpad-samples: https://www.linqpad.net/nugetsamples.aspx. This also allows users to download the NuGet package using the free edition of LINQPad.

albahari commented May 14, 2017

I've now released LINQPad 5.22.06 beta which exposes Util.IsDarkThemeEnabled and includes the fix when calling Console.Write with an empty string.

A couple more points:

  • You can customize how LINQPad dumps the Summary by writing a ToDump method: https://www.linqpad.net/CustomizingDump.aspx. This doesn't require taking a dependency on LINQPad.exe, and even allows custom HTML.

  • You can include LINQPad samples in the NuGet package to get people started - just include them in a folder called linqpad-samples: https://www.linqpad.net/nugetsamples.aspx. This also allows users to download the NuGet package using the free edition of LINQPad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment