C# compiler crashes with NullReferenceException in Microsoft.CodeAnalysis.CompilationOptions.get_Errors()

[danielearwicker]

Version Used: Visual Studio 2015 Update 3, C# compilier

Steps to Reproduce:

Try to build a project via TFS build agent

Expected Behavior: Compilation of C# files

Actual Behavior:

From the log file created on the build machine:

 Using shared compilation with compiler from directory: C:\Program Files (x86)\MSBuild\14.0\Bin

  Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
     at Microsoft.CodeAnalysis.CompilationOptions.get_Errors()
     at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.CommonParse(IEnumerable`1 args, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CommonCompiler..ctor(CommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CSharpCompiler..ctor(CSharpCommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc..ctor(String responseFile, BuildPaths buildPaths, String[] args, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.RunLocalCompilation(String[] arguments, BuildPaths buildPaths, TextWriter textWriter)
     at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter)
     at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.Run(IEnumerable`1 arguments, IEnumerable`1 extraArguments, RequestLanguage language, CompileFunc compileFunc, IAnalyzerAssemblyLoader analyzerAssemblyLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args, String[] extraArgs)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args)
C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.CSharp.Core.targets(67,5): error MSB6006: "csc.exe" exited with code 255.

I applied Update 3 about two weeks ago, to my two build machines. One of them started failing like this after a day or two. I disabled it because the other seemed fine, so I wouldn't have to investigate immediately.

Then the other machine failed after a week. I tried doing a Visual Studio repair (with the installer) on the first machine. After that it seemed to work. I did the same on the second machine and it also worked. So both seemed okay.

This morning the first build I started on the first machine has failed with same stack trace. I'm running a build on the second machine and it's working fine (the build completes without any errors in our code or yours!)

It's very odd because once a given machine has failed once, it then no longer works, although a repair install fixes it temporarily.

[danielearwicker]

As the problem occurs in parsing the command line arguments, I've compared the args passed in a build that succeeds with one that fails, on the same project. They were identical:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE /highentropyva+ /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\mscorlib.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Configuration.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Core.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.DataSetExtensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Drawing.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.Linq.dll" /debug:pdbonly /filealign:512 /keyfile:ProductResourcesKey.snk /optimize+ /out:obj\Release\ProductResources.Interface.dll /ruleset:"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Static Analysis Tools\\Rule Sets\MinimumRecommendedRules.ruleset" /subsystemversion:6.00 /target:library /utf8output IProductResources.cs NativeMethods.cs ProductResourcesLoader.cs Properties\AssemblyInfo.cs TileLayoutImageMode.cs "C:\Users\tfs_services\AppData\Local\Temp\.NETFramework,Version=v4.5.AssemblyAttributes.cs"

[danielearwicker]

If I open a command prompt on the build machine, logged in under the account that our TFS services use, and pass the same arguments to the exact same csc.exe:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe

Then it succeeds and produces output. So problem only occurs when build is started by TFS build controller.

[danielearwicker]

Just had some good results by rebooting the machine, so csc.exe not currently crashing. Guess I'll have to schedule a nightly reboot on each build machine.

[danielearwicker]

Unfortunately rebooting only seems to solve the problem temporarily, e.g. for a handful of builds, on one of my build machines.

[danielearwicker]

I still have two build machines that do this every so often. Let me know if I can provide further info.

[jaredpar]

@danielearwicker would it be possible for you to get a dump of the process when it throws that exception?

We've looked at the code and it simply doesn't seem possible that it could throw a NRE at that location. All of the code definitely assigns non-null values to the variables involved. It's a bit perplexing. A dump would give us some insight here and likely point out what we are missing.

[danielearwicker]

Sure, will disable the nightly reboot to encourage it to happen sooner.

[TyOverby]

Much appreciated @danielearwicker!

[danielearwicker]

Got a crash last night, here's a full dump file called csc.exe.9248.dmp shared via OneDrive.

[jaredpar]

@danielearwicker been looking at the dump. One thing that stands out in the disassembly is that a profiler is running on the machine. Specifically TypeMock. It is definitely inserting itself into the assembly of the function that is crashing.

There are instances in the past where we've seen profilers cause errors like this in our code. Can you disable TypeMock re-run the scenario again and see if it comes up?

It seems likely this is the cause of the crash. The code in question here is really straight forward:

• It's NRE on an instance field.
• The instance field is readonly and unconditionally assigned in the types only constructor.

In short there is really no way that it should ever hit this error. Seems much more likely to be a result of TypeMock.

[jaredpar]

Want to thank you again for providing the dump. Lacking a dump this bug would've been unactionable.

[danielearwicker]

Excellent, thanks for this tip-off, will try today.

EDIT Have deleted the installed TypeMock stuff and it doesn't appear to be necessary to build successfully. Will take a few more builds before I feel confident it's resolved but this certainly seems like the "smoking gun".

Having looked at the module list in the crash dump I can see the TypeMock assemblies are loaded - this looks like a step too far in "clever" behaviour IMO, especially given this outcome. Is there a legit way to get 3rd party code into csc.exe or have they done something horrible? (I'm guessing the latter...)

In any case this was the first I've become aware it was being used here, and I think I'm going to just rip it out ASAP.

[TyOverby]

It seems like TypeMock is indeed the culprit, so I'm going to close this bug for now. If it reproduces without TypeMock, please re-open, we'd be really interested in that.