New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C# compiler crashes with NullReferenceException in Microsoft.CodeAnalysis.CompilationOptions.get_Errors() #12993

Closed
danielearwicker opened this Issue Aug 8, 2016 · 13 comments

Comments

Projects
None yet
5 participants
@danielearwicker

danielearwicker commented Aug 8, 2016

Version Used: Visual Studio 2015 Update 3, C# compilier

Steps to Reproduce:

Try to build a project via TFS build agent

Expected Behavior: Compilation of C# files

Actual Behavior:

From the log file created on the build machine:

 Using shared compilation with compiler from directory: C:\Program Files (x86)\MSBuild\14.0\Bin

  Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of an object.
     at Microsoft.CodeAnalysis.CompilationOptions.get_Errors()
     at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CSharp.CSharpCommandLineParser.CommonParse(IEnumerable`1 args, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CommandLineParser.Parse(IEnumerable`1 args, String baseDirectory, String sdkDirectory, String additionalReferenceDirectories)
     at Microsoft.CodeAnalysis.CommonCompiler..ctor(CommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CSharpCompiler..ctor(CSharpCommandLineParser parser, String responseFile, String[] args, String clientDirectory, String baseDirectory, String sdkDirectoryOpt, String additionalReferenceDirectories, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc..ctor(String responseFile, BuildPaths buildPaths, String[] args, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Csc.Run(String[] args, BuildPaths buildPaths, TextWriter textWriter, IAnalyzerAssemblyLoader analyzerLoader)
     at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.RunLocalCompilation(String[] arguments, BuildPaths buildPaths, TextWriter textWriter)
     at Microsoft.CodeAnalysis.CommandLine.BuildClient.RunCompilation(IEnumerable`1 originalArguments, BuildPaths buildPaths, TextWriter textWriter)
     at Microsoft.CodeAnalysis.CommandLine.DesktopBuildClient.Run(IEnumerable`1 arguments, IEnumerable`1 extraArguments, RequestLanguage language, CompileFunc compileFunc, IAnalyzerAssemblyLoader analyzerAssemblyLoader)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args, String[] extraArgs)
     at Microsoft.CodeAnalysis.CSharp.CommandLine.Program.Main(String[] args)
C:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.CSharp.Core.targets(67,5): error MSB6006: "csc.exe" exited with code 255.

I applied Update 3 about two weeks ago, to my two build machines. One of them started failing like this after a day or two. I disabled it because the other seemed fine, so I wouldn't have to investigate immediately.

Then the other machine failed after a week. I tried doing a Visual Studio repair (with the installer) on the first machine. After that it seemed to work. I did the same on the second machine and it also worked. So both seemed okay.

This morning the first build I started on the first machine has failed with same stack trace. I'm running a build on the second machine and it's working fine (the build completes without any errors in our code or yours!)

It's very odd because once a given machine has failed once, it then no longer works, although a repair install fixes it temporarily.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Aug 8, 2016

As the problem occurs in parsing the command line arguments, I've compared the args passed in a build that succeeds with one that fails, on the same project. They were identical:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE /highentropyva+ /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\mscorlib.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Configuration.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Core.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.DataSetExtensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Drawing.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.Linq.dll" /debug:pdbonly /filealign:512 /keyfile:ProductResourcesKey.snk /optimize+ /out:obj\Release\ProductResources.Interface.dll /ruleset:"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Static Analysis Tools\\Rule Sets\MinimumRecommendedRules.ruleset" /subsystemversion:6.00 /target:library /utf8output IProductResources.cs NativeMethods.cs ProductResourcesLoader.cs Properties\AssemblyInfo.cs TileLayoutImageMode.cs "C:\Users\tfs_services\AppData\Local\Temp\.NETFramework,Version=v4.5.AssemblyAttributes.cs"

danielearwicker commented Aug 8, 2016

As the problem occurs in parsing the command line arguments, I've compared the args passed in a build that succeeds with one that fails, on the same project. They were identical:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe /noconfig /nowarn:1701,1702 /nostdlib+ /errorreport:prompt /warn:4 /define:TRACE /highentropyva+ /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\mscorlib.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Configuration.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Core.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.DataSetExtensions.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Data.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Drawing.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.dll" /reference:"C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.5\System.Xml.Linq.dll" /debug:pdbonly /filealign:512 /keyfile:ProductResourcesKey.snk /optimize+ /out:obj\Release\ProductResources.Interface.dll /ruleset:"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Static Analysis Tools\\Rule Sets\MinimumRecommendedRules.ruleset" /subsystemversion:6.00 /target:library /utf8output IProductResources.cs NativeMethods.cs ProductResourcesLoader.cs Properties\AssemblyInfo.cs TileLayoutImageMode.cs "C:\Users\tfs_services\AppData\Local\Temp\.NETFramework,Version=v4.5.AssemblyAttributes.cs"

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Aug 8, 2016

If I open a command prompt on the build machine, logged in under the account that our TFS services use, and pass the same arguments to the exact same csc.exe:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe

Then it succeeds and produces output. So problem only occurs when build is started by TFS build controller.

danielearwicker commented Aug 8, 2016

If I open a command prompt on the build machine, logged in under the account that our TFS services use, and pass the same arguments to the exact same csc.exe:

C:\Program Files (x86)\MSBuild\14.0\bin\csc.exe

Then it succeeds and produces output. So problem only occurs when build is started by TFS build controller.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Aug 8, 2016

Just had some good results by rebooting the machine, so csc.exe not currently crashing. Guess I'll have to schedule a nightly reboot on each build machine.

danielearwicker commented Aug 8, 2016

Just had some good results by rebooting the machine, so csc.exe not currently crashing. Guess I'll have to schedule a nightly reboot on each build machine.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Aug 10, 2016

Unfortunately rebooting only seems to solve the problem temporarily, e.g. for a handful of builds, on one of my build machines.

danielearwicker commented Aug 10, 2016

Unfortunately rebooting only seems to solve the problem temporarily, e.g. for a handful of builds, on one of my build machines.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Sep 22, 2016

I still have two build machines that do this every so often. Let me know if I can provide further info.

danielearwicker commented Sep 22, 2016

I still have two build machines that do this every so often. Let me know if I can provide further info.

@jaredpar

This comment has been minimized.

Show comment
Hide comment
@jaredpar

jaredpar Sep 22, 2016

Member

@danielearwicker would it be possible for you to get a dump of the process when it throws that exception?

We've looked at the code and it simply doesn't seem possible that it could throw a NRE at that location. All of the code definitely assigns non-null values to the variables involved. It's a bit perplexing. A dump would give us some insight here and likely point out what we are missing.

Member

jaredpar commented Sep 22, 2016

@danielearwicker would it be possible for you to get a dump of the process when it throws that exception?

We've looked at the code and it simply doesn't seem possible that it could throw a NRE at that location. All of the code definitely assigns non-null values to the variables involved. It's a bit perplexing. A dump would give us some insight here and likely point out what we are missing.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Sep 26, 2016

Sure, will disable the nightly reboot to encourage it to happen sooner.

danielearwicker commented Sep 26, 2016

Sure, will disable the nightly reboot to encourage it to happen sooner.

@TyOverby

This comment has been minimized.

Show comment
Hide comment
@TyOverby

TyOverby Sep 27, 2016

Contributor

Much appreciated @danielearwicker!

Contributor

TyOverby commented Sep 27, 2016

Much appreciated @danielearwicker!

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@jaredpar

This comment has been minimized.

Show comment
Hide comment
@jaredpar

jaredpar Oct 3, 2016

Member

@danielearwicker been looking at the dump. One thing that stands out in the disassembly is that a profiler is running on the machine. Specifically TypeMock. It is definitely inserting itself into the assembly of the function that is crashing.

There are instances in the past where we've seen profilers cause errors like this in our code. Can you disable TypeMock re-run the scenario again and see if it comes up?

It seems likely this is the cause of the crash. The code in question here is really straight forward:

  • It's NRE on an instance field.
  • The instance field is readonly and unconditionally assigned in the types only constructor.

In short there is really no way that it should ever hit this error. Seems much more likely to be a result of TypeMock.

Member

jaredpar commented Oct 3, 2016

@danielearwicker been looking at the dump. One thing that stands out in the disassembly is that a profiler is running on the machine. Specifically TypeMock. It is definitely inserting itself into the assembly of the function that is crashing.

There are instances in the past where we've seen profilers cause errors like this in our code. Can you disable TypeMock re-run the scenario again and see if it comes up?

It seems likely this is the cause of the crash. The code in question here is really straight forward:

  • It's NRE on an instance field.
  • The instance field is readonly and unconditionally assigned in the types only constructor.

In short there is really no way that it should ever hit this error. Seems much more likely to be a result of TypeMock.

@jaredpar

This comment has been minimized.

Show comment
Hide comment
@jaredpar

jaredpar Oct 3, 2016

Member

Want to thank you again for providing the dump. Lacking a dump this bug would've been unactionable.

Member

jaredpar commented Oct 3, 2016

Want to thank you again for providing the dump. Lacking a dump this bug would've been unactionable.

@danielearwicker

This comment has been minimized.

Show comment
Hide comment
@danielearwicker

danielearwicker Oct 4, 2016

Excellent, thanks for this tip-off, will try today.

EDIT Have deleted the installed TypeMock stuff and it doesn't appear to be necessary to build successfully. Will take a few more builds before I feel confident it's resolved but this certainly seems like the "smoking gun".

Having looked at the module list in the crash dump I can see the TypeMock assemblies are loaded - this looks like a step too far in "clever" behaviour IMO, especially given this outcome. Is there a legit way to get 3rd party code into csc.exe or have they done something horrible? (I'm guessing the latter...)

In any case this was the first I've become aware it was being used here, and I think I'm going to just rip it out ASAP.

danielearwicker commented Oct 4, 2016

Excellent, thanks for this tip-off, will try today.

EDIT Have deleted the installed TypeMock stuff and it doesn't appear to be necessary to build successfully. Will take a few more builds before I feel confident it's resolved but this certainly seems like the "smoking gun".

Having looked at the module list in the crash dump I can see the TypeMock assemblies are loaded - this looks like a step too far in "clever" behaviour IMO, especially given this outcome. Is there a legit way to get 3rd party code into csc.exe or have they done something horrible? (I'm guessing the latter...)

In any case this was the first I've become aware it was being used here, and I think I'm going to just rip it out ASAP.

@TyOverby

This comment has been minimized.

Show comment
Hide comment
@TyOverby

TyOverby Oct 6, 2016

Contributor

It seems like TypeMock is indeed the culprit, so I'm going to close this bug for now. If it reproduces without TypeMock, please re-open, we'd be really interested in that.

Contributor

TyOverby commented Oct 6, 2016

It seems like TypeMock is indeed the culprit, so I'm going to close this bug for now. If it reproduces without TypeMock, please re-open, we'd be really interested in that.

@TyOverby TyOverby closed this Oct 6, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment