ExecutionEngineException in RegisterWebApiControllers

[mscrivo]

This may have nothing to do with Simple Injector and something we're doing entirely wrong, but I am unable to troubleshoot this any further and was wondering if you could help.

Almost every time we build, and IIS reloads all assemblies, we get a crash in w3wp. Debugging always takes us to this line in one of our web apps' Global.asax:

container.RegisterWebApiControllers(GlobalConfiguration.Configuration);

Here's what happens before that line for context:

var container = new Container();
container.Options.DefaultScopedLifestyle = new AsyncScopedLifestyle();
container.Options.PropertySelectionBehavior = new InjectPropertySelectionBehavior();

#if DEBUG
// tried adding this to make the exception go away, but it did not work
container.Options.EnableDynamicAssemblyCompilation = false;
#endif

// this enumerates all our dependencies and registers them.
var registry = new Registry();
registry.All(container);

container.RegisterMvcControllers(Assembly.GetExecutingAssembly());
container.RegisterWebApiControllers(GlobalConfiguration.Configuration);

This only happens on dev machines, not Production, thankfully. But it's super annoying as it dings productivity.

Anecdotally, it started happening when we started converting some our of assemblies to .NET Standard 2.0 from .NET Framework. Our solution is now a mix of .NET 4.7.1 and .NET Standard 2.0 assemblies.

Do you have any suggestions on how we could get to the bottom of this?

[dotnetjunkie]

Which version of Simple Injector are you using and have you tried upgrading?

[mscrivo]

4.0.12

[dotnetjunkie]

Some users reported getting ExecutionEngineExceptions, but that issue has been fixed with #274 by disabling EnableDynamicAssemblyCompilation by default.

If you're accidentally enabling EnableDynamicAssemblyCompilation, you could run into problems, when you are running tools like IntelliTrace, dotTrace, MSTest and possibly some weird configurations of IIS.

However, if you leave EnableDynamicAssemblyCompilation to the default of false, this problem should not exist.

Please make sure that you aren't setting EnableDynamicAssemblyCompilation somewhere in your code. If you are sure it is false, it could be that the problem is not related to Simple Injector.

You can check this by looking at a crash dump (example). The stack trace will reveal whether the error originates from Simple Injector. If it does, please post the stack trace.

[mscrivo]

Ok thanks for the quick response, I'll get back to you on that stack trace, or close the issue if SI is not implicated in it.

[dotnetjunkie]

In case Simple Injector is not involved, it would still be good to explain what caused the problem, in case others are having the same issue.

[mscrivo]

Here's what I managed to get after taking a dump when w3wp crashed and analyzing with DebugDiag:

clr!EEClassHashTable::ConstructKeyFromData+9f 
clr!EEClassHashTable::FindItem+b5 
clr!ClassLoader::GetClassValue+123 
clr!ClassLoader::FindClassModuleThrowing+f0 
clr!ClassLoader::LoadTypeHandleThrowing+ac 
clr!ClassLoader::LoadTypeHandleThrowIfFailed+27 
clr!ClassLoader::LoadTypeDefOrRefThrowing+399 
clr!SigPointer::GetTypeHandleThrowing+42f 
clr!SigPointer::GetTypeHandleThrowing+27e 
clr!FieldDesc::GetFieldTypeHandleThrowing+189 
clr!MethodTable::DoFullyLoad+61b 
clr!ClassLoader::LoadTypeHandleForTypeKey+12d 
clr!ClassLoader::LoadTypeDefThrowing+227 
clr!ClassLoader::LoadTypeDefOrRefThrowing+279 
clr!GetTypesInner+216 
[[GCFrame]] 
[[GCFrame]] 
[[GCFrame]] 
clr!COMModule::GetTypes+ba 
[[HelperMethodFrame_2OBJ] (System.Reflection.RuntimeModule.GetTypes)] System.Reflection.RuntimeModule.GetTypes(System.Reflection.RuntimeModule) 
mscorlib_ni!System.Reflection.Assembly.GetTypes()+70 
System.Web.Http.Dispatcher.DefaultHttpControllerTypeResolver.GetControllerTypes(System.Web.Http.Dispatcher.IAssembliesResolver)+ea 
System.Web.Http.WebHost.WebHostHttpControllerTypeResolver.GetControllerTypes(System.Web.Http.Dispatcher.IAssembliesResolver)+58 
SimpleInjector.SimpleInjectorWebApiExtensions.GetControllerTypesFromConfiguration(System.Web.Http.HttpConfiguration, System.Web.Http.Dispatcher.IAssembliesResolver)+22 
SimpleInjector.SimpleInjectorWebApiExtensions.RegisterWebApiControllers(SimpleInjector.Container, System.Web.Http.HttpConfiguration, System.Collections.Generic.IEnumerable`1<System.Reflection.Assembly>)+68 
Nudge.ApplicationServices.Web.WebApiApplication.Application_Start()+173 
clr!CallDescrWorkerInternal+83 
clr!CallDescrWorkerWithHandler+4e 
clr!CallDescrWorkerReflectionWrapper+1a 
clr!RuntimeMethodHandle::InvokeMethod+45f 
[[DebuggerU2MCatchHandlerFrame]] 
[[HelperMethodFrame_PROTECTOBJ] (System.RuntimeMethodHandle.InvokeMethod)] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) 
mscorlib_ni!System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[])+72 
mscorlib_ni!System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)+92 
mscorlib_ni!System.Reflection.MethodBase.Invoke(System.Object, System.Object[])+22 
System.Web.HttpApplication.InvokeMethodWithAssert(System.Reflection.MethodInfo, Int32, System.Object, System.EventArgs)+54 
System.Web.HttpApplication.ProcessSpecialRequest(System.Web.HttpContext, System.Reflection.MethodInfo, Int32, System.Object, System.EventArgs, System.Web.SessionState.HttpSessionState)+12d 
System.Web.HttpApplicationFactory.EnsureAppStartCalledForIntegratedMode(System.Web.HttpContext, System.Web.HttpApplication)+17b 
System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr, System.Web.HttpContext, System.Reflection.MethodInfo[])+b9 
System.Web.HttpApplication.InitSpecial(System.Web.HttpApplicationState, System.Reflection.MethodInfo[], IntPtr, System.Web.HttpContext)+ac 
System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr, System.Web.HttpContext)+1a2 
System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr)+171 
DomainNeutralILStubClass.IL_STUB_COMtoCLR(Int64)+18 
clr!COMToCLRDispatchHelper+39 
clr!InvokeStub+24 
clr!COMToCLRWorkerBody_SOIntolerant+2bf 
clr!COMToCLRWorkerBodyWithADTransition+240 
[[ContextTransitionFrame]] 
clr!COMToCLRWorker+ffef567a 
clr!GenericComCallStub+57 
[[ComMethodFrame]] 
webengine4!W3_MGD_APP_CONTEXT::InitializeMgdApplication+bb 
webengine4!W3_MGD_APP_CONTEXT::OnApplicationResolveModules+ef 
webengine4!CMgdEngGlobalModule::OnGlobalApplicationResolveModules+6f 
iiscore!W3_SERVER::GlobalNotify+df 
iiscore!W3_APPLICATION::ResolveModules+2b 
iiscore!IISCORE_PROTOCOL_MANAGER::Release+a64 
iiscore!W3_CONTEXT::SetupStateMachinePhase2+443 
iiscore!W3_CONTEXT::SetupStateMachine+69c 
iiscore!W3_MAIN_CONTEXT::OnNewRequest+25f 
w3dt!WP_CONTEXT::OnCompletion+b5 
w3tp!THREAD_POOL_DATA::ThreadPoolThread+73 
w3tp!THREAD_POOL_DATA::ThreadPoolThread+30 
w3tp!THREAD_MANAGER::ThreadManagerThread+49 
kernel32!BaseThreadInitThunk+14 
ntdll!RtlUserThreadStart+21 

These exceptions were listed as well:

Cannot create an instance of abstract type
The configuration is invalid. The type is directly or indirectly depending on itself.

[dotnetjunkie]

That seems like a weird stack trace. Are you sure this is where the ExecutionEngineException originates from?

If it is the source, what you can see it that it is actually the DefaultHttpControllerTypeResolver that calls into Assembly.GetTypes(). In that case it is not related to Simple Injector, since with Simple Injector this problem happens at the moment that Expression trees are being compiled.

It would be interesting to find out which assembly is it is that GetTypes() is failing on. Perhaps its a dynamically generated assembly. Simple Injector actually generated assemblies, but only when EnableDynamicAssemblyCompilation is set.

[mscrivo]

I can't seem to get a dump of exactly when the exception happens, when I try doing that, it just hangs the system indefinitely, never creating the dump. I'm only able to take the dump immediately after the crash happens and IIS is starting to shutdown. Two things stand out to me, the above stack trace is indeed the one where the ExecutionEngineException is happening, but what actually caused it is still unknown. The other interesting thing is that the analysis is saying that the reason IIS is shutting down is because the contents of the bin folder have changed, which I find very odd, but inline with your conjecture that some assembly is being dynamically generated causing the whole thing to fall over.

I've attached the analysis if you have any interest in taking a look.

Also, I've verified that EnableDynamicAssemblyCompilation is not being set to true anywhere in the codebase and I will try to find out which assembly GetTypes() is failing on.

w3wp.exe_CrashHangAnalysis.mht.zip

[mscrivo]

Getting a bit closer, got the real reason for the exception now:

In w3wp.exe.9696.dmp the assembly instruction at clr!EEClassHashTable::ConstructKeyFromData+9f in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll from Microsoft Corporation has caused an access violation exception (0xC0000005) when trying to read from memory location 0x00000000 on thread 5

and it's definitely from GetTypes() as you said .. now just gotta track down the assembly and type.

[dotnetjunkie]

Do note that even when Simple Injector generates an assembly, this will not cause the /bin folder to change. That assembly will only live in memory. There must be some other tool causing this change.

Might there be a virus scanner screwing things up here?

[mscrivo]

No AV other than build in Windows Defender, and this is happening across at least 3 machines that I know of so far.

[mscrivo]

Ok, built SI from source and using that in our project, I was able to see exactly where it's happening while debugging the w3wp crash in VS:

 	mscorlib.dll!System.Reflection.RuntimeConstructorInfo.GetParametersNoCopy()	Unknown	No symbols loaded.
 	mscorlib.dll!System.Reflection.RuntimeConstructorInfo.GetParameters()	Unknown	No symbols loaded.
 	SimpleInjector.dll!SimpleInjector.Advanced.AdvancedExtensions.Verify(SimpleInjector.Advanced.IDependencyInjectionBehavior behavior, System.Reflection.ConstructorInfo constructor) Line 195	C#	Symbols loaded.
>	SimpleInjector.dll!SimpleInjector.ContainerOptions.IsConstructableType(System.Type implementationType, out string errorMessage) Line 422	C#	Symbols loaded.
 	SimpleInjector.dll!SimpleInjector.Container.ThrowArgumentExceptionWhenTypeIsNotConstructable(System.Type implementationType, string parameterName) Line 1229	C#	Symbols loaded.
 	SimpleInjector.dll!SimpleInjector.Container.Register(System.Type serviceType, System.Type implementationType, SimpleInjector.Lifestyle lifestyle, string serviceTypeParamName, string implementationTypeParamName) Line 1197	C#	Symbols loaded.
 	SimpleInjector.dll!SimpleInjector.Container.Register(System.Type serviceType, System.Type implementationType, SimpleInjector.Lifestyle lifestyle) Line 605	C#	Symbols loaded.
 	Nudge.Core.dll!Nudge.Core.Client.GenericClient.Registry.Local(SimpleInjector.Container container) Line 10	C#	Symbols loaded.
 	Nudge.Core.dll!Nudge.Core.Injection.Registry<Nudge.ApplicationServices.Web.Registry>.All(SimpleInjector.Container container) Line 41	C#	Symbols loaded.
 	Nudge.ApplicationServices.Web.dll!Nudge.ApplicationServices.Web.WebApiApplication.Application_Start() Line 26	C#	Symbols loaded.

The type it's failing on is from our code, trying to figure out what's special about that type now.

[mscrivo]

So the only interesting thing I can discern about this class of ours, is that it uses HttpClient which has different implementations in .NET Framework and CoreClr.

[mscrivo]

Turns out this is some nasty incompatibility between System.Net.Http assemblies in .net standard 2.0 and .net framework 4.7.1. If you have a .net 4.7.1 web project calling into a .net standard assembly and you pass objects from the System.Net.Http namespace, such as HttpClient, from one to the other, this exception happens.

To fix it, I've had to do a combination of things:

1. Set <ImplicitlyExpandNETStandardFacades>False</ImplicitlyExpandNETStandardFacades> in the .net Framework projects that reference the .net standard projects and have the cross boundary usages of System.Net.Http
2. Force the web projects to use an older version of System.Net.Http which does not exhibit the problem:

<dependentAssembly>
  <assemblyIdentity name="System.Net.Http" publicKeyToken="B03F5F7F11D50A3A" culture="neutral"/>
  <bindingRedirect oldVersion="0.0.0.0-4.2.0.0" newVersion="4.0.0.0"/>
</dependentAssembly>

So nothing at all to do with SI, but I appreciate the help!

[dotnetjunkie]

Wow, that's really nasty. I'm really glad you figured this out. In case someone else finds this page by googling, do you have any sources or references (such as MSDN, Stackoverflow or Github) that explain this problem?

[mscrivo]

I pieced it together from a bunch of different threads, still not entirely confident on what the right thing to do is.

dotnet/sdk#1647 (seems like a fix should be incoming for .net 4.7.2)
https://github.com/dotnet/corefx/issues/25773
dotnet/msbuild#2527

[mscrivo]

Confirmation that the issue is real and that it should be fixed in the VS 2017 15.6 tooling: dotnet/sdk#1712 (comment) and then the above workaround should no longer be needed.