Intermittent System.ExecutionEngineException with EF Core DBContext

[camainc]

After updating the nuget packages in my solution, I started experiencing this error when starting a debug session. It happens almost every other time I start debugging. The stack trace is empty, there is no inner exception, and the only thing I can do is stop execution and then restart the debugging session.

Exception message:

System.ExecutionEngineException
HResult=0x80131506
Message=Exception of type 'System.ExecutionEngineException' was thrown.

Stack trace: Empty
Steps to reproduce

protected WinsDbContextBase(DbContextOptions options, ICurrentUserContext userContext)
: base(options) <== The error occurs here
{
...
}```

Further technical details

EF Core version: (found in project.csproj or packages.config) 2.2.1
Database Provider: (e.g. Microsoft.EntityFrameworkCore.SqlServer) SqlServer
Operating system: Windows 10
IDE: (e.g. Visual Studio 2017 15.4) VS 2017 15.9.5

[RussKeldorph]

This issue was moved to dotnet/efcore#14495

[ajcvickers]

@RussKeldorph See dotnet/efcore#14487 If there is something more that we can do here on the EF side, then please let us know.

[RussKeldorph]

@camainc Sorry, there was no reference to the EF issue so I assumed you had come straight here. We may not be able to help without the ability to reproduce it. @dotnet/dotnet-diag may have more specific advice.

[camainc]

I understand. If there were more information available, it would at least give me a pointer in the right direction, but there is no stack trace, no inner exception, or anything else to go on. And I'm not smart enough to delve into the underlying CLR source.

[camainc]

I just wonder if it has anything to do with dependency injection.

[noahfalk]

What would be most helpful to debug this is to capture a dump of the debuggee prior to continuing from that exception/killing the debuggee. If you are using Visual Studio you should be able to capture the dump via the Debug menu > Save Dump As. Once you have that dump just share it with us and we can investigate what may be going wrong.

[camainc]

Here is a link to the file on my OneDrive.

https://1drv.ms/u/s!AucEIaq2vxfqoel-LXOwxmLPgLaNyA

[noahfalk]

Thanks @camainc ! I've started investigating and I see likely chain of things going wrong, I don't have a root cause yet but making progress. So far I can't find any obvious connection between using the debugger and this failure. It could be that connection is subtle and we haven't dug deep enough yet, but if you are able to reproduce this failure without the debugger please let me know. Also if the failure happens again it would be great to get a second dump for comparison in order to learn if the issue is consistent.

In terms of investigating, here is what I am seeing so far.

1. The process is running clr version 4.7.3221.0 built by: NET472REL1LAST_C
2. Autofac has called DbContext.ctor, which has invoked the JIT compiler to create code for Microsoft.EntityFrameworkCore.Internal.ServiceProviderCache..ctor()

20 00000014`ca4fa040 00007ff7`fefd032c clr!ThePreStub+0x55
21 00000014`ca4fa0f0 00007ff8`5a806d93 Microsoft_EntityFrameworkCore!Microsoft.EntityFrameworkCore.Internal.ServiceProviderCache..cctor()+0x2c
22 00000014`ca4fa130 00007ff8`5a806c48 clr!CallDescrWorkerInternal+0x83
23 00000014`ca4fa170 00007ff8`5a848f8f clr!CallDescrWorkerWithHandler+0x4e
24 00000014`ca4fa1b0 00007ff8`5a848f63 clr!DispatchCallDebuggerWrapper+0x1f
25 00000014`ca4fa210 00007ff8`5a848afa clr!DispatchCallSimple+0x93
26 00000014`ca4fa2b0 00007ff8`5a848cab clr!MethodTable::RunClassInitEx+0xf6
27 00000014`ca4fa3e0 00007ff8`5a8489f9 clr!MethodTable::DoRunClassInitThrowing+0x308
28 00000014`ca4fae60 00007ff8`5a989879 clr!MethodTable::CheckRunClassInitThrowing+0xa3
29 00000014`ca4fae90 00007ff7`fefd01fa clr!JIT_GetSharedNonGCStaticBase_Helper+0xa9
2a 00000014`ca4faff0 00007ff7`fefcf473 Microsoft_EntityFrameworkCore!Microsoft.EntityFrameworkCore.DbContext..ctor(Microsoft.EntityFrameworkCore.DbContextOptions)+0x8a
2b 00000014`ca4fb030 00007ff7`fefcae77 WinsApp_Infrastructure_Data!WinsApp.Infrastructure.Data.WinsDbContextBase..ctor(Microsoft.EntityFrameworkCore.DbContextOptions, WinsApp.Core.Utilities.ICurrentUserContext)+0x53
2c 00000014`ca4fb0b0 00007ff7`ff18026a WinsApp_Infrastructure_Data!WinsApp.Infrastructure.Data.WinsAppContext..ctor(WinsApp.Core.Interfaces.Utils.IDbOptionsProvider, WinsApp.Core.Utilities.ICurrentUserContext)+0x67
2d 00000014`ca4fb110 00007ff7`fefbf51e DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object[])+0x8a
2e 00000014`ca4fb160 00007ff7`fefc1f5e Autofac!Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()+0x10e

3. The JIT is trying to resolve the type System.ValueTuple`2 in netstandard.dll, referenced from module Microsoft.EntityFrameworkCore.dll.

4. netstandard.dll should have a forwarder that indicates System.ValueTuple`2 is defined in mscorlib.dll, but somehow the runtime type loader computed that the type resolved to an unnamed module that appears to be in a uninitialized or unloaded state.

5. Shortly later the the runtime type loader attempts get metadata about System.ValueTuple`2 from the invalid module and this triggers an access violation because module doesn't have any metadata loaded.

6. The AV in clr.dll's code is transformed into the FatalExecutionEngineException observed by the debugger.

Progress continues, I'll let you know once we've got more...

[camainc]

Wow, thanks! I'll upload another dump tomorrow. The error happens literally every other time I start debugging, so it won't be hard to get a new dump file.

[camainc]

Here is the second dump file. Thanks again for your help.

https://1drv.ms/u/s!AucEIaq2vxfqoeoHCkxn1TM11bGVAQ

[noahfalk]

@camainc thanks for providing these dumps. With some further investigating by my teammate we've likely connected your issue to a bug in the runtime that was discovered last year. We have implemented a fix that is included in the forthcoming .Net 4.8 release and we've got some options for you to try that will hopefully workaround the issue using the build of .NET you already have.

The primary goal of all these workarounds is to avoid using an NGEN image for netstandard.dll and instead to use the IL image. The options are different ways of achieving that and you can pick whichever one is easiest to use/affects the right scope of processes:

1. [only works for x64 apps] Set environment variable COMPLUS_DisableNativeImageLoadList=netstandard.
2. [only works for x64 apps] Set the registry variable: HKEY_CURRENT_USER\SOFTWARE\Microsoft.NETFramework\DisableNativeImageLoadList = netstandard
   This will affect all .NET processes that run under this user account.
3. [only works for x64 apps] Set the registry variable: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\DisableNativeImageLoadList = netstandard
   This will affect all .NET processes that run on the machine.
4. Add an app.exe.config file adjacent to app.exe (where 'app' is replaced by the name of the .NET exe you have). This affects only the application associated with this config file
   Then add the following content to the file:

    <configuration>
        <runtime>
          <disableNativeImageLoad>
            <assemblyIdentity name="netstandard" />
          </disableNativeImageLoad>
        </runtime>
    </configuration>

If there is an existing file with a <configuration>, <runtime> or <disableNativeImageLoad> block, then the elements described above need to be "spliced in" to the existing blocks (i.e., I believe the CLR config file parser might have trouble if multiple blocks of the same name are listed).

5. Add an entry to machine.config which will affect all .NET processes running on the machine. This is done by editing the relevant config file:
   For 32 bit apps: %SystemRoot%\Microsoft.NET\Framework\v4.0.30319\config\machine.config
   For 64 bit apps: %SystemRoot%\Microsoft.NET\Framework64\v4.0.30319\config\machine.config

Before editing it is a good precaution to save a copy of the existing file in case you need to roll back the change for any reason, then make the same edit as used for the app.exe.config case above.

I hope one of these options can you get debugging successfully and sorry for the trouble this has caused.

-Noah

[camainc]

Thank you so much for your excellent work and the detailed workarounds.

Am I safe to assume that this bug only affects debugging, and won't be a problem in a release version? We haven't deployed this yet, but I don't know when the IT folks will get around to upgrading to version 4.8. We had to push a bit to get them to allow us to use 4.6.2.

Thank you again! I will try your suggested workarounds and see what happens.

[camainc]

I think dotnet/coreclr#5 has fixed this for me. Thank you!

[noahfalk]

Am I safe to assume that this bug only affects debugging?

As best I can tell there actually wasn't anything debugger specific about the bug, but its always possible there was a connection there I missed. I would suggest that if you are already running this in production and see no issue, don't fix what isn't broken. However if you haven't deployed this to production yet it might be worth doing one of these workarounds to head off potential trouble. All of these workarounds should be safe for production usage because NGEN images have always been optional. Depending on your application you might see slight differences in performance numbers such as startup time or working set. If those stats have high importance for you can always measure to determine the exact impact.

Sorry on the slow response, I broke my github notifications for a bit there. Glad to hear you were able to get things working though!

[tommcdon]

Closing this issue as it seems resolved.

[camainc]

Thanks for the awesome help. I would never have figured this out on my own, and neither StackOverflow or Google was any help with this. Being able to get a response directly from the developers is GOLDEN. You guys rock.

[pfaustinopt]

However if you haven't deployed this to production yet it might be worth doing one of these workarounds to head off potential trouble

How can I accomplish this if my application is deployed to the client environment? I was moving my code from EF6 to EF Core but this error started to occur (I'm also using Autofac). Should I keep using EF6 until .net 4.8 is released?

Btw, this seems to have fixed the problem:

Add an entry to machine.config which will affect all .NET processes running on the machine. This is done by editing the relevant config file:
For 32 bit apps: %SystemRoot%\Microsoft.NET\Framework\v4.0.30319\config\machine.config
For 64 bit apps: %SystemRoot%\Microsoft.NET\Framework64\v4.0.30319\config\machine.config

[noahfalk]

How can I accomplish this if my application is deployed to the client environment?

It really depends what control you have over that client environment. For example in the likely case that you make an installer that contains your app and it copies files to the client then you could probably do option 4 and include the config file as part of your app. Does that answer what you were looking for or was there some other issue you forsee?

[pfaustinopt]

Option 4 will work on both x86 and x64? A sexta, 24/05/2019, 22:59, Noah Falk <notifications@github.com> escreveu:

…

How can I accomplish this if my application is deployed to the client environment? It really depends what control you have over that client environment. For example in the likely case that you make an installer that contains your app and it copies files to the client then you could probably do option 4 and include the config file as part of your app. Does that answer what you were looking for or was there some other issue you forsee? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/dotnet/coreclr/issues/22137?email_source=notifications&email_token=ABJXYAEQ6NCMZGPNSNXN4IDPXBQNNA5CNFSM4GRXBBU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWGUWCA#issuecomment-495799048>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJXYABVIPU6OA476XAIYYLPXBQNNANCNFSM4GRXBBUQ> .

[noahfalk]

Option 4 will work on both x86 and x64?

Yep : )

[pfaustinopt]

Then that option is valid. I'll give it a try and give you feedback. Thanks for the help! A sexta, 24/05/2019, 23:06, Noah Falk <notifications@github.com> escreveu:

…

Option 4 will work on both x86 and x64? Yep : ) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/dotnet/coreclr/issues/22137?email_source=notifications&email_token=ABJXYAELP4RCYKFG54X2Q43PXBRE3A5CNFSM4GRXBBU2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWGVAFY#issuecomment-495800343>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJXYAARYMZZA5G46RE3SZ3PXBRE3ANCNFSM4GRXBBUQ> .

[pfaustinopt]

Where do I add <disableNativeImageLoad> on a web application? The machine.config method solved the problem, but I can't manipulate that file on all my clients.

Any ideas to solve this issue? Btw, is this related directly to EF Core? If I change back to EF6 this problem will go away?