FatalExecutionEngineException when Seq.distinct is called on seq of KeyValuePair

[hrichardlee]

Environment:

• Windows 7 x64
• VS 2015RC installed
• HKLM/SOFTWARE/Microsoft/NET Framework Setup/NDP/v4/Full/Release = 393273

Repro steps:

1. Open VS 2015 RC, create a new F# Console app.
2. Edit the contents of Program.fs so that it looks like:

open System.Collections.Generic

[<EntryPoint>]
let main argv = 
    let r = [KeyValuePair("a", "b")] |> Seq.distinct |> Seq.toList
    0

1. Create an x64 configuration in Configuration Manager and make it active
2. Go into the project properties and change the Target F# runtime to F# 3.1 (FSharp.Core, 4.3.1.0). Leave the Target Framework as .NET Framework 4.5.2
3. Run under the debugger (exception also occurs when not running under the debugger, but it is harder to tell what is going on)

Expected result: Program runs and finishes successfully
Actual result: FatalExecutionEngineError was detected.

The runtime has encountered a fatal error. The address of the error was at 0xf7781d8c, on thread 0x4684. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-interop or PInvoke, which may corrupt the stack.

Severity: Seems pretty severe--existing F# code that currently targets F# 3.1 and .NET 4.5.1 (or 4.5 or 4.5.2) that currently works will not work when run on a machine that has .NET 4.6RC installed.

[latkin]

Nice find. I can repro on my machine.

[vladima]

Me too, result assembly passes PEVerify, looks like a JIT bug to me

0:000> k
 # Child-SP          RetAddr           Call Site
00 000000ca`e8d5e0d0 00007ff8`e2bb84b8 clr!VirtualCallStubManager::ResolveWorker+0x46 [f:\dd\ndp\clr\src\vm\virtualcallstub.cpp @ 1688]
01 000000ca`e8d5e390 00007ff8`e2b8220d clr!VSD_ResolveWorker+0x255 [f:\dd\ndp\clr\src\vm\virtualcallstub.cpp @ 1594]
02 000000ca`e8d5e530 00007ff8`e3009e71 clr!ResolveWorkerAsmStub+0x6d [f:\dd\ndp\clr\src\vm\amd64\VirtualCallStubAMD64.asm @ 36]
03 000000ca`e8d5e5e0 00007ff8`835a5a79 clr!JIT_TailCall+0x1f1 [f:\dd\ndp\clr\src\vm\jithelpers.cpp @ 6334]
04 000000ca`e8d5ec30 00007ff8`835a5618 FSharp_Core_caeb8d0000!Microsoft.FSharp.Core.CompilerServices.RuntimeHelpers+get_Comparer@531[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]], mscorlib]].System-Collections-Generic-IEqualityComparer`1-GetHashCode(StructBox`1<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>>)+0x89*** ERROR: Module load completed but symbols could not be loaded for FSharp.Core.dll

05 000000ca`e8d5eca0 00007ff8`835a41a7 mscorlib_ni!System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Insert(System.__Canon, System.__Canon, Boolean)+0xffffffff`a72aeaf8*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\3694cde256ec672c9f9380d530b2118d\mscorlib.ni.dll
 [f:\dd\ndp\clr\src\BCL\system\collections\generic\dictionary.cs @ 321]
06 000000ca`e8d5ed70 00007ff8`835a353b FSharp_Core_caeb8d0000!Microsoft.FSharp.Collections.SeqModule+Distinct@1300[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]], mscorlib]].GenerateNext(System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>> ByRef)+0x1d7
07 000000ca`e8d5ee10 00007ff8`835a2e01 FSharp_Core_caeb8d0000!Microsoft.FSharp.Core.CompilerServices.GeneratedSequenceBase`1[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]], mscorlib]].MoveNextImpl()+0x4b
08 000000ca`e8d5ee80 00007ff8`835a1a7b FSharp_Core_caeb8d0000!Microsoft.FSharp.Collections.SeqModule.ToList[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]], mscorlib]](System.Collections.Generic.IEnumerable`1<System.Collections.Generic.KeyValuePair`2<System.__Canon,System.__Canon>>)+0x131
09 000000ca`e8d5ef10 00007ff8`e2b838b3 ConsoleApplication2!Program.main(System.String[])+0xcb*** WARNING: Unable to verify checksum for ConsoleApplication2.exe
 [c:\users\vladima\documents\visual studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Program.fs @ 6]
0a 000000ca`e8d5ef70 00007ff8`e2b83775 clr!CallDescrWorkerInternal+0x83 [f:\dd\ndp\clr\src\vm\amd64\CallDescrWorkerAMD64.asm @ 97]
0b 000000ca`e8d5efb0 00007ff8`e2b83d26 clr!CallDescrWorkerWithHandler+0x4e [f:\dd\ndp\clr\src\vm\callhelpers.cpp @ 89]
0c 000000ca`e8d5eff0 00007ff8`e2c774c1 clr!MethodDescCallSite::CallTargetWorker+0xf8 [f:\dd\ndp\clr\src\vm\callhelpers.cpp @ 649]
0d (Inline Function) --------`-------- clr!MethodDescCallSite::Call_RetArgSlot+0x5 [f:\dd\ndp\clr\src\vm\callhelpers.h @ 423]
0e 000000ca`e8d5f0f0 00007ff8`e2c7765a clr!RunMain+0x1ee [f:\dd\ndp\clr\src\vm\assembly.cpp @ 2598]
0f 000000ca`e8d5f2d0 00007ff8`e2c7754a clr!Assembly::ExecuteMainMethod+0xb6 [f:\dd\ndp\clr\src\vm\assembly.cpp @ 2719]
10 000000ca`e8d5f5c0 00007ff8`e2c776f0 clr!SystemDomain::ExecuteMainMethod+0x5ea [f:\dd\ndp\clr\src\vm\appdomain.cpp @ 3738]
11 000000ca`e8d5fbf0 00007ff8`e2c777b6 clr!ExecuteEXE+0x3f [f:\dd\ndp\clr\src\vm\ceemain.cpp @ 3016]
12 000000ca`e8d5fc60 00007ff8`e2c899d0 clr!_CorExeMainInternal+0xb2 [f:\dd\ndp\clr\src\vm\ceemain.cpp @ 2850]
13 000000ca`e8d5fcf0 00007ff8`f0e89d21 clr!_CorExeMain+0x14 [f:\dd\ndp\clr\src\vm\ceemain.cpp @ 2777]
14 000000ca`e8d5fd30 00007ff8`f0f2ea5b mscoreei!_CorExeMain+0xe0 [f:\dd\ndp\clr\src\dlls\shim\shim.cpp @ 6420]
15 (Inline Function) --------`-------- MSCOREE!ShellShim__CorExeMain+0xc7 [d:\wbrtm\com\netfx\windowsbuilt\shell_shim\v2api.cpp @ 284]
16 000000ca`e8d5fd80 00007ff8`f67213d2 MSCOREE!_CorExeMain_Exported+0xcb [d:\wbrtm\com\netfx\windowsbuilt\shell_shim\v2api.cpp @ 1222]
17 000000ca`e8d5fdb0 00007ff8`f8ab5444 KERNEL32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78]
18 000000ca`e8d5fde0 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\9140\minkernel\ntdll\rtlstrt.c @ 1026]

[latkin]

We raised this to the JIT team, they acknowledged it's a bug and are looking into it.

[hrichardlee]

Awesome! Thanks!

[eriawan]

@latkin if this bug truly a JIT bug, then it's quite a showstopper bug.

I can reproduce and verify it on VS2015 RTM:

I hope it should be fixed soon. Any update/news when we will have the fix for this? And what will the update be? I'm quite curious, as AFAIK it's closely related to .NET runtime or even infrastructure.

[sivarv]

I have a fix for this and verified that the repro case works. I am currently further testing this.

[latkin]

@hrichardlee or anyone else who comes across this in the near-term -- please file a formal ticket at https://connect.microsoft.com/VisualStudio for this, too, marking it as a JIT bug. That will greatly increase the chances that this will be patched in the very near term, rather than rolled out with whatever the next scheduled release is (probably ~months). Connect leaves the paper trail needed in order to get a patch funded/supported, GitHub doesn't, unfortunately.

[hrichardlee]

Okay, I've posted it here: https://connect.microsoft.com/VisualStudio/feedback/details/1586892

[sivarv]

I am trying to understand the severity and scenario impact of this issue. It would be of great help.

[hrichardlee]

I believe the scenario is as follows:

• An F# developer writes an app targeting FSharp.Core 3.1 and .NET 4.5.2. In other words, this is anyone who is using VS 2013 (perhaps other previous versions have this issue, but I have not checked).
• In their code, they use Seq.distinct on a Seq of KeyValuePair.
• Their app, when it is deployed onto a machine that has .NET 4.6 installed, will throw a FatalExecutionEngineException
• If they recompile their app in VS 2015 without explicitly retargeting to FSharp.Core 4.3.1.0, it will throw a FatalExecutionEngineException

I think this is reasonably severe. Given that .NET 4.6 is an in-place update, this means that anyone who uses Seq.distinct on a seq of KeyValuePair cannot upgrade to .NET 4.6 or VS 2015 without either retargeting their projects or waiting for this bug to get fixed.

[hrichardlee]

Let me try to rephrase the impact one more time 😄 All current F# developers that have Seq.distinct on a Seq of KeyValuePair anywhere in their code must either control the environment that their code is deployed into so that .NET 4.6 is not installed OR they must immediately retarget all of their projects and immediately redeploy these projects.

[sivarv]

This F# issue is different from issue#1296. This issue is a case of a Virtual Stub Dispatch (VSD) call that requires runtime lookup for resolution being marked with ".tail" prefix. Further, this call cannot be dispatched as fast tail call (i.e. epilog+jmp). As a result, RyuJIT makes the call using TailCall helper. Since it is a VSD call CLR calling convention requires passing a secret parameter in R11 register - which is callee trashable. TailCall helper doesn't preserve R11 and hence the AV.

[forki]

So do you already have a fix?

[sivarv]

A fix to this issue was checked-in to dotnet/coreclr PR #1302. We recognize the urgency of this issue and working to release a fix for desktop users soon.

[dotnetchris]

Sorry to go off topic but i have to say this experience as a user is tremendous. Clearly acknowledging a problem, linking it through the work, allowing it to be tracked and code is merged within days to resolve the issue. This is the total antithesis of Microsoft Connect where we were collectively told to go pound sand anytime a bug was found.

I'm immensely happy this is the "new Microsoft"

[forki]

yes big kudos for doing this in the open.

[mjmckp]

Not sure if it's related, but I get the same type of exception occasionally creating values of type float option via reflection. See bug report and reproduction here: #566

[latkin]

FYI a patch has been released for .NET 4.6 which fixes this (and some other RyuJIT issues) https://support.microsoft.com/en-us/kb/3086251. It should have arrived via Windows Update this week.

[ghost]

@latkin, actually it is different (cumulative) KB for each operating system which bundles KB3086251. For example on Windows 10 it is KB3081436 and it was shipped on 11th August through Windows Update. On Vista, it is KB3083186 bundling it and so on. Search for phrase "Security update file names" on https://support.microsoft.com/en-us/kb/3086251 to find KB id for your OS. 😉

[hrichardlee]

Thanks!