StackOverflowException on large lambda, or large entrypoint, crashing VS2015, or fsc.exe commandline

[abelbraaksma]

Reporting this separately, as it is unrelated to the issue where I first reported it: #758.

Cause

Large lambda or a large entrypoint may result in a StackOverflowException reported by fsc.exe, or a crash of Visual Studio 2015. In both cases, attaching a debugger "after the fact" fails. To see the actual exception, you must attach the debugger earlier.

This may be a security issue, as it can bring down web sites that process F# code, or that use dynamic quotations.

Reproduction

To reproduce this issue, create any console application, in the main argv, copy the following lines approximately 400x (leading to a source file of some 800 lines). Depending on whether you run devenv.exe in safe mode or not, the actual crash may happen earlier or later:

let fu = 0
1 |> ignore

While it does matter what source code lines you use, these are the simplest lines I could come up with that causes this behavior. For instance, if you repeat the same line, or use structured code (i.e., many different, indented lines, variables etc), this error appears much later or not at all.

The crash occurs when the file is open. It doesn't always occur when it is auto-opened (single click), but always occurs when it is focus-opened (double-click in solution explorer).

Running fsc.exe from the command line also shows the same crash, though the number of lines is typically a bit higher (which makes sense, the stack is not eaten up by the Visual Studio environment).

Reproducing example solution

A minimal reproducing example solution, including a full dump of the stack trace as it happened from devenv.exe: FSharpBug#779.zip

Affected versions

I could reproduce the issue on:

• Visual Studio 2015 (didn't test other versions)
• F# 2.3
• F# 3.0
• F# 3.1
• F# 4.0
• .NET 2.0
• .NET 4.0
• .NET 4.5
• .NET 4.5.2
• .NET 4.6

More info

Head of the stack trace looks something like this:

FSharp.Core.dll!Microsoft.FSharp.Collections.FSharpList<System.Tuple<Microsoft.FSharp.Compiler.Tast.Typar, Microsoft.FSharp.Compiler.Tast.TType>>.Cons(head, tail) Unknown
Sharp.Core.dll!Microsoft.FSharp.Primitives.Basics.List.zipToFreshConsTail<Microsoft.FSharp.Compiler.Tast.Typar, Microsoft.FSharp.Compiler.Tast.TType>(cons, xs1, xs2) Unknown
FSharp.Core.dll!Microsoft.FSharp.Primitives.Basics.List.zip<Microsoft.FSharp.Compiler.Tast.Typar, Microsoft.FSharp.Compiler.Tast.TType>(xs1, xs2) Unknown
FSharp.LanguageService.Compiler.dll!Microsoft.FSharp.Compiler.Tastops.applyTyconAbbrev(abbrevTy, tycon, tyargs) Unknown

My guess would be that the (recursive) code that walks the F# source lines, in some situations, does not get TCO'ed, which would explain why, on some different source code lines, the crash does not occur.

[KevinRansom]

Ouch!!!! thank you for this bug report, just loading the solution in VS crashes for me.

Kevin

[abelbraaksma]

Yes, it's a nasty one.

Sometimes you won't even get the "Debug" option, but even when you do get it, VS2015, when opened, will open with an empty stacktrace and no breakpoint or assembler view. To catch it, you'll need to attach the debugger before the crash happens (I use a plugin for that which automatically selects a new devenv.exe, but I assume there are other ways of getting that done).

I'm actually not quite sure why the stack is invisible to the debugger, but I think it has something to do with the SOE triggering a FailFast which in turn won't allow the executable image to linger around in memory, waiting for the breakpoint?

[dsyme]

Fix for this linked above