tailcall regression with compiled F#

[brettfo]

Machine:
Latest Windows 10 x64

Pre-verification:

1. On a clean machine install latest VS with the following workloads:
   • .NET desktop development
     • Add optional component F# desktop language support
   • .NET Core cross-platform development
2. Clone https://github.com/dotnet/fsharp and reset to commit 1c1b5ac7eacbbfd79e7277982e15178cecee20b4
3. Build with .\Build.cmd -c Release
4. Run test with dotnet test tests\fsharp\FSharpSuite.Tests.fsproj --no-restore --no-build --filter LargeRecordDoesNotStackOverflow -f netcoreapp3.1 -c Release -v n

At this point the test should pass.

Bug repro:

5. Install latest .NET 5 SDK from here.
6. Invalidate the .NET 3 SDK and runtime by doing the following:
   • Rename/delete C:\Program Files\dotnet\sdk\3.1.401
   • Rename/delete C:\Program Files\dotnet\shared\Microsoft.NETCore.App\3.1.7
   • Ensure they're not listed via dotnet --list-sdks and dotnet --list-runtimes
7. Re-run the test from step (4) exactly as above.
   N.b., you may need to alter global.json with the following:

    {
   +  "sdk": {
   +    "version": "5.0.100-preview.7.20366.6"
   +   },
      "tools": {
   -    "dotnet": "3.1.302",
   +    "dotnet": "5.0.100-preview.7.20366.6",
    ...

Result:
Stack overflow

Possibly related to #40581

[Dotnet-GitSync-Bot]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[AndyAyersMS]

Presume codegen for now, and marking as 5.0 pending investigation.

cc @dotnet/jit-contrib

[AndyAyersMS]

Going to take a preliminary look while waiting for other intermittent bugs to repro.

@dotnet/jit-contrib feel free to jump in and take this one.

[AndyAyersMS]

Managed stack bactrace at point of overflow...

00000AA618C5710 00007ffa1664ece3 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
000000AA618C5790 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA618C57E0 00007ffa166511ff ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>>>, Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>>>, Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>, Expr, TOp, Microsoft.FSharp.Collections.FSharpList`1, Microsoft.FSharp.Collections.FSharpList`1)
000000AA618C5830 00007ffa16651165 ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)
000000AA618C58C0 00007ffa156e4f06 ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)
000000AA618C5940 00007ffa1664f796 FSharp.Compiler.AutoBox.DecideExpr(cenv, Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>>>, Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>>>, Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1>, Expr) [C:\repos\fsharp\src\fsharp\autobox.fs @ 101]
000000AA618C5A40 00007ffa1664eec8 ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2>>>>, Microsoft.FSharp.Core.FSharpFunc`2, Microsoft.FSharp.Core.FSharpFunc`2, Internal.Utilities.Collections.Tagged.Set`2, Expr) [/_/src/libraries/System.Private.CoreLib/src/System/Object.cs @ 27]
000000AA618C5A80 00007ffa1664ee40 ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)
000000AA618C5AF0 00007ffa156e4f06 ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)
000000AA618C5B70 00007ffa1664ed17 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
000000AA618C5BF0 00007ffa16650380 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprNoInterceptF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6523]
000000AA618C5C70 00007ffa1664eec8 ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2>>>>, Microsoft.FSharp.Core.FSharpFunc`2, Microsoft.FSharp.Core.FSharpFunc`2, Internal.Utilities.Collections.Tagged.Set`2, Expr) [/_/src/libraries/System.Private.CoreLib/src/System/Object.cs @ 27]
000000AA618C5CB0 00007ffa1664ee40 ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)
000000AA618C5D20 00007ffa156e4f06 ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)
000000AA618C5DA0 00007ffa1664ed17 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
000000AA618C5E20 00007ffa1664eec8 ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2>>>>, Microsoft.FSharp.Core.FSharpFunc`2, Microsoft.FSharp.Core.FSharpFunc`2, Internal.Utilities.Collections.Tagged.Set`2, Expr) [/_/src/libraries/System.Private.CoreLib/src/System/Object.cs @ 27]
000000AA618C5E60 00007ffa1664ee40 ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)
000000AA618C5ED0 00007ffa156e4f06 ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)
000000AA618C5F50 00007ffa1664ed17 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
000000AA618C5FD0 00007ffa16650380 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprNoInterceptF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6523]
000000AA618C6050 00007ffa1664eec8 ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.FSharpFunc`2,Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2>>>>, Microsoft.FSharp.Core.FSharpFunc`2, Microsoft.FSharp.Core.FSharpFunc`2, Internal.Utilities.Collections.Tagged.Set`2, Expr) [/_/src/libraries/System.Private.CoreLib/src/System/Object.cs @ 27]
000000AA618C6090 00007ffa1664ee40 ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)
000000AA618C6100 00007ffa156e4f06 ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)
000000AA618C6180 00007ffa1664ed17 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
... (frames omitted) ...
000000AA61A3DF70 00007ffa1664ed17 FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr) [C:\repos\fsharp\src\fsharp\TypedTreeOps.fs @ 6511]
000000AA61A3DFF0 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA61A3E040 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA61A3E090 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA61A3E0E0 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA61A3E130 00007ffa1586f706 Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.__Canon>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 224]
000000AA61A3E180 00007ffa1664afbd FSharp.Compiler.AutoBox.TransformImplFile(TcGlobals, ImportMap, TypedImplFile) [C:\repos\fsharp\src\fsharp\autobox.fs @ 169]
000000AA61A3E220 00007ffa165d9ce3 FSharp.Compiler.CompileOptions+ApplyAllOptimizations@1726-1.Invoke(TypedImplFile) [C:\repos\fsharp\src\fsharp\CompileOptions.fs @ 1735]
000000AA61A3E460 00007ffa1599f99c Microsoft.FSharp.Primitives.Basics.List.mapFold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.Tuple`2<System.__Canon,System.__Canon>>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\local.fs @ 389]
000000AA61A3E580 00007ffa15e860ab Microsoft.FSharp.Collections.ListModule.MapFold[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,System.Tuple`2<System.__Canon,System.__Canon>>>, System.__Canon, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\list.fs @ 85]
000000AA61A3E5C0 00007ffa165d8d2b FSharp.Compiler.CompileOptions.ApplyAllOptimizations(TcConfig, TcGlobals, Microsoft.FSharp.Core.FSharpFunc`2<ValRef,Microsoft.FSharp.Core.FSharpFunc`2<ValUseFlag,Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Collections.FSharpList`1,Microsoft.FSharp.Core.FSharpFunc`2<range,System.Tuple`2<Expr,TType>>>>>, System.String, ImportMap, Boolean, IncrementalOptimizationEnv, CcuThunk, Microsoft.FSharp.Collections.FSharpList`1) [C:\repos\fsharp\src\fsharp\CompileOptions.fs @ 1723]
000000AA61A3E6E0 00007ffa15fda3e8 FSharp.Compiler.Driver.main2a[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Args`1<System.Tuple`8<System.__Canon,TcConfig,TcImports,TcImports,System.__Canon,ErrorLogger,CcuThunk,System.Tuple`8<System.String,Microsoft.FSharp.Collections.FSharpList`1,System.__Canon,System.__Canon,System.__Canon,System.__Canon,System.__Canon,System.Tuple`1>>>) [C:\repos\fsharp\src\fsharp\fsc.fs @ 2057]
000000AA61A3E950 00007ffa156db3a4 FSharp.Compiler.Driver.typecheckAndCompile(CompilationThreadToken, System.String[], Resolver, Boolean, ReduceMemoryFlag, CopyFSharpCoreFlag, Exiter, ErrorLoggerProvider, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<TcImports,Microsoft.FSharp.Core.Unit>>, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<System.Tuple`3<TcGlobals,System.String,ILModuleDef>,Microsoft.FSharp.Core.Unit>>) [C:\repos\fsharp\src\fsharp\fsc.fs @ 2209]
000000AA61A3EA20 00007ffa156d9159 FSharp.Compiler.Driver.mainCompile(CompilationThreadToken, System.String[], Resolver, Boolean, ReduceMemoryFlag, CopyFSharpCoreFlag, Exiter, ErrorLoggerProvider, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<TcImports,Microsoft.FSharp.Core.Unit>>, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<System.Tuple`3<TcGlobals,System.String,ILModuleDef>,Microsoft.FSharp.Core.Unit>>) [C:\repos\fsharp\src\fsharp\fsc.fs @ 2227]
000000AA61A3EA80 00007ffa156d8e6b FSharp.Compiler.SourceCodeServices.CompileHelpers+result@168-10.Invoke(Exiter) [C:\repos\fsharp\src\fsharp\service\service.fs @ 169]
000000AA61A3EB10 00007ffa156d52d6 FSharp.Compiler.SourceCodeServices.CompileHelpers.tryCompile(ErrorLogger, Microsoft.FSharp.Core.FSharpFunc`2<Exiter,Microsoft.FSharp.Core.Unit>) [C:\repos\fsharp\src\fsharp\service\service.fs @ 157]
000000AA61A3EBE0 00007ffa156d49a8 FSharp.Compiler.SourceCodeServices.CompileHelpers.compileFromArgs(CompilationThreadToken, System.String[], Resolver, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<TcImports,Microsoft.FSharp.Core.Unit>>, Microsoft.FSharp.Core.FSharpOption`1<Microsoft.FSharp.Core.FSharpFunc`2<System.Tuple`3<TcGlobals,System.String,ILModuleDef>,Microsoft.FSharp.Core.Unit>>) [C:\repos\fsharp\src\fsharp\service\service.fs @ 167]
000000AA61A3EC70 00007ffa156d4394 <StartupCode$FSharp-Compiler-Private>.$Service+Compile@1053-1.Invoke(Microsoft.FSharp.Core.Unit) [C:\repos\fsharp\src\fsharp\service\service.fs @ 1053]
000000AA61A3ECB0 00007ffa156d4318 FSharp.Compiler.AbstractIL.Internal.Library+CancellableModule+delay@741[[System.__Canon, System.Private.CoreLib]].Invoke(System.Threading.CancellationToken) [C:\repos\fsharp\src\absil\illib.fs @ 741]
000000AA61A3ECE0 00007ffa156d3b78 <StartupCode$FSharp-Compiler-Private>.$Reactor+EnqueueAndAwaitOpAsync@185-2[[System.__Canon, System.Private.CoreLib]].Invoke(CompilationThreadToken) [C:\repos\fsharp\src\fsharp\service\Reactor.fs @ 188]
000000AA61A3EE10 00007ffa156d2e48 <StartupCode$FSharp-Compiler-Private>.$Reactor+loop@71-115.Invoke(Microsoft.FSharp.Core.FSharpOption`1<FSharp.Compiler.SourceCodeServices.ReactorCommands>) [C:\repos\fsharp\src\fsharp\service\Reactor.fs @ 82]
000000AA61A3EED0 00007ffa156d021f Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvokeNoHijackCheck[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](Microsoft.FSharp.Control.AsyncActivation`1<System.__Canon>, Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,Microsoft.FSharp.Control.FSharpAsync`1<System.__Canon>>, System.__Canon) [C:\repos\fsharp\src\fsharp\FSharp.Core\async.fs @ 411]
000000AA61A3EF20 00007ffa156d037d <StartupCode$FSharp-Compiler-Private>.$Reactor+loop@61-113.Invoke(Microsoft.FSharp.Control.AsyncActivation`1<Microsoft.FSharp.Core.FSharpOption`1<FSharp.Compiler.SourceCodeServices.ReactorCommands>>) [C:\repos\fsharp\src\fsharp\service\Reactor.fs @ 61]
000000AA61A3EF50 00007ffa156cf504 <StartupCode$FSharp-Core>.$Mailbox+processFirstArrival@303-8[[System.__Canon, System.Private.CoreLib]].Invoke(Microsoft.FSharp.Control.AsyncActivation`1<System.__Canon>) [C:\repos\fsharp\src\fsharp\FSharp.Core\mailbox.fs @ 303]
000000AA61A3EFA0 00007ffa1529a364 Microsoft.FSharp.Control.Trampoline.Execute(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.AsyncReturn>) [C:\repos\fsharp\src\fsharp\FSharp.Core\async.fs @ 104]
000000AA61A3F000 00007ffa15299c00 Microsoft.FSharp.Control.TrampolineHolder.ExecuteWithTrampoline(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.AsyncReturn>) [C:\repos\fsharp\src\fsharp\FSharp.Core\async.fs @ 169]
000000AA61A3F060 00007ffa15299636 <StartupCode$FSharp-Core>.$Async+-ctor@155-1.Invoke(System.Object) [C:\repos\fsharp\src\fsharp\FSharp.Core\async.fs @ 157]
000000AA61A3F0B0 00007ffa731b6de9 System.Threading.QueueUserWorkItemCallback+c.<.cctor>b__6_0(System.Threading.QueueUserWorkItemCallback) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs @ 784]
000000AA61A3F0E0 00007ffa731ac88e System.Threading.ExecutionContext.RunForThreadPoolUnsafe[[System.__Canon, System.Private.CoreLib]](System.Threading.ExecutionContext, System.Action`1<System.__Canon>, System.__Canon ByRef) [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ExecutionContext.cs @ 332]
000000AA61A3F120 00007ffa731b6d2f System.Threading.QueueUserWorkItemCallback.Execute() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs @ 800]
000000AA61A3F160 00007ffa731b5d2c System.Threading.ThreadPoolWorkQueue.Dispatch() [/_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPool.cs @ 641]
000000AA61A3F210 00007ffa731a49aa System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() [/_/src/coreclr/src/System.Private.CoreLib/src/System/Threading/ThreadPool.CoreCLR.cs @ 29]
000000AA61A3F600 00007ffa744ffed3 [DebuggerU2MCatchHandlerFrame: 000000aa61a3f600] 

[AndyAyersMS]

Something goes wrong in the new tail call helpers and we end up not reusing stack frames.

cc @jakobbotsch @erozenfeld

[jakobbotsch]

I cannot quite see what segment of the stack is looping in your backtrace. However, it looks like the instantiating stub generated and used by the JIT here also needs to do a tailcall. Since it is leaving a normal frame on the stack the tailcalling mechanism is probably not kicking in. We might be missing tests for this scenario, though it is unclear to me, there are many generic tests around here, and some of them should involve instantiating stubs:

runtime/src/tests/JIT/Directed/tailcall/more_tailcalls.cs

Lines 616 to 626 in f35d747

	[MethodImpl(MethodImplOptions.NoInlining)]
	private static string GenInterfaceForwardG<T1, T2>(T1 a, T2 b, IGenInterface<T1, T2> igen)
	{
	IL.Push(igen);
	IL.Push(new S32());
	IL.Push(a);
	IL.Push(b);
	IL.Emit.Tail();
	IL.Emit.Callvirt(new MethodRef(typeof(IGenInterface<T1, T2>), nameof(IGenInterface<T1, T2>.G)));
	return IL.Return<string>();
	}

I'm not sure how this is normally handled, to me it seems like tail-calling a pointer produced by loadvirtftn will not guarantee that the stack does not grow since that pointer could be an instantiating stub.

[jkotas]

The instantiating stubs for trivial signatures (e.g. where all arguments fit into registers) use regular tail calls. The instantiating stubs for more complex signatures do not tailcall. It is what's causing the problem.

[AndyAyersMS]

Here's the top of the native stack, frame 0a is one of those complex instantiating stubs

00 00007ffa`743e86b1     : 00000000`00000000 00007ffa`743e8461 00007ffa`15a566f0 00000000`00000000 : coreclr!SigPointer::GetTypeHandleThrowing+0x30 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 993] 
01 00007ffa`743e86b1     : 00000000`00000000 00007ffa`00000000 00000000`ffffffe7 00007ffa`15a566f0 : coreclr!SigPointer::GetTypeHandleThrowing+0xde1 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 1379] 
02 00007ffa`743e86b1     : 00000000`00000000 00007ffa`743e8461 00007ffa`15a566f0 00000000`02000076 : coreclr!SigPointer::GetTypeHandleThrowing+0xde1 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 1379] 
03 00007ffa`7444a72b     : 00000000`00000000 00000000`00000003 00007ffa`00000000 000000aa`618c5640 : coreclr!SigPointer::GetTypeHandleThrowing+0xde1 [F:\workspace\_work\1\s\src\coreclr\src\vm\siginfo.cpp @ 1379] 
04 00007ffa`7444998e     : 0000025e`86367300 00000000`fffffffd 00007ffa`168dd020 00007ffa`743e7019 : coreclr!Dictionary::PopulateEntry+0xafb [F:\workspace\_work\1\s\src\coreclr\src\vm\genericdict.cpp @ 1197] 
05 (Inline Function)     : --------`-------- --------`-------- --------`-------- --------`-------- : coreclr!JIT_GenericHandleWorker+0x8e [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 3287] 
06 00007ffa`744491a8     : 0000025e`8930ffb0 00007ffa`00000006 00000000`00000018 00007ffa`1664e530 : coreclr!JIT_GenericHandle_Framed+0x1a6 [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 3351] 
07 00007ffa`1664ece3     : 000000aa`618c5530 00000000`00000003 0000025e`856a33c8 000000aa`618c5820 : coreclr!JIT_GenericHandleClass+0x58 [F:\workspace\_work\1\s\src\coreclr\src\vm\jithelpers.cpp @ 3448] 
08 00007ffa`1586f706     : 0000025e`8930ffb0 0000025e`8930ffd8 000000aa`618c5820 000000aa`618c59e0 : FSharp_Compiler_Private!FSharp.Compiler.TypedTreeOps+ExprFolders`1[[System.__Canon, System.Private.CoreLib]].exprF(System.__Canon, Expr)+0x43
09 00007ffa`166511ff     : 000000aa`618c5760 00000000`00000003 00007ffa`00000001 00007ffa`00000001 : FSharp_Core!Microsoft.FSharp.Collections.ListModule.Fold[[System.__Canon, System.Private.CoreLib],[FSharp.Compiler.Range+range, FSharp.Compiler.Private]](Microsoft.FSharp.Core.FSharpFunc`2<range,Microsoft.FSharp.Core.FSharpFunc`2<System.__Canon,range>>, range, Microsoft.FSharp.Collections.FSharpList`1<System.__Canon>)+0xffffffff`ffee1aa6
0a 00007ffa`16651165     : 00000001`168cc2e8 00000001`00000000 00001653`092758be 00000001`168edc58 : FSharp_Compiler_Private!ILStubClass.IL_STUB_InstantiatingStub(Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1<Val>>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1<Val>>>>, Microsoft.FSharp.Core.FSharpFunc`2<Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1<Val>>,Microsoft.FSharp.Core.FSharpFunc`2<Expr,Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1<Val>>>>, Internal.Utilities.Collections.Tagged.Set`2<Val,System.Collections.Generic.IComparer`1<Val>>, Expr, TOp, Microsoft.FSharp.Collections.FSharpList`1<TType>, Microsoft.FSharp.Collections.FSharpList`1<Expr>)+0x4f
0b 00007ffa`156e4f06     : 0000025e`8930ffd8 ffffffff`00000001 0000025e`00000001 00007ffa`1664e530 : System_Diagnostics_Process!ILStubClass.IL_STUB_CallTailCallTarget(IntPtr, IntPtr, IntPtr)+0x75
0c 00007ffa`1664f796     : 000000aa`618c5760 00007ffa`168edcb8 00000000`00000000 00007ffa`168dd020 : System_Private_CoreLib!ILStubClass.IL_STUB_DispatchTailCalls(IntPtr, IntPtr, IntPtr)+0x76

and the code for this stub:

00007ffa`166511b0 55                   push    rbp
00007ffa`166511b1 4883ec40             sub     rsp, 40h
00007ffa`166511b5 488d6c2440           lea     rbp, [rsp+40h]
00007ffa`166511ba 488bc2               mov     rax, rdx
00007ffa`166511bd 4d8bd0               mov     r10, r8
00007ffa`166511c0 4c894c2420           mov     qword ptr [rsp+20h], r9
00007ffa`166511c5 488b5530             mov     rdx, qword ptr [rbp+30h]
00007ffa`166511c9 4889542428           mov     qword ptr [rsp+28h], rdx
00007ffa`166511ce 488b5538             mov     rdx, qword ptr [rbp+38h]
00007ffa`166511d2 4889542430           mov     qword ptr [rsp+30h], rdx
00007ffa`166511d7 488b5540             mov     rdx, qword ptr [rbp+40h]
00007ffa`166511db 4889542438           mov     qword ptr [rsp+38h], rdx
00007ffa`166511e0 488bd1               mov     rdx, rcx
00007ffa`166511e3 4c8bc0               mov     r8, rax
00007ffa`166511e6 4d8bca               mov     r9, r10
00007ffa`166511e9 48b9c8578e16fa7f0000 mov     rcx, 7FFA168E57C8h
00007ffa`166511f3 48b850e56416fa7f0000 mov     rax, offset CLRStub[MethodDescPrestub]@7ffa1664e550 (00007ffa`1664e550)
00007ffa`166511fd ffd0                 call    rax
00007ffa`166511ff 90                   nop     
00007ffa`16651200 488d6500             lea     rsp, [rbp]
00007ffa`16651204 5d                   pop     rbp
00007ffa`16651205 c3                   ret  

[AndyAyersMS]

So is the fix it as simple as emitting tail prefixes for the callis in CreateInstantiatingILStub and perhaps CreateUnboxingILStubForSharedGenericValueTypeMethods?

[jkotas]

It would introduce very significant performance regression for scenarios that happen to go through these instantiating stubs and do not care about tailcalls.

I think that the best way to solve this is by manually "inlining" the instantiating stub into the tailcall helper: Load address of the actual target + instantiating stub arg in the first tail call helper and match it in the second tail call helper. IIRC, @jakobbotsch tried this, but there were some difficulties with doing that.

Another way to solve this would be to create a tailcalling instantiating stubs (ie slow instantiating stubs that have the tail prefix before the call), but that feels too wide-spread.

[jakobbotsch]

I think that the best way to solve this is by manually "inlining" the instantiating stub into the tailcall helper: Load address of the actual target + instantiating stub arg in the first tail call helper and match it in the second tail call helper. IIRC, @jakobbotsch tried this, but there were some difficulties with doing that.

Related comment: dotnet/coreclr#26418 (comment)
I think at that time I had been through a few edge cases with VSDs and default interface methods and I was on the path of duplicating all the logic of ldvirtftn. I'm sure someone with more knowledge of the runtime/type system would not have significant problems doing this, however.

However doing this will still not fix the underlying issue, the symptom being that you cannot robustly tailcall a pointer returned by ldftn/ldvirtftn. Even with the suggested fix the F# example still breaks if the target method is first assigned to a delegate before being tail called (right?). But maybe this problem is academic, I suppose the old mechanisms did not handle it either.