[cDAC] X86 support TailCallFrame

[max-charlamb]

Contributes to #114019

TailCallFrame are exclusive to Windows x86, add support for handling them in the cDAC. TailCallFrames are only generated when tail calls to VSD methods.

Verified using the following F# script to generate a stack trace with a TailCallFrame.

open System

type ITest =
    [<TailCall>]
    abstract Even: n:int -> bool
    [<TailCall>]
    abstract Odd: n:int -> bool

type Test() =
    interface ITest with
        [<TailCall>]
        member this.Odd (n: int): bool = 
            if n = 0 then 
                Console.ReadLine() |> Console.Write 
                false 
            else (this :> ITest).Even(n - 1)
        [<TailCall>]
        member this.Even (n: int): bool = 
            if n = 0 then 
                Console.ReadLine() |> Console.Write 
                false 
            else (this :> ITest).Odd(n - 1) 
            
let t = Test()
let b = (t :> ITest).Even 11111
printfn "%b" b

[Copilot]

Pull Request Overview

This PR adds support for handling TailCallFrame in the cDAC on Windows x86 by introducing a new TailCallFrame type and integrating it into the stack walking and frame handling mechanism.

• Added TailCallFrame class with proper field calculations.
• Extended the IPlatformFrameHandler interface and updated X86FrameHandler and FrameIterator to process TailCallFrame.
• Updated native support via cdac_data in frames.h and the corresponding data descriptors in datadescriptor.h.

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
Data/Frames/TailCallFrame.cs	New TailCallFrame type that calculates CalleeSavedRegisters and ReturnAddress.
Contracts/StackWalk/FrameHandling/X86FrameHandler.cs	Implements HandleTailCallFrame, updating Eip/Esp and processing callee-saved registers.
Contracts/StackWalk/FrameHandling/IPlatformFrameHandler.cs	Extends the interface to include TailCallFrame handling.
Contracts/StackWalk/FrameHandling/FrameIterator.cs	Adds TailCallFrame support in the frame iteration logic.
Contracts/StackWalk/FrameHandling/BaseFrameHandler.cs	Provides default (exception-throwing) implementation for TailCallFrame.
Abstractions/DataType.cs	Updates enum to include TailCallFrame.
vm/frames.h	Adds native support by specializing cdac_data for TailCallFrame.
debug/runtimeinfo/datadescriptor.h	Registers TailCallFrame with proper conditional compilation for Windows x86.

Comments suppressed due to low confidence (3)

src/native/managed/cdac/Microsoft.Diagnostics.DataContractReader.Contracts/Contracts/StackWalk/FrameHandling/X86FrameHandler.cs:38

• Add tests to verify that the TailCallFrame handling correctly updates Eip and Esp based on the frame data on Windows x86.

        if (_target.GetTypeInfo(DataType.TailCallFrame).Size is not uint tailCallFrameSize)

src/native/managed/cdac/Microsoft.Diagnostics.DataContractReader.Contracts/Contracts/StackWalk/FrameHandling/BaseFrameHandler.cs:75

• [nitpick] Consider overriding HandleTailCallFrame in platform-specific frame handlers to avoid hitting the default exception, or update the documentation to clarify that the default implementation is intentional for unsupported platforms.

    public virtual void HandleTailCallFrame(TailCallFrame tailCallFrame)

src/native/managed/cdac/Microsoft.Diagnostics.DataContractReader.Contracts/Data/Frames/TailCallFrame.cs:1

• [nitpick] Consider adding a comment indicating that TailCallFrame is exclusive to Windows x86 to aid future maintainability.

// Licensed to the .NET Foundation under one or more agreements.

[dotnet-policy-service]

Tagging subscribers to this area: @steveisok, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

[jkotas]

What would it take to switch x86 tailcalls to work the same way as other archs so that we do not need this special handling?

[max-charlamb]

@jkotas

What would it take to switch x86 tailcalls to work the same way as other archs so that we do not need this special handling?

runtime/src/coreclr/vm/frames.h

Lines 2290 to 2307 in 9fe86ca

	//------------------------------------------------------------------------
	// This frame is used as padding for virtual stub dispatch tailcalls.
	// When A calls B via virtual stub dispatch, the stub dispatch stub resolves
	// the target code for B and jumps to it. If A wants to do a tail call,
	// it does not get a chance to unwind its frame since the virtual stub dispatch
	// stub is not set up to return the address of the target code (rather
	// than just jumping to it).
	// To do a tail call, A calls JIT_TailCall, which unwinds A's frame
	// and sets up a TailCallFrame. It then calls the stub dispatch stub
	// which disassembles the caller (JIT_TailCall, in this case) to get some information,
	// resolves the target code for B, and then jumps to B.
	// If B also does a virtual stub dispatch tail call, then we reuse the
	// existing TailCallFrame instead of setting up a second one.
	//
	// We could eliminate TailCallFrame if we factor the VSD stub to return
	// the target code address. This is currently not a very important scenario
	// as tail calls on interface calls are uncommon.
	//------------------------------------------------------------------------

It appears that this is where we take the JIT_TailCall path.

runtime/src/coreclr/jit/morph.cpp

Lines 4614 to 4619 in 9fe86ca

	// On x86 we have a faster mechanism than the general one which we use
	// in almost all cases. See fgCanTailCallViaJitHelper for more information.
	if (fgCanTailCallViaJitHelper(call))
	{
	tailCallViaJitHelper = true;
	}

There seems two ways to remove the TailCallFrames

1. Refactor x86 VSD stubs to return the address rather than jumping directly to the code as the comment above suggests.
2. Remove the TailCallFrame 'optimization' on x86. From reading about the pr it appears this could probably be removed. We would fall back to the standard path using the emitted stubs. I'm unsure on the performance hit but could look more into it.

[filipnavara]

What would it take to switch x86 tailcalls to work the same way as other archs so that we do not need this special handling?

It's a big chunk of work. We don't have fast tailcalls for x86 JIT. I resurrected @jakobbotsch's branch and updated it but it still had too many unhandled edge cases and unoptimal codegen (https://github.com/filipnavara/runtime/tree/FEATURE_INSTANTIATINGSTUB_AS_IL). I may give it another go at some point but even if I manage to get it working the risk of the change is high.

[filipnavara]

For posterity, I wrote a comment why I gave up doing the x86 fast tailcalls last time: #116331 (comment)

[jakobbotsch]

We can switch explicit tailcalls to the portable tailcall mechanism easily. It works fine for x86, it is just a bit slower compared to the existing optimized helper-based path. Here are the old measurements:
dotnet/coreclr#26418 (comment)

We could also switch only VSD tailcalls to the portable mechanism if that gets us most of the way of simplifying this. We already switched delegates like this in #70269.