[RFC] Add support for Fastcall function pointers

[frabert]

Fixes #113851

I was trying to figure out how to add support to calling Fastcall function pointers, and it appears to me that nothing is actually missing...?

Please do comment on all the obvious issues I missed, I am not familiar with the codebase at all.
If there is a chance of this actually becoming a viable PR, I am open to cleaning up and adding tests.

[Copilot]

Copilot wasn't able to review any files in this pull request.

Files not reviewed (2)

• src/coreclr/jit/importercalls.cpp: Language not supported
• src/coreclr/vm/dllimport.cpp: Language not supported

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[frabert]

@dotnet-policy-service agree

[jkotas]

What do you need the Windows x86 fascall calling convention for?

Fascall is 32-bit window x86 specific calling convention. Windows x86 is a legacy platform that we are not very interested in improving and adding new features to.

Please do comment on all the obvious issues I missed,

The obvious issue is that it has zero testing. Once you add comprehensive tests, you are very likely going to run into actual issues that would need to be fixed. cc @dotnet/jit-contrib Anything known?

[frabert]

What do you need the Windows x86 fastcall calling convention for?

The short of it is that I'm embedding CoreCLR and want to use it to detour various functions, some of which use the fastcall calling convention. Due to the nature of the application, I have no say in what architecture (x86) or calling convention (Fastcall) I can use.

it has zero testing

Noted! I have done some very preliminary testing locally, but without knowing the internal details of the JIT engine it's hard for me to know where the corner cases are, and I didn't feel like spending a bunch of time investigating this further only to then discover this is not going anywhere...

[jakobbotsch]

__fastcall happens to match the managed calling convention in large parts, and the JIT falls back to the managed calling convention in this scenario. That's why it appears to work. But the calling conventions do differ in some key aspects, like handling of structs where the managed calling convention will enregister trivial wrapper structs, which fastcall should not be doing.

With that said I do not think it would be particularly hard to expand the support given recent work done in the JIT on ABI unification/clean up. It should just need some special handling in X86Classifier::Classify.

[jakobbotsch]

It also looks like __fastcall does not enregister the return buffer. That's also different from the managed calling convention.

[frabert]

Does the fact that [UnmanagedCallersOnly(CallConvs = [typeof(CallConvFastcall)])] already works without this PR change anything in this discussion?

[jakobbotsch]

Does the fact that [UnmanagedCallersOnly(CallConvs = [typeof(CallConvFastcall)])] already works without this PR change anything in this discussion?

It's most likely just an oversight that this does not throw some kind of compilation exception. If you try some of the things I pointed at above you will likely see incorrect results in interop with native code.

[AaronRobinsonMSFT]

@frabert I appreciate the desire to get this to work, but it isn't something that is currently on our roadmap. The x86 target isn't something we tend to invest heavily in and to properly add support would represent a fair bit of effort. I've placed the issue in "Future" and if there is broader interest from the community we can look at this. For now I'm going to close this PR as it isn't likely to move forward without team support and at present it isn't there.