Need calling convention for stdcall methods (different from functions!)

[retep998]

The crux of the matter is that stdcall methods on Windows using msvc handle struct returns differently than stdcall functions. This is particularly noticeable in COM, where although most of COM doesn't use struct returns, a few of them do, causing incorrect behavior when you attempt to call them from Rust (or even C for the matter).

Thus I propose that we add a new calling convention specifically for stdcall methods. When faced with this new calling convention Rust should do what Clang does:

(2:07:35 PM) rnk: right, this is sret vs. this
(2:07:47 PM) rnk: clang handles this by swapping the order of the arguments
(2:07:54 PM) rnk: and moving the sret attribute to the second parameter

Relevant discussion in #LLVM https://gist.github.com/retep998/604861ea06aa984ee6c7
Relevant discussion on discourse https://internals.rust-lang.org/t/need-custom-calling-convention-for-com/2389

gist demonstrating the calling convention difference: https://gist.github.com/retep998/9503145841a61551d3c6

[eefriedman]

extern "stdcall_method" and friends seems reasonable to me.

[mzabaluev]

Notably, the MIDL compiler seems to have no problem annotating __stdcall in the plain-C bindings for COM vtables. If it turns out that on -msvc targets rustc or LLVM does not do what MSVC does for any C function, then it's simply a matter of fixing "stdcall".

[retep998]

MSVC for C actually suffers from the same issue. I tested it myself. Functions in COM that return structs have incorrect behavior when called from the official Microsoft C bindings using the MSVC compiler. I'm assuming nobody at Microsoft noticed or cared about this since very few people use C when working with COM.

[DemiMarie]

What about thiscall? That is what LLVM calls it I believe.

[retep998]

@DemiMarie thiscall is not the same as an stdcall method.

[retep998]

It turned out that the workaround I use for this in winapi was actually slightly wrong. When the function takes parameters beyond this, those parameters need to go after the return value parameter. This just goes to show how important language support for this is, because it would remove the need to implement hacky workarounds that might be broken.
retep998/winapi-rs@6754bdd

[Bastacyclop]

Hi everyone, any updates on this? Still not sure how to fix Bastacyclop/rust_box2d#3 . Do I need to transform Struct f(); into void f(Struct* ret); on the C side ?