Passing Procs with non-default calling conventions across libs

[HertzDevil]

A lib fun can take a @[CallConvention] annotation, which tells it to use a non-default calling convention, such as X86_StdCall. A lib itself can also take it, in which case all its funs will use that convention unless otherwise annotated. But this only concerns calling the funs themselves, and there is no such thing for fun parameters and return values. This means it is undefined behavior to deal with those functions across lib boundaries:

/* ext.c */
/* compile with `cl /c ext.c` inside the x64_x86 Cross Tools Command Prompt */

#include <stdint.h>

typedef int32_t (__stdcall *callback)(int32_t, int32_t, int32_t, int32_t, int32_t);

int32_t foo(callback cb) {
    return cb(1, 2, 3, 4, 5);
}

int32_t __stdcall f(int32_t a, int32_t b, int32_t c, int32_t d, int32_t e) {
    return a + b + c + d + e;
}

callback bar(void) {
    return &f;
}

# compile with `crystal build --target=i386-windows-msvc --prelude=empty --static`

class String
  def to_unsafe
    pointerof(@c)
  end
end

@[Link("legacy_stdio_definitions")]
lib LibC
  fun printf(fmt : UInt8*, ...) : Int32
  fun exit(status : Int32) : NoReturn
end

def raise(msg)
  LibC.printf("Unhandled exception: %s\n", msg)
  LibC.exit(1)
end

@[Link(ldflags: "#{__DIR__}/ext.obj")]
lib LibFoo
  alias Callback = Int32, Int32, Int32, Int32, Int32 -> Int32

  fun foo(cb : Callback) : Int32
  fun bar : Callback
end

# prints garbage
f = LibFoo.bar
LibC.printf("%d", f.call(1, 2, 3, 4, 5))

# Exception 0xc0000005 encountered at address 0x000002:
# User-mode data execution prevention (DEP) violation at location 0x00000002
f2 = ->(a : Int32, b : Int32, c : Int32, d : Int32, e : Int32) : Int32 { a &+ b &+ c &+ d &+ e }
LibC.printf("%d", LibFoo.foo(f2))

This directly blocks x86 Windows support, as the Win32 API uses X86_StdCall for everything, including callbacks: (for x86-64 this call convention is accepted and ignored, which is why stdlib doesn't need it currently)

/* consoleapi.h etc. */

#define WINAPI __stdcall
typedef BOOL (WINAPI *PHANDLER_ROUTINE)(DWORD CtrlType);

__declspec(dllimport) BOOL WINAPI SetConsoleCtrlHandler(PHANDLER_ROUTINE HandlerRoutine, BOOL Add);

There are no relevant non-default calling conventions for x86-64. (It seems MSVC has __vectorcall for both x86 and x86-64, but LLVM doesn't appear to implement the latter.) For other platforms there are AArch64_VectorCall and WASM_EmscriptenInvoke; @[CallConvention] doesn't support them yet.

To avoid attaching calling conventions to types themselves, we could extend the lib syntax to allow annotations for parameter and return types, similar to defs:

lib LibFoo
  fun foo(@[CallConvention("X86_StdCall")] cb : Callback) : Int32
  fun bar : @[CallConvention("X86_StdCall")] Callback
end

Though I don't really know how all of this could work at the codegen level without storing each Proc's calling convention dynamically.

[straight-shoota]

To avoid attaching calling conventions to types themselves

Why would you want to avoid that? In the C headers, its also attached to the function type, right? So I figure it would make sense to do the same in Crystal lib definition.
That means if you want to call the same function signature with different calling conventions, you'll need to define separate proc types for that. I don't see how this could be a problem, though.

[HertzDevil]

Maybe we could allow the alias to "carry" a @[CallConvention] annotation that is only meaningful in those lib fun contexts. What I mean is that IMO we should strive to avoid making Callback an entirely different Crystal type from Proc because that would be quite a significant breaking change.

[straight-shoota]

But if alias means more than just a different name, it would also be quite a change.

I think this needs to go on the Proc type itself. This could work based on just an annotation, meaning @[CallingConventionA] Proc(Int) is a different type from Proc(Int).

However, maybe an optional generic type argument would be better suited for that? Proc(Int, calling_convention: CallingConventionA)

[ysbaddaden]

I wondered "how does Rust do it" and they annotate the type declaration (system is stdcall on windows x86):

type PHANDLER_ROUTINE = Option<unsafe extern "system" fn(CtrlType: DWORD) -> BOOL>;
pub unsafe extern "system" fn SetConsoleCtrlHandler(HandlerRoutine: PHANDLER_ROUTINE, Add: BOOL) -> BOOL;

type PEVENT_CALLBACK = Option<unsafe extern "system" fn(pEvent: PEVENT_TRACE)>;
pub unsafe fn EventCallback(&self) -> &PEVENT_CALLBACK;

What's the issue with annotating the type?