Use rustc_target for function call ABI compatibility check

[RalfJung]

@eddyb suggests to use something in https://doc.rust-lang.org/nightly/nightly-rustc/rustc_target/abi/call/index.html instead of https://github.com/rust-lang/rust/blob/c610be92e2925ab8e789efc835c88247e46c6cbf/src/librustc_mir/interpret/terminator/mod.rs#L191.

IRC log:

<eddyb> RalfJ: yeah you can basically emulate ABIs!
<eddyb> RalfJ: the PassMode variant must always match. direct (including pairs) must have the same layout primitive, cast must use the same register "shape", and indirect has no requirements
<eddyb> RalfJ: for direct I doubt you can have anything other than scalars, scalar pairs and vectors, and for at least the first two you can just reuse the miri value and change the type
<eddyb> RalfJ: for cast & indirect, you basically need to do a memcpy/union type-cast
<eddyb> if you want you can check that the size matches *but that's not actually required* (in terms of things working, that is)
<eddyb> RalfJ: especially if we'll get virtual calls with by-value self, it's gonna be indirect on both sides but one of the sides might only know the zero-sized unsized `dyn Trait` type, whereas the other has the full type
<eddyb> RalfJ: now this is still somewhat imprecise

Unfortunately I have no idea yet what all of that means. ;)

[RalfJung]

@eddyb has dropped some more hints at https://botbot.me/mozilla/rustc/2018-07-14/?msg=102084910&page=1

[eddyb]

One annoyance is that there's some code stuck in rustc_codegen_llvm::abi that belongs somewhere in rustc::ty - I think rustc::ty::layout should be renamed to rustc::ty::abi and reexport + extend all of rustc_target::abi, but @irinagpopa has a WIP codegen PR I want to wait for first.

But once you have a FnType, you can just look at arguments/returns pairwise, and compare their PassModes for compatibility, instead of making assumptions based on type/layout.
This will even work across different extern "ABI"s, if the calling convention is equal, that is.

[RalfJung]

So you're saying I can ignore the sig.abi at that point and just compare the arguments? Nice!

[RalfJung]

@eddyb So how do I go from FnSig to FnType?

Also, I likely still have to look at sig.abi for the RustCall ABI, right? I will never understand why closures are so complex, vs. just being desugared to dyn Fn{,Mut,Once}...

[eddyb]

I will never understand why closures are so complex, vs. just being desugared to dyn Fn{,Mut,Once}...

Well, they're not dyn for performance reasons. And the hacks are in lieu of VG (variadic generics).

[eddyb]

So how do I go from FnSig to FnType?

I'd start by grepping for FnType::. Ping me on IRC or Discord if you want more details.

[RalfJung]

Details seem to be https://github.com/rust-lang/rust/blob/63d66494aff57411bfec1dd2a3a5f1af900feab6/src/librustc_codegen_llvm/abi.rs#L363-L379

(I'd like them here so I know where to look if I need them again)

[RalfJung]

This code has been overhauled by rust-lang/rust#53779 and is mostly sane now.