Code generated for function returning struct incompatible with native ABI

[ZviRackover]

Bugzilla Link	9166
Resolution	FIXED
Resolved on	Feb 15, 2012 17:50
Version	trunk
OS	Windows XP
Attachments	Reproducer test, Fix Candidate
CC	@AaronBallman,@asl,@jckarter

Extended Description

The Win32 native compiler, VC, generates a call to a function returning a struct in the following manner:

1. The caller allocates a buffer which will be filled by the caller with the return value
2. Before calling, the caller pushes the address of this buffer on the stack
3. Unlike other targets, such as Linux32, the callee does not pop the address before returning

Currently, the Win32 target does not follow the above rules. This becomes an issue when native-compiled and LLVM-compiled codes are mixed.

[asl]

How does mingw gcc behave in this case?

[ZviRackover]

Fix Candidate
This patch would brake Mingw which does not behave the same as Win32 native.

[ZviRackover]

Good catch. Mingw does not behave the same as Win32 native. I will post a revised patch.

How does mingw gcc behave in this case?

[ZviRackover]

Fix Candidate
Fix candidate. This version preserves correctness for the Mingw target.

[AaronBallman]

Unfortunately, it's not quite that simple. MS doesn't document their ABI thoroughly, but I did a lot of research into this topic and what I found was that the return mechanism depends on the calling convention, the structure size and the structure packing.

For instance, a three byte structure returned from a cdecl function returns in EAX for all packing values except for 1, which then uses a hidden pointer in EAX cleaned up by the caller.

Also, "thiscall" is special in that it always uses the hidden parameter (cleaned by callee) trick, regardless of packing or structure size.

I'm exploring ways to patch this so we cover the entire ABI instead of just the common cases.

[AaronBallman]

*** Bug llvm/llvm-bugzilla-archive#11688 has been marked as a duplicate of this bug. ***

[jckarter]

I think this can be closed. I submitted a fix for this that was committed as r148992.

[AaronBallman]

I've been looking at the output from LLVM on ToT and can't seem to find a case that doesn't match MSVC 10. So I think you're right!