LLVM generated code crashes if stack frame > 4k on MinGW64

[llvmbot]

Bugzilla Link	8919
Resolution	FIXED
Resolved on	Mar 28, 2011 00:26
Version	trunk
OS	Windows XP
Blocks	llvm/llvm-bugzilla-archive#9100
Attachments	Patch for fixing the above issue.
Reporter	LLVM Bugzilla Contributor
CC	@asl

Extended Description

LLVM incorrectly generates "_alloca" as the stack probing call. That works only on MinGW32. On 64-bit, the function to call is "__chkstk". I've attached a patch to fix it.

[llvmbot]

Applied r122934.
Thanks!

[asl]

We need to be extremely carefully here. __alloca and __chkstk have different semantics in general. Also, I believe mingw-w64 provided both __alloca and __chkstk.

[llvmbot]

I have to reopen this. Sreeram's patch would be effective only on TDM.
I think it would not be needed to revert r122934. Prior implementation (_alloca by %eax) was bogus, too.

in http://tdm-gcc.tdragon.net/development

TDM64 edition only: Includes a patch to remove a leading underscore from the 64-bit cygwin.asm symbols "__chkstk" and "_alloca", allowing them to be exported with the same rule as the 32-bit versions after underscoring rules are applied.

The symbol "__chkstk" is unique to tdm. gcc's cygwin.asm and mingw-w64-gcc (and I) assume "___chkstk".

ps. see also bug 8777 ;)

Anton,

Also, I believe mingw-w64 provided both __alloca and
__chkstk.

We may prefer ___chkstk on win64 rather than __alloca.
___chkstk does not modify %rcx %rdx %r8 and %r9. (__alloca does not)

Or shall we implement our chkstk? :D

[asl]

Also, I believe mingw-w64 provided both __alloca and
__chkstk.

We may prefer ___chkstk on win64 rather than __alloca.
___chkstk does not modify %rcx %rdx %r8 and %r9. (__alloca does not)

Or shall we implement our chkstk? :D
It's good question and honestly speaking I don't know yet. In theory I'd prefer the variant which clobbers less registers.

However, I'd really prefer __alloca simply because we won't get into trouble if we'd try to link stuff with VCPP runtime, where __chkstk only probes the stack, but does not allocate it.

[llvmbot]

However, I'd really prefer __alloca simply because we won't get into trouble if
we'd try to link stuff with VCPP runtime, where __chkstk only probes the stack,
but does not allocate it.

MSVC has "__chkstk"(2 underscores) and cygming-w64 has "___chkstk" (3 underscores).
We don't need to be worried symbol clash. ;)

I propose we may expand alloca(const) stuff to;

addq $4096-8,%rsp
pushq %rax
addq $xxx-8,%rsp
pushq %rax

I reminded now tdm-w64 has "__chkstk". ooooops!

[asl]

I propose we may expand alloca(const) stuff to;

addq $4096-8,%rsp
pushq %rax
addq $xxx-8,%rsp
pushq %rax
This is not enough. We have to touch every 4k page in correct order!

I reminded now tdm-w64 has "__chkstk". ooooops!
Exactly!

[llvmbot]

addq $4096-8,%rsp
pushq %rax
addq $xxx-8,%rsp
pushq %rax
This is not enough. We have to touch every 4k page in correct order!

Excuse me, I intended below;

loop to be expanded {
addq $4096-8,%rsp
pushq %rax
}
addq $reminder,%rsp
pushq %rax

Anyway it would be too far from practice :(

[llvmbot]

I've investigated this issue further.

Here is the full story:

LLVM generated code was crashing with large stack frames. I inspected
the generated code and compared it with what GCC generates for large
frames. Basically the only difference was that LLVM generated a call
to "_alloca", whereas GCC generated a call to "__chkstk". Replacing
"_alloca" with "__chkstk" fixed the problem for me. Problem solved,
and I fired off the patch.

My initial testing was performed with TDM's GCC 4.5.1.

After Mr.Takumi mentioned that mainline GCC was different from TDM's
distro, I tested with a binary from http://mingw-w64.sf.net. This was
GCC 4.5.2 (20101129). There was only one difference. Both _alloca and
__chkstk had an extra underscore in their names: __alloca and
___chkstk.

Later, I wanted to dig more into why LLVM's _alloca call didn't
work. A disassembly of _alloca confirmed that it too touched every 4k
page sequentially during allocation. It turns out, the problem was
that _alloca expects its argument in the ECX register, whereas LLVM
passed the argument in the EAX register. __chkstk worked because it
accepts its argument in the EAX register. The only difference between
__chkstk and _alloca appears to be the register in which the argument
is passed.

I compared the disassembly of _alloca/__chkstdk between TDM's GCC and
Mingw-w64's GCC. There were zero differences in the code. The only
difference is that Mingw-w64 has an extra underscore in the names.

For reference, I've included the disassembly of _alloca and __chkstk
below:

_alloca:
0x0000000000402a10 <+0>: mov %rcx,%rax
0x0000000000402a13 <+3>: add $0x7,%rax
0x0000000000402a17 <+7>: and $0xfffffffffffffff8,%rax
0x0000000000402a1b <+11>: pop %rcx
0x0000000000402a1c <+12>: pop %r10
0x0000000000402a1e <+14>: mov %rsp,%r10
0x0000000000402a21 <+17>: cmp $0x1000,%rax
0x0000000000402a27 <+23>: jb 0x402a42 <alloca+50>
0x0000000000402a29 <+25>: sub $0x1000,%r10
0x0000000000402a30 <+32>: orq $0x0,(%r10)
0x0000000000402a34 <+36>: sub $0x1000,%rax
0x0000000000402a3a <+42>: cmp $0x1000,%rax
0x0000000000402a40 <+48>: ja 0x402a29 <alloca+25>
0x0000000000402a42 <+50>: sub %rax,%r10
0x0000000000402a45 <+53>: orq $0x0,(%r10)
0x0000000000402a49 <+57>: mov %r10,%rax
0x0000000000402a4c <+60>: sub $0x8,%r10
0x0000000000402a50 <+64>: mov %r10,%rsp
0x0000000000402a53 <+67>: push %rcx
0x0000000000402a54 <+68>: retq

__chkstk:
0x0000000000402a55 <+0>: add $0x7,%rax
0x0000000000402a59 <+4>: and $0xfffffffffffffff8,%rax
0x0000000000402a5d <+8>: pop %r11
0x0000000000402a5f <+10>: mov %rsp,%r10
0x0000000000402a62 <+13>: cmp $0x1000,%rax
0x0000000000402a68 <+19>: jb 0x402a83 <_chkstk+46>
0x0000000000402a6a <+21>: sub $0x1000,%r10
0x0000000000402a71 <+28>: orl $0x0,(%r10)
0x0000000000402a75 <+32>: sub $0x1000,%rax
0x0000000000402a7b <+38>: cmp $0x1000,%rax
0x0000000000402a81 <+44>: ja 0x402a6a <_chkstk+21>
0x0000000000402a83 <+46>: sub %rax,%r10
0x0000000000402a86 <+49>: orl $0x0,(%r10)
0x0000000000402a8a <+53>: mov %r10,%rsp
0x0000000000402a8d <+56>: push %r11
0x0000000000402a8f <+58>: retq

[llvmbot]

Fixed in r128206.

[llvmbot]

mentioned in issue llvm/llvm-bugzilla-archive#9100