CRASH (win8 x64 any app) x64 stack is up high => dynamorio!__chkstk does wrong thing when looking at TEB.StackLimit

[derekbruening]

From bruen...@google.com on September 28, 2012 16:24:36

after fixing issue #902 I tried running win8 x64 hello.exe w/ -no_native_exec_syscalls (to work around issue #901 ) and I hit a crash at exit:

0:000> kn
*** Stack trace for last set context - .thread/.cxr resets it

Child-SP RetAddr Call Site

00 000000001b21d950 00000000151f9879 dynamorio!__chkstk+0x37 [f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\startup\amd64\chkstk.asm @ 117]
01 000000001b21d968 00000000151f9987 dynamorio!my_getenv+0x19 [c:\derek\dr\git\src\core\config.c @ 206]
02 000000001b21d970 00000000151f931b dynamorio!set_config_from_env+0x77 [c:\derek\dr\git\src\core\config.c @ 260]
03 000000001b21e1c0 00000000151f97c5 dynamorio!config_reread+0x32b [c:\derek\dr\git\src\core\config.c @ 558]
04 000000001b21e240 000000001500192a dynamorio!get_parameter_ex+0x35 [c:\derek\dr\git\src\core\config.c @ 693]
05 000000001b21e280 00000000150919a3 dynamorio!synchronize_dynamic_options+0x4fa [c:\derek\dr\git\src\core\options.c @ 1989]
06 000000001b21e360 0000000015352a45 dynamorio!dynamo_process_exit+0x73 [c:\derek\dr\git\src\core\dynamo.c @ 1259]
07 000000001b21e3d0 0000000000000005 dynamorio!cleanup_and_terminate+0x79 [C:\derek\dr\git\build_x64_dbg_tests\core\CMakeFiles\dynamorio.dir\x86\x86.asm.obj.s @ 1283]
08 000000001b21e3d8 0000000015352a2b 0x5
09 000000001b21e3e0 000000001b1a3700 dynamorio!cleanup_and_terminate+0x5f [C:\derek\dr\git\build_x64_dbg_tests\core\CMakeFiles\dynamorio.dir\x86\x86.asm.obj.s @ 1274]
0a 000000001b21e3e8 0000001bc251fdf0 0x1b1a3700
0b 000000001b21e3f0 0000000015352ad6 0x0000001bc251fdf0 0c 000000001b21e3f8 000000001b21f000 dynamorio!global_do_syscall_syscall [C:\derek\dr\git\build_x64_dbg_tests\core\CMakeFiles\dynamorio.dir\x86\x86.asm.obj.s @ 1486] 0d 000000001b21e400 00000000`00000000 0x1b21f000

0:000> U my_getenv
dynamorio!my_getenv [c:\derek\dr\git\src\core\config.c @ 206]:
00000000151f9860 4c89442418 mov qword ptr [rsp+18h], r8 00000000151f9865 4889542410 mov qword ptr [rsp+10h],rdx
00000000151f986a 48894c2408 mov qword ptr [rsp+8],rcx 00000000151f986f b828100000 mov eax,1028h
00000000151f9874 e8d7182200 call dynamorio!__chkstk (000000001541b150)
00000000151f9879 482be0 sub rsp,rax 0:000> !vprot 1b21c928 BaseAddress: 000000001b21c000 AllocationBase: 000000001b150000 AllocationProtect: 00000001 PAGE_NOACCESS RegionSize: 0000000000003000 State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE 0:000> Uf dynamorio!__chkstk dynamorio!__chkstk [f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\startup\amd64\chkstk.asm @ 75]: 75 000000001541b150 4883ec10 sub rsp,10h
91 000000001541b154 4c891424 mov qword ptr [rsp], r10 92 000000001541b158 4c895c2408 mov qword ptr [rsp+8], r11 94 000000001541b15d 4d33db xor r11 , r11 95 000000001541b160 4c8d542418 lea r10 ,[rsp+18h]
96 000000001541b165 4c2bd0 sub r10 ,rax 97 000000001541b168 4d0f42d3 cmovb r10 , r11 105 000000001541b16c 654c8b1c2510000000 mov r11 ,qword ptr gs:[10h] 106 000000001541b175 4d3bd3 cmp r10 , r11 107 000000001541b178 7316 jae dynamorio!__chkstk+0x40 (000000001541b190)

dynamorio!__chkstk+0x2a [f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\startup\amd64\chkstk.asm @ 115]:
115 00000000`1541b17a 664181e200f0 and r10w,0F000h

dynamorio!__chkstk+0x30 [f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\startup\amd64\chkstk.asm @ 116]:
116 000000001541b180 4d8d9b00f0ffff lea r11 ,[ r11 -1000h] 117 000000001541b187 41c60300 mov byte ptr [ r11 ],0
118 000000001541b18b 4d3bd3 cmp r10 , r11 119 000000001541b18e 75f0 jne dynamorio!__chkstk+0x30 (00000000`1541b180)

dynamorio!__chkstk+0x40 [f:\dd\vctools\crt_bld\SELF_64_amd64\crt\prebuild\startup\amd64\chkstk.asm @ 120]:
120 000000001541b190 4c8b1424 mov r10 ,qword ptr [rsp] 121 000000001541b194 4c8b5c2408 mov r11 ,qword ptr [rsp+8]
122 000000001541b199 4883c410 add rsp,10h 123 000000001541b19d c3 ret
0:000> r
Last set context:
rax=0000000000001028 rbx=000000001b21f000 rcx=0000000015461af8
rdx=000000001b21d9b0 rsi=0000000000000001 rdi=000000001b1a3700
rip=000000001541b187 rsp=000000001b21d950 rbp=000000001b21e408 r8 =0000000000000800 r9 =0000000000000001 r10 =000000001b21c000 r11 =0000001bc2423000 r12 =0000001bc2647f90 r13 =0000000000000000 r14 =0000000000000000 r15 =0000001bc2647f98
iopl=0 nv up ei ng nz na po cy
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287
dynamorio!__chkstk+0x37:
000000001541b187 41c60300 mov byte ptr [ r11 ],0 ds:0000001bc2423000=00

the TEB.StackLimit is higher than dstack so it thinks we're way off
0:000> !teb
TEB at 000007f79ddee000
ExceptionList: 0000000000000000
StackBase: 0000001bc2520000
StackLimit: 0000001bc2421000

dynamorio!__chkstk looks the same on VS2010 win7, but:
0:001> !teb
TEB at 000007fffffdc000
ExceptionList: 0000000000000000
StackBase: 0000000003090000
StackLimit: 000000000308c000

really we want to get rid of __chkstk altogether

so we can either use #pragma check_stack
or build with /Gs65536 (default is 4K)

Original issue: http://code.google.com/p/dynamorio/issues/detail?id=921

[derekbruening]

From bruen...@google.com on September 29, 2012 15:28:31

*** DONE adding to target COMPILE_FLAGS messes up rc
CLOSED: [2012-09-28 Fri 16:30]
- State "DONE" from "TODO" [2012-09-28 Fri 16:30]

FAILED: "C:/PROGRA2/CMAKE21.8/bin/cmcldeps.exe" RC C:\derek\dr\git\src\core\win32\resources.rc "core/CMakeFiles/dynamorio.dir/win32/resources.rc.res.d" core\CMakeFiles\dynamorio.dir\win32\resources.rc.res "Note: including file: " "C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin/amd64/cl.exe" C:\PROGRA2\WI3CF21\8.0\bin\x64\rc.exe /nologo -IC:/derek/dr/git/src/core/win32 -IC:/derek/dr/git/src/core/x86 -IC:/derek/dr/git/src/core/lib -IC:/derek/dr/git/build_x64_dbg_tests -DRC_IS_CORE -DINCLUDE_EVENTS /Gs65536 -Ddynamorio_EXPORTS /focore\CMakeFiles\dynamorio.dir\win32\resources.rc.res C:\derek\dr\git\src\core\win32\resources.rc
fatal error RC1203 : invalid language ID or language name specified.
ninja: build stopped: subcommand failed.

cmake must know to strip out params and leave only defs from *_C_FLAGS?

xref similar issue here: http://www.cmake.org/pipermail/cmake/2009-August/031672.html and here: http://www.cmake.org/pipermail/cmake/2008-June/022381.html though those are both putting flags into add_definitions()

=> adding to each source file

except I really want target-specific source properties which this implies
do not exist: http://www.cmake.org/pipermail/cmake/2011-March/043497.html set_target_properties(dynamorio PROPERTIES
COMPILE_FLAGS "-DRC_IS_CORE -DINCLUDE_EVENTS /Gs65536")
% grep -2 RC_IS_CORE build.ninja | grep -A 3 resources.rc
build core\CMakeFiles\dynamorio.dir\win32\resources.rc.res: RC_COMPILER C$:\derek\dr\git\src\core\win32\resources.rc || core\generate_syslog core\generate_events api_headers core\ntdll_imports.lib
DEFINES = -Ddynamorio_EXPORTS
DEP_FILE = core/CMakeFiles/dynamorio.dir/win32/resources.rc.res.d
FLAGS = /nologo -IC:/derek/dr/git/src/core/win32 -IC:/derek/dr/git/src/core/x86 -IC:/derek/dr/git/src/core/lib -IC:/derek/dr/git/build_x64_dbg_tests -DRC_IS_CORE -DINCLUDE_EVENTS /Gs65536
TARGET_PDB = lib64\debug\dynamorio.pdb

set_target_properties(dynamorio PROPERTIES
COMPILE_DEFINITIONS "RC_IS_CORE;INCLUDE_EVENTS"
COMPILE_FLAGS "/Gs65536")
% grep -2 RC_IS_CORE build.ninja | grep -A 3 resources.rc
build core\CMakeFiles\dynamorio.dir\win32\resources.rc.res: RC_COMPILER C$:\derek\dr\git\src\core\win32\resources.rc || core\generate_events core\generate_syslog api_headers core\ntdll_imports.lib
DEFINES = -Ddynamorio_EXPORTS -DRC_IS_CORE -DINCLUDE_EVENTS
DEP_FILE = core/CMakeFiles/dynamorio.dir/win32/resources.rc.res.d
FLAGS = /nologo -IC:/derek/dr/git/src/core/win32 -IC:/derek/dr/git/src/core/x86 -IC:/derek/dr/git/src/core/lib -IC:/derek/dr/git/build_x64_dbg_tests /Gs65536

go to custom command for running rc.exe?
first: is this only with ninja generator?
=> yes it is!
VS generator works just fine:
ResourceCompile:
C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\rc.exe /D RC_IS_CORE /D INCLUDE_EVENTS /D "CMAKE_INTDIR=\"Debug\"" /D dynamorio_EXPORTS /l"0x0409" /ID:/derek/dr/git/src/core/win32 /ID:/derek/dr/git/src/core/x86 /ID:/derek/dr/git/src/core/lib /I"D:/derek/dr/build_suite/build_debug-internal-64" /nologo /fo"dynamorio.dir\Debug\resources.res" ......\git\src\core\win32\resources.rc

and in fact it's already been filed and very recently fixed: http://www.cmake.org/Bug/view.php?id=13486 I tried to sed build.ninja via a PRE_BUILD custom command but:
Note that the PRE_BUILD option is only supported on Visual Studio 7 or
later. For all other generators PRE_BUILD will be treated as PRE_LINK.
=> not executed in time for ninja

gave up. requiring building from sources. default is /MD so:

% git clone git://cmake.org/cmake.git cmake.git
% mkdir build; cd build
% cmake -G"Ninja" -DCMAKE_BUILD_TYPE:STRING=Release -DCMAKE_C_FLAGS_RELEASE:STRING="/MT /O2 /Ob2 /D NDEBUG" -DCMAKE_CXX_FLAGS_RELEASE:STRING="/MT /O2 /Ob2 /D NDEBUG" ../cmake.git && ninja

[derekbruening]

From derek.br...@gmail.com on October 02, 2012 15:42:24

This issue was closed by revision r1623 .

Status: Fixed