CRASH get_ldr_module_64() win8+ due to Norton's injected early thread

[derekbruening]

A user running Dr. Memory in Visual Studio hit this crash:

Dr. Memory internal crash at PC 0x70a0e328
0xc0000005 0x00000000 0x70a0e328 0x70a0e328 0x00000000 0x4cb60340
Base: 0x70950000
Registers: eax=0x7f2bf000 ebx=0x4cb60340 ecx=0x0000009c edx=0xffffffff
        esi=0x23081030 edi=0x23081024 esp=0x010bf2e8 ebp=0x00000000
        eflags=0x000
version 5.0.16470, custom build

% bin/symquery -e dynamorio/lib32/release/dynamorio.dll -f -a 0xbe328
get_ldr_module_64+0x28
d:\drmemory_package\dynamorio\core\win32\module_shared.c:878+0x0

We managed to get an ldmp by passing -dr_ops "-dumpcore_mask 0xf".

0:000> .cxr @@(cxt)
eax=7f2bf000 ebx=4cb60340 ecx=0000009c edx=ffffffff esi=23081030 edi=23081024
eip=70a0e328 esp=010bf2e8 ebp=00000000 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
dynamorio!get_ldr_module_64+0x28:
70a0e328 8b33            mov     esi,dword ptr [ebx]  ds:002b:4cb60340=????????
0:000> kn
 # ChildEBP RetAddr  
00 010bf2f4 70a0e3fc dynamorio!get_ldr_module_64+0x28 [d:\drmemory_package\dynamorio\core\win32\module_shared.c @ 878]
01 010bf300 709f5d5d dynamorio!thread_get_context_64+0xc [d:\drmemory_package\dynamorio\core\win32\module_shared.c @ 1040]
02 010bf328 709f6158 dynamorio!os_take_over_wow64_extra+0x4d [d:\drmemory_package\dynamorio\core\win32\os.c @ 1942]
03 010bf7c4 709f6391 dynamorio!os_take_over_thread+0x1a8 [d:\drmemory_package\dynamorio\core\win32\os.c @ 2133]
04 010bf7f4 7097e90e dynamorio!os_take_over_all_unknown_threads+0x121 [d:\drmemory_package\dynamorio\core\win32\os.c @ 2238]
05 010bf804 709ee312 dynamorio!dynamorio_take_over_threads+0xe [d:\drmemory_package\dynamorio\core\dynamo.c @ 2637]
06 010bf81c 709ee508 dynamorio!auto_setup+0x42 [d:\drmemory_package\dynamorio\core\arch\x86_code.c @ 183]

dynamorio!get_ldr_module_64:
70a0e300 51              push    ecx
70a0e301 6aff            push    0FFFFFFFFh
70a0e303 e89851ffff      call    dynamorio!is_wow64_process (70a034a0)
70a0e308 83c404          add     esp,4
70a0e30b 84c0            test    al,al
70a0e30d 7504            jne     dynamorio!get_ldr_module_64+0x13 (70a0e313)
70a0e30f 33c0            xor     eax,eax
70a0e311 eb0c            jmp     dynamorio!get_ldr_module_64+0x1f (70a0e31f)
70a0e313 65a160000000    mov     eax,dword ptr gs:[00000060h]
70a0e319 890424          mov     dword ptr [esp],eax   <--- PEB64
70a0e31c 8b0424          mov     eax,dword ptr [esp]
70a0e31f 53              push    ebx
70a0e320 8b5818          mov     ebx,dword ptr [eax+18h]  <--- LDR64
70a0e323 55              push    ebp
70a0e324 83c320          add     ebx,20h
70a0e327 56              push    esi
70a0e328 8b33            mov     esi,dword ptr [ebx]  <--- CRASH b/c just bottom bits

0:000> dds esp
010bf2e8  23081030
010bf2ec  00000000
010bf2f0  00001778
010bf2f4  7f2bf000
010bf2f8  70a0e3fc dynamorio!thread_get_context_64+0xc [d:\drmemory_package\dynamorio\core\win32\module_shared.c @ 1040]

Probable TEB for unknown thread region (or x64 PEB/TEB?) addr 0x7f2bf000 size 0x00001000
ERROR: unable to allocate memory at 0x7f2bf000 size 0x00001000
         will be copied to 0x00040000 instead

    X64_PEB_TIB_OFFSET        = 0x060,
    X64_LDR_PEB_OFFSET        = 0x018,

0:000> dq 0x00040000
00040000  00000000`04000000 ffffffff`ffffffff
00040010  00000000`00f90000 00007fff`4cb60320
00040020  00000000`010e24e0 00000000`00000000

ebx=4cb60340

So PEB->LoaderData is stored up >4GB and our code just gets the bottom 32 bits. Which is DynamoRIO/dynamorio#1035

We would only run this code at init if there's a 2nd thread in there, which normally should not happen.

After some sleuthing we have the 2nd thread's start addr and arg:

   +0x0a4 Ebx              : 0xee0000
   +0x0a8 Edx              : 0
   +0x0ac Ecx              : 0
   +0x0b0 Eax              : 0xee0064

Further sleuthing and we see that this routine loaded a library at 0x70d20000, which is Norton:

70d20000 70d5e000 UMEngx86 (export symbols) C:\Program Files (x86)\Norton Internet Security\NortonData\21.1.0.18\Definitions\BASHDefs\20150203.001\UMEngx86.dll

[HaibeiZhu]

Hi, I am the user actually. Yes i believed the problem is Norton, when I uninstalled Norton, Dr Memory worked fine. So it mean I can only use Dr Memory without Norton? Actually I tried when I switched off all Norton's scan, Dr Memory still doesn't work...

[derekbruening]

It may be that Norton still injects and creates its thread even when switched "off".

We will augment Dr. Memory to handle these unusual early threads soon and then we can see whether there are any other issues or whether everything works then. I'll update the issue when we have a fixed build to test.

[derekbruening]

Could you try this build (it's a self-extracting archive, so unpack somewhere and run bin/drmemory from there) with Norton enabled?

http://build.chromium.org/p/client.drmemory/builds/DrMemory-Windows-1.8.16511-0x4b30da9-sfx.exe

[HaibeiZhu]

Sorry for the late reply, last week I uninstall Norton because I need Dr Memory for my class. And I re-install Norton today, trying to test the file
http://build.chromium.org/p/client.drmemory/builds/DrMemory-Windows-1.8.16511-0x4b30da9-sfx.exe
Actually the result is Dr Memory works fine now !!! But I think the thing is strange:

1. when I saw your reply, I first download the above file you gave me to my desktop (which is C disk, and I install Dr Memory in D disk)
2. and then I go to Norton website to re-install Norton Internet Security (NIS), but this time I change the installation path to D disk, not C disk (I used to install Norton in C)
3. and then I found Dr Memory still work in Visual Studio... at this time, I didn't do anything to the file you gave me yet
4. I restart my computer, and restart Visual Studio (Norton start itself when computer start)
5. I found the Norton still work... which is really strange!! Because this is impossible a week ago
   I am not sure what happened, I am now using NIS full system scan my computer, and I will try again after the scan.
   And I will try to run bin/drmemory from the file later.
   Thank you so much!

[HaibeiZhu]

After I re-install the Norton and update the virus data base, then the Dr Memory doesn't work with Visual Studio again...

1. I put the DrMemory-Windows-1.8.16511-0x4b30da9-sfx.exe file in the same folder where I install Dr Memory and double click on it.
2. It run something really fast and then its window disappear, so I think the process is done.
3. but I am not sure what you mean "run bin/drmemory from there", what should I do?
4. but then the Dr Memory still doesn't work with visual studio...

[derekbruening]

The sfx.exe self-extracting archive does not install into Program Files or affect the Visual Studio External Tool setup, so it sounds like the fix has not yet been tested. If you are more comfortable with running in Visual Studio than the command line, I have created an installer here: http://drmemory.org/DrMemory-Windows-1.8.1-RC2.exe Please uninstall the current version of Dr. Memory on your machine and install this version, which should update what's run from the Visual Studio External Tool menu item.

[HaibeiZhu]

Thank you for the new installer!! Yes I think I am more familiar with running Dr Memory in Visual Studio. I already uninstalled the old version of drmemory, and install the new one 1.8.1_RC2.exe.
But there is another error message, let me restart the computer.

[HaibeiZhu]

WARNING: application exited with abnormal code 0xc0000005

this time, the error code is 0xc0000005

[derekbruening]

OK I installed Norton and it looks like the original early thread problem is fixed and the new problem is due to the many, many hooks that Norton installs. It looks like the same problem as #1686.

[derekbruening]

There are further problems as well. Normally we might split them into separate issues but to ensure user notifications we'll keep them all here. Re-opening.

We have these problems:

1. Early thread that on win8+ triggered the high-ntdll bug, now fixed
2. Syscall hooks: Comodo Internet Security 8.1 makes Dr. Memory unable to find find system calls #1686
3. A bug in the early thread fix in release build (works in debug build): TOCHECK: why didn't new test catch it
4. An ASSERT about the Norton thread's xsp not being in the TEB range
5. Several unaddr and uninits reported, presumably related to the Norton thread

Elaborating on the 4) ASSERT:

% bin/drmemory -debug -batch -pause_at_assert -- c:/derek/hello.exe
~~Dr.M~~ Dr. Memory version 1.8.16544
~~Dr.M~~ Running "c:/derek/hello.exe"
~~5060~~ ASSERT FAILURE (thread 5060): D:\derek\drmemory\git\src\drmemory\drmemory.c:1159: mc.xsp <= (reg_t)teb->StackBase && mc.xsp > (reg_t)teb->StackLimit (initial xsp for thread invalid)Hello world!

Looks like the Norton thread, at takeover point at init time:

04 1e7ae5d8 738072c0 drmemorylib!drmemory_abort+0x18 [d:\derek\drmemory\git\src\common\utils.c @ 128]
05 1e7ae8ac 7380c333 drmemorylib!set_thread_initial_structures+0x12d0 [d:\derek\drmemory\git\src\drmemory\drmemory.c @ 1159]
06 1e7aecd0 73a9669c drmemorylib!event_thread_init+0x253 [d:\derek\drmemory\git\src\drmemory\drmemory.c @ 550]
07 1e7aee34 72ddc8d5 drmemorylib!drmgr_thread_init_event+0xbc [d:\derek\dr\git\src\ext\drmgr\drmgr.c @ 1567]
08 1e7aee60 72d9f4de dynamorio!instrument_thread_init+0xc5 [d:\derek\dr\git\src\core\lib\instrument.c @ 1198]
09 1e7aee80 72e17dc9 dynamorio!dynamo_thread_init+0x20e [d:\derek\dr\git\src\core\dynamo.c @ 2198]
0a 1e7aee9c 72e2062d dynamorio!thread_attach_setup+0xa9 [d:\derek\dr\git\src\core\win32\os.c @ 2309]
0b 1e7aeea4 72ea1f1f dynamorio!thread_attach_takeover_callee+0xd [d:\derek\dr\git\src\core\win32\callback.c @ 8368]
0c 1e7aeeac 00000000 dynamorio!interception_code_array+0xf1f

0:001> !teb
TEB at 7ebaa000
    ExceptionList:        0136fb08
    StackBase:            00000000
    StackLimit:           01360000


gui-inject native, 2nd thread:
    StackBase:            00c20000
    StackLimit:           00c10000
0:001> kn
 # ChildEBP RetAddr  
WARNING: Frame IP not in any known module. Following frames may be wrong.
00 00c1ff80 75157c04 0x1b02c8
01 00c1ff94 7797b5af KERNEL32!BaseThreadInitThunk+0x24
02 00c1ffdc 7797b57a ntdll!__RtlUserThreadStart+0x2f
03 00c1ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

Later run:
    StackBase:            00d90000
    StackLimit:           00d80000

mc->esp is dstack

0:001> dt mc
Local var @ 0x167ce760 Type _dr_mcontext_t
   +0x000 size             : 0x148
   +0x004 flags            : 2 ( DR_MC_CONTROL )
   +0x008 xdi              : 0xcccccccc
   +0x008 edi              : 0xcccccccc
   +0x00c xsi              : 0xcccccccc
   +0x00c esi              : 0xcccccccc
   +0x010 xbp              : 0xcccccccc
   +0x010 ebp              : 0xcccccccc
   +0x014 xsp              : 0x167ce550
   +0x014 esp              : 0x167ce550
   +0x018 xbx              : 0xcccccccc
   +0x018 ebx              : 0xcccccccc
   +0x01c xdx              : 0xcccccccc
   +0x01c edx              : 0xcccccccc
   +0x020 xcx              : 0xcccccccc
   +0x020 ecx              : 0xcccccccc
   +0x024 xax              : 0xcccccccc
   +0x024 eax              : 0xcccccccc
   +0x028 xflags           : 0x287
   +0x028 eflags           : 0x287

Related to i#117: mcontext of 1st thread?

thread_attach_takeover_callee has the mc right:
0:001> ?? state->mc.esp
unsigned int 0xd8f80c

esp is changed, rest is the same, in dcontext copy:

0:001> ?? dcontext->upcontext.upcontext.mcontext.esp
unsigned int 0x167ce550

watchpoint => changed here:

0:001> kn
 # ChildEBP RetAddr  
00 2159e424 73806036 dynamorio!interception_code_array+0x1c4b
01 2159e6fc 7380c333 drmemorylib!set_thread_initial_structures+0x46 [d:\derek\drmemory\git\src\drmemory\drmemory.c @ 1104]
02 2159eb20 73a9669c drmemorylib!event_thread_init+0x253 [d:\derek\drmemory\git\src\drmemory\drmemory.c @ 550]
03 2159ec84 7186bc70 drmemorylib!drmgr_thread_init_event+0xbc [d:\derek\dr\git\src\ext\drmgr\drmgr.c @ 1567]
04 2159ecbc 716d652d dynamorio!instrument_thread_init+0x120 [d:\drmemory_package\dynamorio\core\lib\instrument.c @ 1198]
05 2159ed30 7192d7b4 dynamorio!dynamo_thread_init+0x81d [d:\drmemory_package\dynamorio\core\dynamo.c @ 2198]
06 2159ee94 71965240 dynamorio!thread_attach_setup+0xb4 [d:\drmemory_package\dynamorio\core\win32\os.c @ 2305]
07 2159eea4 71ab7f1f dynamorio!thread_attach_takeover_callee+0x10 [d:\drmemory_package\dynamorio\core\win32\callback.c @ 8368]
08 2159eeac 00000000 dynamorio!interception_code_array+0xf1f

71ab8c41 648b0df00e0000  mov     ecx,dword ptr fs:[0EF0h]
71ab8c48 89610c          mov     dword ptr [ecx+0Ch],esp

71ab5b4c          dynamorio!tls_dcontext_offs = 0xef0

Looks like the native_exec_syscalls hook for
ntdll!NtQueryInformationThread.

We are on the initstack right now (xref i#1105 though that's primary thread
init), and these hooks only check whether we are currently on the dstack.

Possible solutions:

1. Have callback hooks also check whether on initstack before deciding to
   switch to dstack
2. Have dynamo_thread_init switch to dstack as soon as it's created. Maybe
   re-architect to solve i#1105 (and i#1152 and i#626?) at the same time,
   and have caller create dstack and switch to it earlier?

-no_native_exec_syscalls => no assert, but still lots of unaddrs and
uninits, in 2 different threads

[HaibeiZhu]

Thank you for your analysis ! Actually I do not quite understand what is going on...
The current thing is that I re-install DrMemory 1.8.1 RC1, and re-install Norton 2 weeks ago. Recently, sometimes DrMemory works with Visual Studio, but sometimes the error code still show up (Norton is always on with full function). I am not sure whether the C++ code or project working in VS matter to this issue (I think it does not...).

[HaibeiZhu]

Although I do not quite understand your last comment, thank you for your help. Right now DrMemory works fine in VS with Norton. If next time DrMemory fail to work and error shows, I will post the new error code here. It is really confused to me why sometimes it works, sometime it does not...

[derekbruening]

Items 1 through 4 above are all fixed. Only the false positives (or maybe true positives: not analyzed) from Norton's own code remain and I will file a separate issue on that.