add Windows 10 support #1714

derekbruening · 2015-06-26T14:33:23Z

This issue covers adding Windows 10 support to DynamoRIO

xref DrMem issue: DynamoRIO/drmemory#1669

derekbruening · 2015-08-11T13:42:30Z

There are not many new ntoskrnl syscalls:

37a38
> NtAlpcImpersonateClientContainerOfPort
63a65
> NtCompareObjects
87a90
> NtCreatePartition
158a162
> NtGetCurrentProcessorNumberEx
185a190
> NtManagePartition
209a215
> NtOpenPartition
326a333
> NtRevertContainerImpersonation
358a366
> NtSetInformationSymbolicLink

derekbruening · 2015-08-11T13:44:06Z

** TODO wow64 syscall using near indirect call

ntdll!NtAcceptConnectPort:
77cd8be0 b802000000      mov     eax,2
77cd8be5 bab0d5ce77      mov     edx,offset ntdll!Wow64SystemServiceCall (77ced5b0)
77cd8bea ffd2            call    edx
77cd8bec c21800          ret     18h
77cd8bef 90              nop

ntdll!Wow64SystemServiceCall:
77ced5b0 648b1530000000  mov     edx,dword ptr fs:[30h]
77ced5b7 8b9254020000    mov     edx,dword ptr [edx+254h]
77ced5bd f7c202000000    test    edx,2
77ced5c3 7403            je      ntdll!Wow64SystemServiceCall+0x18 (77ced5c8)
77ced5c5 cd2e            int     2Eh
77ced5c7 c3              ret
77ced5c8 eacfd5ce773300  jmp     0033:77CED5CF
77ced5cf 41              inc     ecx

Symbols don't show a field beyond +0x248 =
CsrServerReadOnlySharedMemoryBase but it's clearly there:

0:000> dd $peb+248
7ea2a248  bc1b0000 00007ff7 00001000 00000001

Compare to pre-win10:

ntdll!NtAcceptConnectPort:
7755025c b860000000      mov     eax,60h
77550261 33c9            xor     ecx,ecx
77550263 8d542404        lea     edx,[esp+4]
77550267 64ff15c0000000  call    dword ptr fs:[0C0h]
7755026e 83c404          add     esp,4
77550271 c21800          ret     18h
0:000> U poi($teb+c0)
74132320 ea1e2713743300  jmp     0033:7413271E

We made the far call* the syscall gateway before to have it inlined where
it's easier to see the syscall #. We'll have to figure out the best
approach here -- try to do the same thing w/ some pattern matching, or
wait for the far jmp?

derekbruening · 2015-08-11T16:06:15Z

Running c:/windows/system32/calc.exe and you end up with C:\Program Files\WindowsApps\Microsoft.WindowsCalculator_10.1507.15010.0_x64__8wekyb3d8bbwe\Calculator.exe
ProcessExplorer shows the parent as svchost.exe.
Run under DR, calc.exe exits while the "Calculator" app stays up.

derekbruening · 2015-08-12T16:39:26Z

Back to the wow64 syscalls: I was going to use the far jmp and live with the perf hit of always using shared_syscall, but it complicates does_syscall_ret_to_callsite() and also our own generation of syscall instrs, so it's not clear what the best solution is.

derekbruening · 2015-08-12T16:40:03Z

** TODO wow64 NtQueryInformationProcess has weird wrapper

ntdll!NtQueryInformationProcess:
77cd8d50 b819000000      mov     eax,19h
77cd8d55 e804000000      call    ntdll!NtQueryInformationProcess+0xe (77cd8d5e)
77cd8d5a 0000            add     byte ptr [eax],al
77cd8d5c c7              ???
77cd8d5d 775a            ja      ntdll!NtCreateKey+0x9 (77cd8db9)

77cd8d5e 5a              pop     edx
77cd8d5f 807a034b        cmp     byte ptr [edx+3],4Bh
77cd8d63 750a            jne     ntdll!NtQueryInformationProcess+0x1f (77cd8d6f)
77cd8d65 64ff15c0000000  call    dword ptr fs:[0C0h]
77cd8d6c c21400          ret     14h
77cd8d6f bab0d5ce77      mov     edx,offset ntdll!Wow64SystemServiceCall (77ced5b0)
77cd8d74 ffd2            call    edx
77cd8d76 c21400          ret     14h

0:000> U 77c70000 
ntdll!__guard_fids_table <PERF> (ntdll+0x0):
77c70000 4d              dec     ebp

using 64-bit windbg, here is fs:c0:
0:000:x86> U poi(000000007fa7f000+c0) L10
wow64cpu!TurboDispatchJumpAddressEnd+0x544:
59da1e60 6a33            push    33h
59da1e62 50              push    eax
59da1e63 e804000000      call    wow64cpu!TurboDispatchJumpAddressEnd+0x550 (59da1e6c)
59da1e68 0b00            or      eax,dword ptr [eax]
59da1e6a 0000            add     byte ptr [eax],al
59da1e6c 58              pop     eax
59da1e6d 0300            add     eax,dword ptr [eax]
59da1e6f 870424          xchg    eax,dword ptr [esp]
59da1e72 cb              retf
59da1e73 41              inc     ecx
59da1e74 ffa7f8000000    jmp     dword ptr [edi+0F8h]
0:000:x86> !sw
Switched to 64bit mode
0:000> U 59da1e73 
wow64cpu!TurboDispatchJumpAddressEnd+0x557:
00000000`59da1e73 41ffa7f8000000  jmp     qword ptr [r15+0F8h]

Vs the regular path:
ntdll_77c70000!Wow64SystemServiceCall+0x1f:
00000000`77ced5cf 41ffa7f8000000  jmp     qword ptr [r15+0F8h]

So it adds 0xb to 0x59da1e68 => 0x59da1e73, and w/ the push of 0x33, ends
up doing a far transition.

So the question is: why does it read a byte from 77cd8d5d and if it's 0x4b
it goes that other path to get to the same destination?

This is the only syscall like this, as evidenced by it being the only one
that winsysnums couldn't find the number for.

derekbruening · 2015-08-12T22:31:12Z

** TODO ASSERT os_take_over_wow64_extra()

Running wow64 notepad
<!check_sole_thread() case 9423>

x64 PEB is still in low 4GB
But nt_wow64_read_virtual_memory64 failed:
c0000008 == STATUS_INVALID_HANDLE
But we passed NT_CURRENT_PROCESS! Very strange. When I pass in a real
handle it works.

derekbruening · 2015-08-13T21:41:39Z

** TODO ASSERT on safe_read fault continuation for Control Flow Guard binaries

Running notepad on x86 kernel (hello works fine) (and it turns out,
notepad on x64 or wow64 -- b/c notepad has CFG enabled):

0:000> kn
05 1ab0db60 68d66334 dynamorio!report_dynamorio_problem+0x529 [d:\dynamorio_package\core\utils.c @ 2284]
06 1ab0dc98 68f4c196 dynamorio!internal_error+0x124 [d:\dynamorio_package\core\utils.c @ 187]
07 1ab0dcd0 68f41e9d dynamorio!nt_continue+0x166 [d:\dynamorio_package\core\win32\ntdll.c @ 2216]
08 1ab0e0dc 6909fbec dynamorio!intercept_exception+0xa2d [d:\dynamorio_package\core\win32\callback.c @ 5242]
09 1ab0e6c0 68f16e51 dynamorio!interception_code_array+0xbec
0a 1ab0e6e0 68e635c9 dynamorio!safe_read+0x21 [d:\dynamorio_package\core\win32\os.c @ 5353]
0b 1ab0e730 68e633a6 dynamorio!is_annotation_tag+0x129 [d:\dynamorio_package\core\annotations.c @ 868]
0c 1ab0e750 68e6218c dynamorio!identify_annotation+0x26 [d:\dynamorio_package\core\annotations.c @ 973]
0d 1ab0e858 68ebbf34 dynamorio!instrument_annotation+0x11c [d:\dynamorio_package\core\annotations.c @ 359]
0e 1ab0ed64 68ecd9b7 dynamorio!build_bb_ilist+0x1d04 [d:\dynamorio_package\core\arch\interp.c @ 3609]
0f 1ab0ef00 68d47f0d dynamorio!build_basic_block_fragment+0x397 [d:\dynamorio_package\core\arch\interp.c @ 5050]
10 1ab0eff4 1aac2107 dynamorio!dispatch+0x8bd [d:\dynamorio_package\core\dispatch.c @ 209]

I'm omitting a lot of debugging here and putting in some summaries:

Return value is 0xc000000d == STATUS_INVALID_PARAMETER

Raw sysenter code seems fine, and it's not some sysenter source security
check b/c DR makes a lot of syscalls via dynamorio_syscall_sysenter prior
to this point (and hello.exe runs to completion just fine).

Setting just integer (0x10002) works!
Adding CONTEXT_CONTROL is what results in 0xc000000d

*** TODO is it TEB stack fields? TODO -- try tweaking them, and review DrMem#1676

notepad:

            C140 DLL characteristics
                   Dynamic base
                   NX compatible
                   RESERVED - UNKNOWN
                   Terminal Server Aware

VS2013 dumpbin:

            C140 DLL characteristics
                   Dynamic base
                   NX compatible
                   Guard
                   Terminal Server Aware

We need 8.1 SDK for this one:
#define IMAGE_DLLCHARACTERISTICS_GUARD_CF 0x4000 // Image supports Control Flow Guard.

derekbruening · 2015-08-13T21:42:51Z

** TODO ASSERT ntdll.c:1097 TESTALL(CONTEXT_DR_STATE_NO_YMM, cxt->ContextFlags)

Running wow64 winthread.exe

04 0052f800 708c6334 dynamorio!report_dynamorio_problem+0x529 [d:\dynamorio_package\core\utils.c @ 2284]
05 0052f938 70aa9bf7 dynamorio!internal_error+0x124 [d:\dynamorio_package\core\utils.c @ 187]
06 0052f960 70aa092d dynamorio!context_to_mcontext+0x67 [d:\dynamorio_package\core\win32\ntdll.c @ 1097]
07 0052fb9c 70aa046e dynamorio!intercept_new_thread+0x11d [d:\dynamorio_package\core\win32\callback.c @ 3071]
08 0052fbb8 70c008a6 dynamorio!intercept_ldr_init+0x7e [d:\dynamorio_package\core\win32\callback.c @ 3322]
09 00000000 00000000 dynamorio!interception_code_array+0x8a6

+0x000 ContextFlags : 0x10007

That's CONTROL, INTEGER, and SEGMENTS

CONTEXT_DR_STATE_NO_YMM wants CONTROL, INTEGER, and CONTEXT_EXTENDED_REGISTERS

derekbruening · 2015-08-18T18:10:51Z

Back to the syscall wrappers again here is more discussion:

user32, etc. have own copy of Wow64SystemServiceCall

USER32!NtUserRegisterClassExWOW:
74dd5820 b8b2100000      mov     eax,10B2h
74dd5825 ba80acdd74      mov     edx,offset USER32!Wow64SystemServiceCall (74ddac80)
74dd582a ffd2            call    edx
74dd582c c21c00          ret     1Ch

USER32!Wow64SystemServiceCall:
74ddac80 648b1530000000  mov     edx,dword ptr fs:[30h]
74ddac87 8b9254020000    mov     edx,dword ptr [edx+254h]
74ddac8d f7c202000000    test    edx,2
74ddac93 7403            je      USER32!Wow64SystemServiceCall+0x18 (74ddac98)
74ddac95 cd2e            int     2Eh
74ddac97 c3              ret
74ddac98 ea9facdd743300  jmp     0033:74DDAC9F
74ddac9f 41              inc     ecx

0:000> x *!Wow64SystemServiceCall
74ddac80          USER32!Wow64SystemServiceCall (<no parameter info>)
77ced5b0          ntdll!Wow64SystemServiceCall (<no parameter info>)
74fb8dd0          KERNELBASE!Wow64SystemServiceCall (<no parameter info>)
759b7700          GDI32!Wow64SystemServiceCall (<no parameter info>)
773774b0          KERNEL32!Wow64SystemServiceCall (<no parameter info>)
0:000> U Wow64SystemServiceCall
USER32!Wow64SystemServiceCall:
74ddac80 648b1530000000  mov     edx,dword ptr fs:[30h]
74ddac87 8b9254020000    mov     edx,dword ptr [edx+254h]
74ddac8d f7c202000000    test    edx,2
74ddac93 7403            je      USER32!Wow64SystemServiceCall+0x18 (74ddac98)
74ddac95 cd2e            int     2Eh
74ddac97 c3              ret
74ddac98 ea9facdd743300  jmp     0033:74DDAC9F
74ddac9f 41              inc     ecx
0:000> U GDI32!Wow64SystemServiceCall 
GDI32!Wow64SystemServiceCall:
759b7700 648b1530000000  mov     edx,dword ptr fs:[30h]
759b7707 8b9254020000    mov     edx,dword ptr [edx+254h]
759b770d f7c202000000    test    edx,2
759b7713 7403            je      GDI32!Wow64SystemServiceCall+0x18 (759b7718)
759b7715 cd2e            int     2Eh
759b7717 c3              ret
759b7718 ea1f779b753300  jmp     0033:759B771F
759b771f 41              inc     ecx
0:000> U KERNEL32!Wow64SystemServiceCall
KERNEL32!Wow64SystemServiceCall:
773774b0 648b1530000000  mov     edx,dword ptr fs:[30h]
773774b7 8b9254020000    mov     edx,dword ptr [edx+254h]
773774bd f7c202000000    test    edx,2
773774c3 7403            je      KERNEL32!Wow64SystemServiceCall+0x18 (773774c8)
773774c5 cd2e            int     2Eh
773774c7 c3              ret
773774c8 eacf7437773300  jmp     0033:773774CF
773774cf 41              inc     ecx

The user32 one (and presumably all of them) targets the central
USER32!Wow64SystemServiceCall+0x1f:

00000000`74ddac9f 41ffa7f8000000  jmp     qword ptr [r15+0F8h] ds:00000000`59da3658={wow64cpu!CpupReturnFromSimulatedCode (00000000`59da18e6)}
=>
wow64cpu!CpupReturnFromSimulatedCode

And comes back here -- this jmp targets the retaddr of "call edx":

wow64cpu!RunSimulatedCode+0x6b:
00000000`59da184b 41ff2e          jmp     fword ptr [r14] ds:00000000`04ebe850=002374dd979c

I'm sticking with the near call* as the "syscall" for these reasons:

does_syscall_ret_to_callsite()
related: don't need to go hook the wow64 routines to regain control
easy to generate our own syscall instr
syscall # is in same bb
ends up looking similar to 32-bit kernel for 64-on-64

We have to pattern-match to handle the multiple copies, and tools will miss a few instrs, but these negatives seem outweighed by the positives.

derekbruening · 2015-08-18T18:12:34Z

Re: the NtContinue assert: it is indeed the TEB stack fields from DynamoRIO/drmemory#1676 DynamoRIO/drmemory#1723. Looks like we need to implement #1692 and swap TEB stack fields even if no client.

derekbruening · 2015-08-18T18:13:26Z

** TODO (#899) ASSERT NYI syscall.c:2740 NtSetInformationVirtualMemory

Running notepad, typing something, and then hitting Ctrl+S

<Application C:\Windows\SysWOW64\notepad.exe (4432). Internal Error: DynamoRIO debug check failure: Not implemented @d:\dynamorio_package\core\win32\syscall.c:2740 (0)

It's the new syscall NtSetInformationVirtualMemory which is related to CFG => #899

derekbruening · 2015-08-18T20:47:40Z

** TODO CURIOSITY about missing an unmap

Running notepad, typing something, then closing:

<CURIOSITY : !last_deallocated->unload_in_progress in file d:\dynamorio_package\core\vmareas.c line 10916
version 5.0.16657, custom build
-no_dynamic_options -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct -pad_jmps_mark_no_trace
0x00000096d13fd900 0x00007ff84d35ba10>

Turns out we're not calling postsys_UnmapViewOfSection for NtUnmapViewOfSectionEx.

derekbruening · 2015-08-18T21:03:42Z

*** TODO related curiosity still there

x86 notepad after saving in dialog:

<CURIOSITY : !last_deallocated->unload_in_progress || ((last_deallocated->last_unload_base <= module_base && module_base < (last_deallocated->last_unload_base + last_deallocated->last_unload_size)) && "race - multiple unmaps") in file d:\dynamorio_package\core\vmareas.c l

** TODO CURIOSITY "read off end of rsrc version"

notepad save dialog:

<CURIOSITY : (0) && "read off end of rsrc version" in file d:\dynamorio_package\core\win32\module.c line 6075

derekbruening · 2015-08-20T02:35:10Z

** TODO AppInit injection => works on wow64 now, leaving #1522 for others

% bin32/drconfig -debug -ops "-msgbox_mask 15" -reg notepad.exe -c samples/bin32/bbcount.dll
% bin32/drconfig -syswide_on
ERROR: syswide_on is not yet supported on Windows 8+ non-WOW64

That's #1522.

*** DONE missing get_platform() libutil win10 version
CLOSED: [2015-08-19 Wed 22:34]
- State "DONE" from "TODO" [2015-08-19 Wed 22:34]

*** DONE ldr_module_statically_linked() thinks user32 is not statically linked
CLOSED: [2015-08-19 Wed 22:33]
- State "DONE" from "TODO" [2015-08-19 Wed 22:33]

direct PEB version check doesn't handle win10

derekbruening · 2015-08-20T02:36:59Z

Test suites -- I didn't copy src dir over though:

*** TODO x64 2nd attempt: 10 failures (6 expected)

These failed b/c I didn't copy src dir over:
4 - samples_proj (Failed)
93 - code_api|client.file_io (Failed)
142 - code_api|tool.drltrace (Failed)
143 - code_api|tool.drcov.fib (Failed)
153 - code_api|api.dis (Failed)
157 - code_api|api.symtest (Failed)

These may be real bugs:
49 - code_api|win32.mixedmode (Failed)
50 - code_api|win32.x86_to_x64 (Failed)
51 - code_api|win32.x86_to_x64_ibl_opt (Failed)
56 - code_api|security-common.retnonexisting (Failed)

*** TODO wow64: 9 failures

      4 - samples_proj (Failed)
     29 - code_api|win32.reload (Failed)
    117 - code_api|client.file_io (Failed)
    135 - code_api|client.drsyms-test (Failed)
    168 - code_api|tool.drltrace (Failed)
    169 - code_api|tool.drcov.fib (Failed)
    182 - code_api|api.dis (Failed)
    186 - code_api|api.symtest (Failed)
    188 - code_api,opt_speed|client.events (Failed)

**** DONE -opt_speed failing => fixed now

*** TODO x86: 16 failures but 6 from missing filter event => #1769

      4 - samples_proj (Failed)
     29 - code_api|win32.reload (Failed)
    111 - code_api|client.events (Failed)
    112 - code_api|client.events_cpp (Failed)
    116 - code_api|client.fibers (Failed)
    117 - code_api|client.file_io (Failed)
    135 - code_api|client.drsyms-test (Failed)
    139 - code_api|client.pcache-use (Failed)
    168 - code_api|tool.drltrace (Failed)
    169 - code_api|tool.drcov.fib (Failed)
    182 - code_api|api.dis (Failed)
    186 - code_api|api.symtest (Failed)
    187 - code_api,opt_memory|client.events (Failed)
    189 - code_api,thread_private|client.events (Failed)
    190 - code_api,disable_traces|client.events (Failed)
    191 - code_api,thread_private,disable_traces|client.events (Failed)

derekbruening · 2015-08-20T02:39:29Z

Split mixed-mode off as #1770 b/c I do not plan to fix it

derekbruening · 2015-08-22T00:50:48Z

For reference these are the commits that added win10 support:

72bc044 i#1714 windows 10: add version info and x64 syscalls
673b03a i#1714 windows 10: fix known dll assert
21ac24d i#1714 windows 10: update winsysnums for new WOW64 gateway
ea0fb62 i#1714 windows 10: add x86 and wow64 syscall numbers
1810420 i#1714 windows 10: handle wow64 syscalls
abb846a i#1714 windows 10: pass real handle to NtWow64ReadVirtualMemory64
fef0150 i#1714 windows 10: wow64 takeover
3d132bb i#1714 windows 10: add new API pseudo-dll mappings
958f581 i#1714 windows 10: swap to priv stack for initial dispatch
b0eedbd i#1692: split TEB stack field swapping from other swaps
b6f25ff i#1714 windows 10: document WOW64 syscall behavior
0b4b9e9 i#1714 windows 10: handle missing xmm state in new threads
f773d41 i#899 win8+ syscalls: handle NtSetInformationVirtualMemory
32db494 i#1714 windows 10: fix missing post-unmap handling
e5ae1bb i#1714 windows 10: reduce noisy syslogs
e950b81 i#1768: fix missing path on private lib load
23531a5 i#1732 drcpusim: add library blacklist
1047053 i#1714 windows 10: wow64 call conversion
4ea4b6c i#1769: fix client.events failures on 32-bit sysenter kernels
bc6fc54 i#1714 windows 10: AppInit WOW64 support
a409821 i#1714 windows 10: update client and test version code
f997cb0 i#1714 windows 10: update limitations section in docs
c1d10ee i#1598: attempt to run on unknown Windows versions
6f67757 DrM#1750 win10 syscalls: add tool to locate usercall syscalls

derekbruening added Type-Feature OpSys-Windows labels Jun 26, 2015

derekbruening self-assigned this Jun 26, 2015

derekbruening mentioned this issue Aug 19, 2015

client.events tests fail on 32-bit kernels due to missing syscall filter event #1769

Closed

derekbruening closed this as completed in a409821 Aug 20, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add Windows 10 support #1714

add Windows 10 support #1714

derekbruening commented Jun 26, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 13, 2015

derekbruening commented Aug 13, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 22, 2015

add Windows 10 support #1714

add Windows 10 support #1714

Comments

derekbruening commented Jun 26, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 11, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 12, 2015

derekbruening commented Aug 13, 2015

derekbruening commented Aug 13, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 18, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 20, 2015

derekbruening commented Aug 22, 2015