i#803: Cross-arch Windows injection

Adds a long-missing feature: following into a Windows child process of
a different bitwidth.

Switches injection from DR and from drinjectlib (including drrun and
drinject) to use -early_inject_map.  This was most easily done by
turning on -early_inject by default as well.  However, the
-early_inject_location default is INJECT_LOCATION_ImageEntry, which is
the same late takeover point as with thread injection.  Switching all
injection over to map-from-the-parent simplifies cross-arch following,
as well as making it easier to shift the takeover point to an earlier
spot in the future.  This is a step toward #607 by switching
drinjectlib to use map injection; the takeover point, as mentioned, is
still the image entry.

Adds an -inject_x64 option to inject a 64-bit DR lib into a 32-bit
child from a 64-bit parent, but this option is only sketched out and
is not fully supported yet: #49 covers adding tests and official
support.

Adds library swapping code to find the other-bitwidth library, which
assumes a parallel directory structure.  Add a new fatal error if the
library for a child is not found.

To support generating code for all 3 child-parent cases (same-same,
32-64, and 64-32), and in particular for 32-64, switches the small
gencode sequence for -early_inject_map from using IR to using raw
bytes.  A multi-arch encoder (#1684) would help but we would need
cross-bitwidth support there, which is not on the horizon.  Fixes what
look like bugs in the original gencode generation along the way
(s/pc/cur_local_pos/ and s/local_code_buf/remote_code_buf/): it's not
clear how it worked before.

Adds support for several system calls from a 32-bit parent to a 64-bit
child where the desired NtWow64* system call does not exist.  We use
switch_modes_and_call() for NtProtectVirtualMemory and
NtQueryVirtualMemory.

Changes all types in the injection code to handle 64-bit addresses in
32-bit code.  Adds UNICODE_STRING_32 and
RTL_USER_PROCESS_PARAMETERS_32 for handling 32-bit structures from
64-bit parents.  Similarly, adds RTL_USER_PROCESS_PARAMETERS_64 and
PROCESS_BASIC_INFORMATION64.

Adds get_process_imgname_cmdline() capability for 64-bit remote from 32-bit.

Adds get_remote_proc_address() and uses it to look up
dynamorio_earliest_init_takeover() in a child DR.

Finds the remote ntdll base via a remote query memory walk plus remote
image header parsing.  This requires adding a switch_modes_and_call()
version of NtQueryVirtualMemory (also mentioned above), which needs
64-bit args: so we refactor switch_modes_and_call() to take in a
struct of all 64-bit fields for the args.

Fixes a few bugs in other routines to properly get the image name and
image entry for 32-bit children of 64-bit parents.

Updates environment variable propagation code to handle a 32-bit
parent and a 64-bit child.  Updates a 64-bit parent and 32-bit child
to insert the variables into the 32-bit PEB (64-bit does no good),
which requires finding the 32-bit PEB.  This is done via the 32-bit
TEB, using a hack due to what seems like a kernel bug where it has the
TebBaseAddress 0x2000 too low.

Makes environment variable propagation failures fatal and visible,
unlike previously where errors would just result in silently letting
the child run natively.  Turns some other prior soft errors into fatal
errors on child takeover.

Moves environment variable propagation to post-CreateUserProcess
instead of waiting for ResumeThread, which avoids having to get the
thread context (for which we have no other-bitwidth support) to figure
out whether it's the first thread in the process or not.  We bail on
propagation for pre-Vista where we'd have to wait for ResumeThred.

Generalizes the other-bitwidth Visual Studio toolchain environment
variable setting for use in a new build-and-test other-bitwidth test
which builds dynamorio and the large_options client (to ensure options
are propagated to children; and it has convenient init and exit time
prints) for the other bitwidth, arranges parallel lib dirs, and runs
the other client

Issue: #803, #147, #607, #49
Fixes #803

core/CMakeLists.txt

@@ -1,5 +1,5 @@
 # **********************************************************
-# Copyright (c) 2010-2020 Google, Inc.    All rights reserved.
+# Copyright (c) 2010-2021 Google, Inc.    All rights reserved.
 # Copyright (c) 2009-2010 VMware, Inc.    All rights reserved.
 # **********************************************************
 
@@ -399,8 +399,8 @@ else (UNIX)
     win32/syscall.c
     win32/callback.c
     win32/drmarker.c
-    win32/ntdll.c
     win32/ntdll_shared.c
+    win32/ntdll.c
     win32/inject.c
     win32/inject_shared.c
     win32/module.c
@@ -424,8 +424,8 @@ else (UNIX)
    )
   set(PRELOAD_SRCS
     win32/pre_inject.c
-    win32/ntdll.c
     win32/ntdll_shared.c
+    win32/ntdll.c
     win32/inject_shared.c
     win32/drmarker.c
     ${preinject_asm_src}
@@ -439,10 +439,10 @@ else (UNIX)
   set(INJECTOR_SRCS
     win32/injector.c
     win32/inject.c
-    win32/ntdll.c
     win32/inject_shared.c
     win32/module_shared.c
     win32/ntdll_shared.c
+    win32/ntdll.c
     win32/resources.rc
     config.c
     win32/os.c

core/arch/arch.c

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2010-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2010-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2000-2010 VMware, Inc.  All rights reserved.
  * **********************************************************/
 

core/drlibc/drlibc.h

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2011-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2011-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2000-2010 VMware, Inc.  All rights reserved.
  * **********************************************************/
 
@@ -131,6 +131,23 @@ find_script_interpreter(OUT script_interpreter_t *result, IN const char *fname,
  */
 void
 d_r_set_ss_selector();
+
+typedef struct {
+    uint64 func;
+    uint64 arg1;
+    uint64 arg2;
+    uint64 arg3;
+    uint64 arg4;
+    uint64 arg5;
+    uint64 arg6;
+} invoke_func64_t;
+
+/* Switches from 32-bit mode to 64-bit mode and invokes func, passing
+ * arg1, arg2, arg3, arg4, and arg5.  Works fine when func takes fewer
+ * than 5 args as well.
+ */
+int
+switch_modes_and_call(invoke_func64_t *info);
 #endif
 
 #endif /* _DR_LIBC_H_ */

core/drlibc/drlibc_x86.asm

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2011-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2011-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2001-2010 VMware, Inc.  All rights reserved.
  * ********************************************************** */
 
@@ -530,20 +530,19 @@ GLOBAL_LABEL(FUNCNAME:)
         END_FUNC(FUNCNAME)
 
 /*
- * int switch_modes_and_call(uint64 func, void *arg1, void *arg2, void *arg3)
+ * int switch_modes_and_call(invoke_uint64_t *args)
  */
 # undef FUNCNAME
 # define FUNCNAME switch_modes_and_call
         DECLARE_FUNC(FUNCNAME)
 GLOBAL_LABEL(FUNCNAME:)
-        mov      eax, esp    /* get.. */
-        add      eax, ARG_SZ /* ...address of func */
-        mov      ecx, ARG3 /* arg1 */
-        mov      edx, ARG4 /* arg2 */
-        /* save callee-saved registers */
+        mov      eax, ARG1
+        /* Save callee-saved registers. */
         push     ebx
-        mov      ebx, ARG6 /* really ARG5==arg3, but we have 1 push */
-        /* far jmp to next instr w/ 64-bit switch: jmp 0033:<smc_transfer_to_64> */
+        push     esi
+        push     edi
+        push     ebp
+        /* Far jmp to next instr w/ 64-bit switch: jmp 0033:<smc_transfer_to_64>. */
         RAW(ea)
         DD offset smc_transfer_to_64
         DB CS64_SELECTOR
@@ -552,26 +551,34 @@ smc_transfer_to_64:
     /* Below here is executed in 64-bit mode, but with guarantees that
      * no address is above 4GB, as this is a WOW64 process.
      */
-        /* save WOW64 state */
+        /* Save WOW64 calee-saved registers. */
         RAW(41) push     esp /* push r12 */
         RAW(41) push     ebp /* push r13 */
         RAW(41) push     esi /* push r14 */
         RAW(41) push     edi /* push r15 */
-        RAW(44) mov      eax, ebx /* mov arg3 from ebx to r8d (3rd arg slot) */
-        /* align the stack pointer */
+        /* Align the stack pointer. */
         mov      ebx, esp        /* save esp in callee-preserved reg */
-        sub      esp, 32         /* call conv */
         and      esp, HEX(fffffff0) /* align to 16-byte boundary */
-        /* arg1 is already in rcx, arg2 in rdx, and arg3 now in r8 */
-        RAW(48) mov eax, DWORD [eax] /* mov rax, qword ptr [rax] */
+        /* Set up args on the stack. */
+        RAW(48) mov      ecx, DWORD [eax + 8*6] /* load args.arg6 */
+        push     ecx /* push args.arg6 */
+        RAW(48) mov      ecx, DWORD [eax + 8*5] /* load args.arg5 */
+        push     ecx /* push args.arg5 */
+        sub      esp, 32         /* Leave slots for args 1-4. */
+        /* arg1 is already in rcx, arg2 in rdx, arg3 in r8, arg4 in r9 */
+        RAW(4c) mov      ecx, DWORD [eax + 8*4] /* load args.arg4 into r9 */
+        RAW(4c) mov      eax, DWORD [eax + 8*3] /* load args.arg3 into r8 */
+        RAW(48) mov      edx, DWORD [eax + 8*2] /* load args.arg2 into rdx */
+        RAW(48) mov      ecx, DWORD [eax + 8*1] /* load args.arg1 into rcx */
+        RAW(48) mov      eax, DWORD [eax] /* load args.func into rax */
         call     eax             /* call rax */
-        mov      esp, ebx        /* restore esp */
-        /* restore WOW64 state */
+        mov      esp, ebx        /* restore rsp */
+        /* Restore WOW64 callee-saved regs. */
         RAW(41) pop      edi /* pop r15 */
         RAW(41) pop      esi /* pop r14 */
         RAW(41) pop      ebp /* pop r13 */
         RAW(41) pop      esp /* pop r12 */
-        /* far jmp to next instr w/ 32-bit switch: jmp 0023:<smc_return_to_32> */
+        /* Far jmp to next instr w/ 32-bit switch: jmp 0023:<smc_return_to_32>. */
         push     offset smc_return_to_32  /* 8-byte push */
         mov      dword ptr [esp + 4], CS32_SELECTOR /* top 4 bytes of prev push */
         jmp      fword ptr [esp]
@@ -586,8 +593,12 @@ smc_return_to_32:
          */
         mov      ebx, DWORD SYMREF(d_r_ss_value)
         mov      ss, ebx
-        pop      ebx             /* restore callee-saved reg */
-        ret                      /* return value already in eax */
+        /* Restore callee-saved regs. */
+        pop      ebp
+        pop      edi
+        pop      esi
+        pop      ebx
+        ret /* return value already in eax */
         END_FUNC(FUNCNAME)
 
 #endif /* WINDOWS && !X64 */

core/module_shared.h

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2011-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2011-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2008-2010 VMware, Inc.  All rights reserved.
  * **********************************************************/
 
@@ -376,7 +376,7 @@ get_proc_address_resolve_forward(module_base_t lib, const char *name);
 #endif /* WINDOWS */
 
 #ifdef WINDOWS
-void *
+uint64
 get_remote_process_entry(HANDLE process_handle, OUT bool *x86_code);
 #endif
 

core/optionsx.h

@@ -1,5 +1,5 @@
 /* *******************************************************************************
- * Copyright (c) 2010-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2010-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2011 Massachusetts Institute of Technology  All rights reserved.
  * Copyright (c) 2003-2010 VMware, Inc.  All rights reserved.
  * *******************************************************************************/
@@ -566,6 +566,11 @@ OPTION_DEFAULT_INTERNAL(bool, mangle_app_seg,
                         "mangle application's segment usage.")
 #endif /* X86 */
 #ifdef X64
+#    ifdef WINDOWS
+/* TODO i#49: This option is still experimental and is not fully tested/supported yet. */
+OPTION_DEFAULT(bool, inject_x64, false,
+               "Inject 64-bit DynamoRIO into 32-bit child processes.")
+#    endif
 OPTION_COMMAND(bool, x86_to_x64, false, "x86_to_x64",
                {
                    /* i#1494: to avoid decode_fragment messing up the 32-bit/64-bit mode,
@@ -1694,10 +1699,12 @@ DYNAMIC_OPTION_DEFAULT(
  * even _init is run needs to have a non-early default.
  * Thus we turn this on in privload_early_inject.
  */
-OPTION_COMMAND(bool, early_inject,
-               IF_WINDOWS_ELSE
-               /* i#980: too early for kernel32 so we disable */
-               (IF_CLIENT_INTERFACE_ELSE(false, true), false),
+/* On Windows this does *not* imply early injection anymore: it just enables control
+ * over where to inject via a hook and alternate injection methods, rather than using
+ * the old thread injection.
+ * XXX: Clean up by removing this option and thread injection completely?
+ */
+OPTION_COMMAND(bool, early_inject, IF_UNIX_ELSE(false /*see above*/, true),
                "early_inject",
                {
                    if (options->early_inject) {
@@ -1706,20 +1713,16 @@ OPTION_COMMAND(bool, early_inject,
                    }
                },
                "inject early", STATIC, OP_PCACHE_GLOBAL)
-#if 0 /* FIXME i#234 NYI: not ready to enable just yet */
-    OPTION_DEFAULT(bool, early_inject_map, true, "inject earliest via map")
-    /* see enum definition is os_shared.h for notes on what works with which
-     * os version */
-    OPTION_DEFAULT(uint, early_inject_location, 5 /* INJECT_LOCATION_KiUserApc */,
-        "where to hook for early_injection.  default is earliest injection: anything else will be later.")
-#else
-OPTION_DEFAULT(bool, early_inject_map, false, "inject earliest via map")
-/* see enum definition is os_shared.h for notes on what works with which
- * os version */
-OPTION_DEFAULT(uint, early_inject_location, 4 /* INJECT_LOCATION_LdrDefault */,
-               "where to hook for early_injection.  default is earliest injection: "
-               "anything else will be later.")
-#endif
+/* To support cross-arch follow-children injection we need to use the map option. */
+OPTION_DEFAULT(bool, early_inject_map, true, "inject earliest via map")
+/* See enum definition is os_shared.h for notes on what works with which
+ * os version.  Our default is late injection to make it easier on clients
+ * (as noted in i#980, we don't want to be too early for a private kernel32).
+ */
+OPTION_DEFAULT(uint, early_inject_location, 7 /* INJECT_LOCATION_ImageEntry */,
+               "where to hook for early_injection.  Use 5 =="
+               "INJECT_LOCATION_KiUserApcdefault for earliest injection; use "
+               "4 == INJECT_LOCATION_LdrDefault for easier-but-still-early.")
 OPTION_DEFAULT(uint_addr, early_inject_address, 0,
                "specify the address to hook at for INJECT_LOCATION_LdrCustom")
 #ifdef WINDOWS /* probably the surrounding options should also be under this ifdef */

core/os_shared.h

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2010-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2010-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2003-2010 VMware, Inc.  All rights reserved.
  * **********************************************************/
 
@@ -636,6 +636,7 @@ typedef struct _dr_mem_info_t {
  * It uses a function call so be careful where performance is critical.
  */
 #define PAGE_START(x) (((ptr_uint_t)(x)) & ~(os_page_size() - 1))
+#define PAGE_START64(x) (((uint64)(x)) & ~((uint64)os_page_size() - 1))
 
 size_t
 os_page_size(void);
@@ -1256,11 +1257,9 @@ enum {
     JMP_REL32_OPCODE = 0xe9,
     JMP_REL32_SIZE = 5, /* size in bytes of 32-bit rel jmp */
     CALL_REL32_OPCODE = 0xe8,
-#    ifdef X64
     JMP_ABS_IND64_OPCODE = 0xff,
     JMP_ABS_IND64_SIZE = 6, /* size in bytes of a 64-bit abs ind jmp */
     JMP_ABS_MEM_IND64_MODRM = 0x25,
-#    endif
 };
 #elif defined(AARCHXX)
 enum {

core/vmareas.c

@@ -1,5 +1,5 @@
 /* **********************************************************
- * Copyright (c) 2010-2020 Google, Inc.  All rights reserved.
+ * Copyright (c) 2010-2021 Google, Inc.  All rights reserved.
  * Copyright (c) 2002-2010 VMware, Inc.  All rights reserved.
  * **********************************************************/
 
@@ -6523,9 +6523,10 @@ app_memory_protection_change_internal(dcontext_t *dcontext, bool update_areas,
             os_terminate(dcontext, TERMINATE_PROCESS);
             ASSERT_NOT_REACHED();
         } else {
-            SYSLOG_INTERNAL_WARNING_ONCE("Application changing protections of "
-                                         "%s memory at least once (" PFX "-" PFX ")",
-                                         target_area_name, base, base + size);
+            /* On Win10 this happens in every run so we do not syslog. */
+            LOG(THREAD, LOG_VMAREAS, 1,
+                "Application changing protections of %s memory (" PFX "-" PFX ")",
+                target_area_name, base, base + size);
             if (how_handle == DR_MODIFY_NOP) {
                 /* we use a separate list, rather than a flag on DR areas, as the
                  * affected region could include non-DR memory

core/win32/events.mc

@@ -1,5 +1,5 @@
 ;// **********************************************************
-;// Copyright (c) 2012-2020 Google, Inc.  All rights reserved.
+;// Copyright (c) 2012-2021 Google, Inc.  All rights reserved.
 ;// Copyright (c) 2003-2010 VMware, Inc.  All rights reserved.
 ;// **********************************************************
 
@@ -702,4 +702,20 @@ Language=English
 Application %1!s! (%2!s!). Private library static TLS limit crossed: %3!s!
 .
 
+MessageId =
+Severity = Error
+Facility = DRCore
+SymbolicName = MSG_INJECTION_LIBRARY_MISSING
+Language=English
+Application %1!s! (%2!s!). The library %3!s! for child process injection is missing.
+.
+
+MessageId =
+Severity = Error
+Facility = DRCore
+SymbolicName = MSG_FOLLOW_CHILD_FAILED
+Language=English
+Application %1!s! (%2!s!). Failed to follow into child process: %3!s!.
+.
+
 ;// ADD NEW MESSAGES HERE