port to Android master issue

[zhaoqin]

To port DynamoRIO to Android, we need build DynamoRIO with Android ndk, and some other challenges. This is a container case for the Android port.

[zhaoqin]

It looks like the ptrace on Android is different from Linux, so we may need re-implement ptrace based injector on Android.

[derekbruening]

drgui fails to build. We should probably just disable it as it doesn't seem likely to be used on Android:

[ 82%] Building CXX object ext/drgui/CMakeFiles/drgui.dir/drgui_options_window.cpp.o
In file included from /usr/include/qt5/QtCore/qglobal.h:51:0,
                 from /usr/include/qt5/QtGui/qwindowdefs.h:37,
                 from /usr/include/qt5/QtWidgets/qwidget.h:37,
                 from /usr/include/qt5/QtWidgets/qframe.h:37,
                 from /usr/include/qt5/QtWidgets/qabstractscrollarea.h:37,
                 from /usr/include/qt5/QtWidgets/qabstractitemview.h:37,
                 from /usr/include/qt5/QtWidgets/qlistview.h:37,
                 from /usr/include/qt5/QtWidgets/qlistwidget.h:37,
                 from /usr/include/qt5/QtWidgets/QListWidget:1,
                 from /work/dr/git/src/ext/drgui/drgui_options_window.cpp:43:
/usr/include/qt5/QtCore/qconfig.h:9:27: fatal error: bits/wordsize.h: No such file or directory
 #include <bits/wordsize.h>

[derekbruening]

For the config dir, since $HOME is read-only and /tmp does not exist, and we do not really want to query at the Java level for the cache dir, the plan is to require cwd to be writable. We'll agument drconfiglib to check for writability (just a stat) and go to the next choice if so. I also plan to replace "/tmp" with "/data/local/tmp" as the latter is often (but not always) avail on Android.

[derekbruening]

To support a private libc, we need to emulate what the Android linker does (or else build our own version of libc). It has a special class object that provides convenience access to the args, env vars, and auxv, and it passes it to libc in a special call to __libc_init_tls() prior to calling any other libc init routine. libc stores it into TLS and accesses it from there from its regular init routines.

[derekbruening]

We need to make every executable PIE. Today there is a piecemeal approach of marking target flags which results in most things NOT being PIE:

> for i in bin32/drrun bin32/drconfig bin32/drinject bin32/runstats bin32/run_in_bg ext/drsyms_bench clients/bin32/drcov2lcov clients/bin32/drcachesim clients/bin32/drcpusim_ops api/bin/tracedump; do printf "%-30s %s\n" $i "`readelf -l $i | grep type`"; done
bin32/drrun                    Elf file type is DYN (Shared object file)
bin32/drconfig                 Elf file type is EXEC (Executable file)
bin32/drinject                 Elf file type is EXEC (Executable file)
bin32/runstats                 Elf file type is EXEC (Executable file)
bin32/run_in_bg                Elf file type is EXEC (Executable file)
ext/drsyms_bench               Elf file type is EXEC (Executable file)
clients/bin32/drcov2lcov       Elf file type is DYN (Shared object file)
clients/bin32/drcachesim       Elf file type is EXEC (Executable file)
clients/bin32/drcpusim_ops     Elf file type is EXEC (Executable file)
api/bin/tracedump              Elf file type is EXEC (Executable file)

I am removing those target flags and changing the global rules in one place which gives us:

> for i in bin32/drrun bin32/drconfig bin32/drinject bin32/runstats bin32/run_in_bg ext/drsyms_bench clients/bin32/drcov2lcov clients/bin32/drcachesim clients/bin32/drcpusim_ops api/bin/tracedump; do printf "%-30s %s\n" $i "`readelf -l $i | grep type`"; done
bin32/drrun                    Elf file type is DYN (Shared object file)
bin32/drconfig                 Elf file type is DYN (Shared object file)
bin32/drinject                 Elf file type is DYN (Shared object file)
bin32/runstats                 Elf file type is DYN (Shared object file)
bin32/run_in_bg                Elf file type is DYN (Shared object file)
ext/drsyms_bench               Elf file type is DYN (Shared object file)
clients/bin32/drcov2lcov       Elf file type is DYN (Shared object file)
clients/bin32/drcachesim       Elf file type is DYN (Shared object file)
clients/bin32/drcpusim_ops     Elf file type is DYN (Shared object file)
api/bin/tracedump              Elf file type is DYN (Shared object file)

[derekbruening]

Emulating the special setup the Android dynamic linker does for Bionic turns out to not be too difficult, but it does add fragile assumptions about the precise internal pthread data structure (the linker initializes a number of fields deep in the struct for the initial thread) in addition to the layout of the kernel arg class described above. We'll have to live with these dependences.

Once that works, we hit an issue where the libdl.so fini array function crashes on exit. It looks like libdl.so's fini array points into libc.so:

# bin32/drrun -debug -loglevel 4 -c api/bin/libempty.so -- ../hello.android

privload_process_imports: libc.so imports from libdl.so

privload_load_finalize: loaded libdl.so @ 0xb6ef1000-0xb6ef5000 from /system/lib/libdl.so
privload_load_finalize: loaded libc.so @ 0xb6ef6000-0xb6f6c000 from /system/lib/libc.so

privload_unload: unloading libc.so @ 0xb6ef6000
privload_call_entry: calling fini routines of libc.so

privload_unload: unloading libdl.so @ 0xb6ef1000
privload_call_entry: calling fini routines of libdl.so
privload_call_entry: calling libdl.so fini array func 0xb6ef156c

(gdb) bt
#0  0xb6f2ecc0 in __cxa_finalize ()
#1  0xb6ef1584 in ?? ()
(gdb) x/20i $pc
=> 0xb6f2ecc0 <__cxa_finalize>: Cannot access memory at address 0xb6f2ecc0
(gdb) x/20i 0xb6ef156c
   0xb6ef156c:  push    {r11, lr}
   0xb6ef1570:  add     r11, sp, #4
   0xb6ef1574:  ldr     r3, [pc, #12]   ; 0xb6ef1588
   0xb6ef1578:  add     r3, pc, r3
   0xb6ef157c:  mov     r0, r3
   0xb6ef1580:  bl      0xb6ef1518
   0xb6ef1584:  pop     {r11, pc}
(gdb) x/20i 0xb6ef1518
   0xb6ef1518:  add     r12, pc, #0, 12
   0xb6ef151c:  add     r12, r12, #8192 ; 0x2000
   0xb6ef1520:  ldr     pc, [r12, #2756]!       ; 0xac4

In any case, rather than trying to reorder their unmapping or sthg, I'm just going to skip libdl's fini calls on Android.

That's enough to get clients that use libc, such as libopcodes, running. Thus we'll consider this core issue finished and open separate issues for any further problems.