[libc++][Android] Support libc++ testing on Android (#69274)

I could probably break this commit into more pieces.

---

This patch adds libc++ support for Android L (Android 5.0+) and up,
tested using the Android team's current compiler, a recent version of
the AOSP sysroot, and the x86[-64] Android Emulator.

CMake and Lit Configuration:

Add runtimes/cmake/android/Arch-${ARCH}.cmake files that configure CMake
to cross-compile to Android without using CMake's built-in NDK support
(which only works with an actual packaged NDK).

Add libcxx/cmake/caches/AndroidNDK.cmake that builds and tests libc++
(and libc++abi) for Android. This file configures libc++ to match what
the NDK distributes, e.g.:
- libc++_shared.so (includes libc++abi objects, there is no
libc++abi.so). libunwind is linked statically but not exported.
 - libc++_static.a (does not include libc++abi) and libc++abi.a
 - `std::__ndk1` namespace
- All the libraries are built with `__ANDROID_API__=21`, even when they
are linked to something targeting a higher API level.

(However, when the Android LLVM team builds these components, they do
not use these CMake cache files. Instead they use Python scripts to
configure the builds. See
https://android.googlesource.com/toolchain/llvm_android/.)

Add llvm-libc++[abi].android-ndk.cfg.in files that test the Android
NDK's libc++_shared.so. These files can target old or new Android
devices. The Android LLVM team uses these test files to test libc++ for
both arm/arm64 and x86/x86_64 architectures.

The Android testing mode works by setting %{executor} to adb_run.py,
which uses `adb push` and `adb shell` to run tests remotely. adb_run.py
always runs tests as the "shell" user even on an old emulator where "adb
unroot" doesn't work. The script has workarounds for old Android
devices. The script uses a Unix domain socket on the host
(--job-limit-socket) to restrict concurrent adb invocations. Compiling
the tests is a major part of libc++ testing run-time, so it's desirable
to exploit all the host cores without overburdening the test devices,
which can have far fewer cores.

BuildKite CI:

Add a builder to run-buildbot, `android-ndk-*`, that uses Android Clang
and an Android sysroot to build libc++, then starts an Android emulator
container to run tests.

Run the emulator and an adb server in a separate Docker container
(libcxx-ci-android-emulator), and create a separate Docker image for
each emulator OS system image. Set ADB_SERVER_SOCKET to connect to the
container's adb server. Running the only adb server inside the container
makes cleanup more reliable between test runs, e.g. the adb client
doesn't create a `~/.android` directory and the adb server can be
restarted along with the emulator using docker stop/run. (N.B. The
emulator insists on connecting to an adb server and will start one
itself if it can't connect to one.)

The suffix to the android-ndk-* job is a label that concisely specifies
an Android SDK emulator image. e.g.:
 - "system-images;android-21;default;x86" ==> 21-def-x86
 - "system-images;android-33;google_apis;x86_64" ==> 33-goog-x86_64

Fixes: #69270
Differential Revision: https://reviews.llvm.org/D139147

libcxx/cmake/caches/AndroidNDK.cmake

@@ -0,0 +1,38 @@
+# Build libc++abi and libc++ closely resembling what is shipped in the Android
+# NDK.
+
+# The NDK names the libraries libc++_shared.so and libc++_static.a. Using the
+# libc++_shared.so soname ensures that the library doesn't interact with the
+# libc++.so in /system/lib[64].
+set(LIBCXX_SHARED_OUTPUT_NAME c++_shared CACHE STRING "")
+set(LIBCXX_STATIC_OUTPUT_NAME c++_static CACHE STRING "")
+
+# The NDK libc++ uses a special namespace to help isolate its symbols from those
+# in the platform's STL (e.g. /system/lib[64]/libc++.so, but possibly stlport on
+# older versions of Android).
+set(LIBCXX_ABI_VERSION 1 CACHE STRING "")
+set(LIBCXX_ABI_NAMESPACE __ndk1 CACHE STRING "")
+
+# CMake doesn't add a version suffix to an Android shared object filename,
+# (because CMAKE_PLATFORM_NO_VERSIONED_SONAME is set), so it writes both a
+# libc++_shared.so ELF file and a libc++_shared.so linker script to the same
+# output path (the script clobbers the binary). Turn off the linker script.
+set(LIBCXX_ENABLE_ABI_LINKER_SCRIPT OFF CACHE BOOL "")
+
+set(LIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY ON CACHE BOOL "")
+set(LIBCXXABI_ENABLE_SHARED OFF CACHE BOOL "")
+
+# Clang links libc++ by default, but it doesn't exist yet. The libc++ CMake
+# files specify -nostdlib++ to avoid this problem, but CMake's default "compiler
+# works" testing doesn't pass that flag, so force those tests to pass.
+set(CMAKE_C_COMPILER_WORKS ON CACHE BOOL "")
+set(CMAKE_CXX_COMPILER_WORKS ON CACHE BOOL "")
+
+# Use adb to push tests to a locally-connected device (e.g. emulator) and run
+# them.
+set(LIBCXX_TEST_CONFIG "llvm-libc++-android-ndk.cfg.in" CACHE STRING "")
+set(LIBCXXABI_TEST_CONFIG "llvm-libc++abi-android-ndk.cfg.in" CACHE STRING "")
+
+# CMAKE_SOURCE_DIR refers to the "<monorepo>/runtimes" directory.
+set(LIBCXX_EXECUTOR "${CMAKE_SOURCE_DIR}/../libcxx/utils/adb_run.py" CACHE STRING "")
+set(LIBCXXABI_EXECUTOR "${LIBCXX_EXECUTOR}" CACHE STRING "")

libcxx/docs/index.rst

@@ -130,6 +130,7 @@ Target platform Target architecture       Notes
 macOS 10.9+     i386, x86_64, arm64       Building the shared library itself requires targetting macOS 10.13+
 FreeBSD 12+     i386, x86_64, arm
 Linux           i386, x86_64, arm, arm64  Only glibc-2.24 and later and no other libc is officially supported
+Android 5.0+    i386, x86_64, arm, arm64
 Windows         i386, x86_64              Both MSVC and MinGW style environments, ABI in MSVC environments is :doc:`unstable <DesignDocs/ABIVersioning>`
 AIX 7.2TL5+     powerpc, powerpc64
 =============== ========================= ============================

libcxx/test/configs/llvm-libc++-android-ndk.cfg.in

@@ -0,0 +1,47 @@
+# This testing configuration handles running the test suite against LLVM's
+# libc++ using adb and a libc++_shared.so library on Android.
+
+lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')
+
+import re
+import site
+
+site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
+
+import libcxx.test.android
+import libcxx.test.config
+import libcxx.test.params
+
+config.substitutions.append(('%{flags}',
+    '--sysroot @CMAKE_SYSROOT@' if '@CMAKE_SYSROOT@' else ''
+))
+
+compile_flags = '-nostdinc++ -I %{include} -I %{target-include} -I %{libcxx}/test/support'
+if re.match(r'i686-linux-android(21|22|23)$', config.target_triple):
+    # 32-bit x86 Android has a bug where the stack is sometimes misaligned.
+    # The problem appears limited to versions before Android N (API 24) and only
+    # __attribute__((constructor)) functions. Compile with -mstackrealign to
+    # work around the bug.
+    # TODO: Consider automatically doing something like this in Clang itself (LIBCXX-ANDROID-FIXME)
+    # See https://github.com/android/ndk/issues/693.
+    compile_flags += ' -mstackrealign'
+config.substitutions.append(('%{compile_flags}', compile_flags))
+
+# The NDK library is called "libc++_shared.so". Use LD_LIBRARY_PATH to find
+# libc++_shared.so because older Bionic dynamic loaders don't support rpath
+# lookup.
+config.substitutions.append(('%{link_flags}',
+    '-nostdlib++ -L %{lib} -lc++_shared'
+))
+config.substitutions.append(('%{exec}',
+    '%{executor}' +
+    ' --job-limit-socket ' + libcxx.test.android.adb_job_limit_socket() +
+    ' --prepend-path-env LD_LIBRARY_PATH /data/local/tmp/libc++ --execdir %T -- '
+))
+
+libcxx.test.config.configure(
+    libcxx.test.params.DEFAULT_PARAMETERS,
+    libcxx.test.features.DEFAULT_FEATURES,
+    config,
+    lit_config
+)

libcxx/utils/adb_run.py

@@ -0,0 +1,256 @@
+#!/usr/bin/env python3
+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+"""adb_run.py is a utility for running a libc++ test program via adb.
+"""
+
+import argparse
+import hashlib
+import os
+import re
+import shlex
+import socket
+import subprocess
+import sys
+from typing import List, Optional, Tuple
+
+
+# Sync a host file /path/to/dir/file to ${REMOTE_BASE_DIR}/run-${HASH}/dir/file.
+REMOTE_BASE_DIR = "/data/local/tmp/adb_run"
+
+g_job_limit_socket = None
+g_verbose = False
+
+
+def run_adb_sync_command(command: List[str]) -> None:
+    """Run an adb command and discard the output, unless the command fails. If
+    the command fails, dump the output instead, and exit the script with
+    failure.
+    """
+    if g_verbose:
+        sys.stderr.write(f"running: {shlex.join(command)}\n")
+    proc = subprocess.run(command, universal_newlines=True,
+                          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+                          stderr=subprocess.STDOUT, encoding="utf-8")
+    if proc.returncode != 0:
+        # adb's stdout (e.g. for adb push) should normally be discarded, but
+        # on failure, it should be shown. Print it to stderr because it's
+        # unrelated to the test program's stdout output. A common error caught
+        # here is "No space left on device".
+        sys.stderr.write(f"{proc.stdout}\n"
+                         f"error: adb command exited with {proc.returncode}: "
+                         f"{shlex.join(command)}\n")
+        sys.exit(proc.returncode)
+
+
+def sync_test_dir(local_dir: str, remote_dir: str) -> None:
+    """Sync the libc++ test directory on the host to the remote device."""
+
+    # Optimization: The typical libc++ test directory has only a single
+    # *.tmp.exe file in it. In that case, skip the `mkdir` command, which is
+    # normally necessary because we don't know if the target directory already
+    # exists on the device.
+    local_files = os.listdir(local_dir)
+    if len(local_files) == 1:
+        local_file = os.path.join(local_dir, local_files[0])
+        remote_file = os.path.join(remote_dir, local_files[0])
+        if not os.path.islink(local_file) and os.path.isfile(local_file):
+            run_adb_sync_command(["adb", "push", "--sync", local_file,
+                                  remote_file])
+            return
+
+    assert os.path.basename(local_dir) == os.path.basename(remote_dir)
+    run_adb_sync_command(["adb", "shell", "mkdir", "-p", remote_dir])
+    run_adb_sync_command(["adb", "push", "--sync", local_dir,
+                          os.path.dirname(remote_dir)])
+
+
+def build_env_arg(env_args: List[str], prepend_path_args: List[Tuple[str, str]]) -> str:
+    components = []
+    for arg in env_args:
+        k, v = arg.split("=", 1)
+        components.append(f"export {k}={shlex.quote(v)}; ")
+    for k, v in prepend_path_args:
+        components.append(f"export {k}={shlex.quote(v)}${{{k}:+:${k}}}; ")
+    return "".join(components)
+
+
+def run_command(args: argparse.Namespace) -> int:
+    local_dir = args.execdir
+    assert local_dir.startswith("/")
+    assert not local_dir.endswith("/")
+
+    # Copy each execdir to a subdir of REMOTE_BASE_DIR. Name the directory using
+    # a hash of local_dir so that concurrent adb_run invocations don't create
+    # the same intermediate parent directory. At least `adb push` has trouble
+    # with concurrent mkdir syscalls on common parent directories. (Somehow
+    # mkdir fails with EAGAIN/EWOULDBLOCK, see internal Google bug,
+    # b/289311228.)
+    local_dir_hash = hashlib.sha1(local_dir.encode()).hexdigest()
+    remote_dir = f"{REMOTE_BASE_DIR}/run-{local_dir_hash}/{os.path.basename(local_dir)}"
+    sync_test_dir(local_dir, remote_dir)
+
+    adb_shell_command = (
+        # Set the environment early so that PATH can be overridden. Overriding
+        # PATH is useful for:
+        #  - Replacing older shell utilities with toybox (e.g. on old devices).
+        #  - Adding a `bash` command that delegates to `sh` (mksh).
+        f"{build_env_arg(args.env, args.prepend_path_env)}"
+
+        # Set a high oom_score_adj so that, if the test program uses too much
+        # memory, it is killed before anything else on the device. The default
+        # oom_score_adj is -1000, so a test using too much memory typically
+        # crashes the device.
+        "echo 1000 >/proc/self/oom_score_adj; "
+
+        # If we're running as root, switch to the shell user. The libc++
+        # filesystem tests require running without root permissions. Some x86
+        # emulator devices (before Android N) do not have a working `adb unroot`
+        # and always run as root. Non-debug builds typically lack `su` and only
+        # run as the shell user.
+        #
+        # Some libc++ tests create temporary files in the working directory,
+        # which might be owned by root. Before switching to shell, make the
+        # cwd writable (and readable+executable) to every user.
+        #
+        # N.B.:
+        #  - Avoid "id -u" because it wasn't supported until Android M.
+        #  - The `env` and `which` commands were also added in Android M.
+        #  - Starting in Android M, su from root->shell resets PATH, so we need
+        #    to modify it again in the new environment.
+        #  - Avoid chmod's "a+rwx" syntax because it's not supported until
+        #    Android N.
+        #  - Defining this function allows specifying the arguments to the test
+        #    program (i.e. "$@") only once.
+        "run_without_root() {"
+        "  chmod 777 .;"
+        "  case \"$(id)\" in"
+        "    *\"uid=0(root)\"*)"
+        "    if command -v env >/dev/null; then"
+        "      su shell \"$(command -v env)\" PATH=\"$PATH\" \"$@\";"
+        "    else"
+        "      su shell \"$@\";"
+        "    fi;;"
+        "    *) \"$@\";;"
+        "  esac;"
+        "}; "
+    )
+
+    # Older versions of Bionic limit the length of argv[0] to 127 bytes
+    # (SOINFO_NAME_LEN-1), and the path to libc++ tests tend to exceed this
+    # limit. Changing the working directory works around this limit. The limit
+    # is increased to 4095 (PATH_MAX-1) in Android M (API 23).
+    command_line = [arg.replace(local_dir + "/", "./") for arg in args.command]
+
+    # Prior to the adb feature "shell_v2" (added in Android N), `adb shell`
+    # always created a pty:
+    #  - This merged stdout and stderr together.
+    #  - The pty converts LF to CRLF.
+    #  - The exit code of the shell command wasn't propagated.
+    # Work around all three limitations, unless "shell_v2" is present.
+    proc = subprocess.run(["adb", "features"], check=True,
+                          stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
+                          encoding="utf-8")
+    adb_features = set(proc.stdout.strip().split())
+    has_shell_v2 = "shell_v2" in adb_features
+    if has_shell_v2:
+        adb_shell_command += (
+            f"cd {remote_dir} && run_without_root {shlex.join(command_line)}"
+        )
+    else:
+        adb_shell_command += (
+            f"{{"
+            f"  stdout=$("
+            f"    cd {remote_dir} && run_without_root {shlex.join(command_line)};"
+            f"    echo -n __libcxx_adb_exit__=$?"
+            f"  ); "
+            f"}} 2>&1; "
+            f"echo -n __libcxx_adb_stdout__\"$stdout\""
+        )
+
+    adb_command_line = ["adb", "shell", adb_shell_command]
+    if g_verbose:
+        sys.stderr.write(f"running: {shlex.join(adb_command_line)}\n")
+
+    if has_shell_v2:
+        proc = subprocess.run(adb_command_line, shell=False, check=False,
+                              encoding="utf-8")
+        return proc.returncode
+    else:
+        proc = subprocess.run(adb_command_line, shell=False, check=False,
+                              stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                              encoding="utf-8")
+        # The old `adb shell` mode used a pty, which converted LF to CRLF.
+        # Convert it back.
+        output = proc.stdout.replace("\r\n", "\n")
+
+        if proc.returncode:
+            sys.stderr.write(f"error: adb failed:\n"
+                             f"  command: {shlex.join(adb_command_line)}\n"
+                             f"  output: {output}\n")
+            return proc.returncode
+
+        match = re.match(r"(.*)__libcxx_adb_stdout__(.*)__libcxx_adb_exit__=(\d+)$",
+                     output, re.DOTALL)
+        if not match:
+            sys.stderr.write(f"error: could not parse adb output:\n"
+                             f"  command: {shlex.join(adb_command_line)}\n"
+                             f"  output: {output}\n")
+            return 1
+
+        sys.stderr.write(match.group(1))
+        sys.stdout.write(match.group(2))
+        return int(match.group(3))
+
+
+def connect_to_job_limiter_server(sock_addr: str) -> None:
+    sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+
+    try:
+        sock.connect(sock_addr)
+    except (FileNotFoundError, ConnectionRefusedError) as e:
+        # Copying-and-pasting an adb_run.py command-line from a lit test failure
+        # is likely to fail because the socket no longer exists (or is
+        # inactive), so just give a warning.
+        sys.stderr.write(f"warning: could not connect to {sock_addr}: {e}\n")
+        return
+
+    # The connect call can succeed before the server has called accept, because
+    # of the listen backlog, so wait for the server to send a byte.
+    sock.recv(1)
+
+    # Keep the socket open until this process ends, then let the OS close the
+    # connection automatically.
+    global g_job_limit_socket
+    g_job_limit_socket = sock
+
+
+def main() -> int:
+    """Main function (pylint wants this docstring)."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--execdir", type=str, required=True)
+    parser.add_argument("--env", type=str, required=False, action="append",
+                        default=[], metavar="NAME=VALUE")
+    parser.add_argument("--prepend-path-env", type=str, nargs=2, required=False,
+                        action="append", default=[],
+                        metavar=("NAME", "PATH"))
+    parser.add_argument("--job-limit-socket")
+    parser.add_argument("--verbose", "-v", default=False, action="store_true")
+    parser.add_argument("command", nargs=argparse.ONE_OR_MORE)
+    args = parser.parse_args()
+
+    global g_verbose
+    g_verbose = args.verbose
+    if args.job_limit_socket is not None:
+        connect_to_job_limiter_server(args.job_limit_socket)
+    return run_command(args)
+
+
+if __name__ == '__main__':
+    sys.exit(main())

libcxx/utils/ci/BOT_OWNERS.txt

@@ -15,3 +15,8 @@ D: Armv7, Armv8, AArch64
 N: LLVM on Power
 E: powerllvm@ca.ibm.com
 D: AIX, ppc64le
+
+N: Android libc++
+E: rprichard@google.com
+H: rprichard
+D: Emulator-based x86[-64] libc++ CI testing