Android testbed fails to build for 32-bit targets #146541
Description
Bug report
Bug description:
Unfortunately, the Android testbed is not currently buildable for 32-bit targets because the 32-bit ABI names are missing from its build system.
The first error that occurs is this:
usage: android.py configure-host [-h] [-v] [--clean] HOST [args ...]
android.py configure-host: error: argument HOST: invalid choice: 'arm-linux-androideabi' (choose from 'aarch64-linux-android', 'x86_64-linux-android')
Another error that occurs is this:
Traceback (most recent call last):
File "<frozen runpy>", line 196, in _run_module_as_main
File "<frozen runpy>", line 87, in _run_code
File "/home/ubuntu/cpython/cross-build/arm-linux-androideabi/../../Lib/sysconfig/__main__.py", line 248, in <module>
_main()
~~~~~^^
File "/home/ubuntu/cpython/cross-build/arm-linux-androideabi/../../Lib/sysconfig/__main__.py", line 235, in _main
_generate_posix_vars()
~~~~~~~~~~~~~~~~~~~~^^
File "/home/ubuntu/cpython/cross-build/arm-linux-androideabi/../../Lib/sysconfig/__main__.py", line 197, in _generate_posix_vars
pybuilddir = _get_pybuilddir()
File "/home/ubuntu/cpython/cross-build/arm-linux-androideabi/../../Lib/sysconfig/__main__.py", line 138, in _get_pybuilddir
pybuilddir = f'build/lib.{get_platform()}-{get_python_version()}'
~~~~~~~~~~~~^^
File "/home/ubuntu/cpython/cross-build/arm-linux-androideabi/../../Lib/sysconfig/__init__.py", line 703, in get_platform
machine = {
~
...<4 lines>...
"armv8l": "armeabi_v7a",
~~~~~~~~~~~~~~~~~~~~~~~~
}[machine]
~^^^^^^^^^
KeyError: 'arm'
Applying several changes allows building the Android testbed for 32-bit targets and running it on them.
This Dockerfile can build the Android testbed for 32-bit ARM and 32-bit x86.
FROM ubuntu:24.04 AS build
# based on
# https://github.com/thunder-coding/containerTube/blob/93062c010527382b0eb5372e9fd0842cb9f9a3f5/python-android-builder/Dockerfile
# Install necessary packages we need to build Python
RUN <<DOCKEREOF
apt-get update && apt-get full-upgrade -y
# Needed for getting the sources and SDK
apt-get install -y git curl unzip
# For actually building Python
apt-get install -y build-essential python-is-python3 autoconf autoconf-archive m4 autotools-dev libffi-dev
# Needed for Android command line tools
apt-get install -y openjdk-21-jdk
# Needed for building zstd in cpython build system
apt-get install -y wget
DOCKEREOF
ADD https://dl.google.com/android/repository/commandlinetools-linux-14742923_latest.zip /root/cmdline-tools.zip
ADD https://github.com/python/cpython/archive/5992238986df094e890a89376970aab6058a0759.tar.gz /root/cpython.tar.gz
ADD https://github.com/beeware/cpython-android-source-deps/archive/refs/tags/zstd-1.5.7-1.tar.gz /root/cpython-android-source-deps.tar.gz
ENV ANDROID_HOME=/root/.android
WORKDIR /root
RUN <<DOCKEREOF
mkdir -p "$ANDROID_HOME/cmdline-tools"
unzip cmdline-tools.zip -d "$ANDROID_HOME/cmdline-tools"
mv "$ANDROID_HOME/cmdline-tools/cmdline-tools" "$ANDROID_HOME/cmdline-tools/latest"
mkdir -p cpython cpython-android-source-deps
tar xf cpython.tar.gz --strip-components=1 -C cpython
tar xf cpython-android-source-deps.tar.gz --strip-components=1 -C cpython-android-source-deps
DOCKEREOF
WORKDIR /root/cpython-android-source-deps
RUN <<DOCKEREOF
git apply -v << 'PATCHEOF'
--- a/android-env.sh
+++ b/android-env.sh
@@ -76,6 +76,13 @@ if [ "$HOST" = "arm-linux-androideabi" ]; then
CFLAGS="$CFLAGS -march=armv7-a -mthumb"
fi
+# prevents this error when linking to the zstd static library built for 32-bit x86:
+# ld.lld: error: relocation R_386_PC32 cannot be used
+# against symbol 'HUF_writeCTable_wksp'; recompile with -fPIC
+if [ "$HOST" = "i686-linux-android" ]; then
+ CFLAGS="$CFLAGS -fPIC"
+fi
+
if [ -n "${PREFIX:-}" ]; then
abs_prefix="$(realpath "$PREFIX")"
CFLAGS="$CFLAGS -I$abs_prefix/include"
PATCHEOF
DOCKEREOF
RUN ./build.sh zstd 1.5.7 2 i686-linux-android
WORKDIR /root/cpython
RUN <<DOCKEREOF
cat > /root/git-ls-files.log << 'EOF'
README.md
android-env.sh
android.py
testbed/.gitignore
testbed/.idea/inspectionProfiles/Project_Default.xml
testbed/app/.gitignore
testbed/app/build.gradle.kts
testbed/app/src/androidTest/java/org/python/testbed/PythonSuite.kt
testbed/app/src/main/AndroidManifest.xml
testbed/app/src/main/c/CMakeLists.txt
testbed/app/src/main/c/main_activity.c
testbed/app/src/main/java/org/python/testbed/MainActivity.kt
testbed/app/src/main/res/drawable-xxhdpi/ic_launcher.png
testbed/app/src/main/res/layout/activity_main.xml
testbed/app/src/main/res/values/strings.xml
testbed/build.gradle.kts
testbed/gradle.properties
testbed/gradle/wrapper/gradle-wrapper.properties
testbed/settings.gradle.kts
EOF
git apply -v << 'PATCHEOF'
--- a/Android/android.py
+++ b/Android/android.py
@@ -34,7 +34,7 @@
TESTBED_DIR = ANDROID_DIR / "testbed"
CROSS_BUILD_DIR = PYTHON_DIR / "cross-build"
-HOSTS = ["aarch64-linux-android", "x86_64-linux-android"]
+HOSTS = ["aarch64-linux-android", "x86_64-linux-android", "arm-linux-androideabi", "i686-linux-android"]
APP_ID = "org.python.testbed"
DECODE_ARGS = ("UTF-8", "backslashreplace")
@@ -743,7 +743,7 @@ def package(context):
# Include all tracked files from the Android directory.
for line in run(
- ["git", "ls-files"],
+ ["cat", "/root/git-ls-files.log"],
cwd=ANDROID_DIR, capture_output=True, text=True, log=False,
).stdout.splitlines():
src = ANDROID_DIR / line
--- a/Android/testbed/app/build.gradle.kts
+++ b/Android/testbed/app/build.gradle.kts
@@ -16,6 +16,8 @@ val inSourceTree = (
val KNOWN_ABIS = mapOf(
"aarch64-linux-android" to "arm64-v8a",
"x86_64-linux-android" to "x86_64",
+ "arm-linux-androideabi" to "armeabi-v7a",
+ "i686-linux-android" to "x86",
)
// Discover prefixes.
--- a/Lib/sysconfig/__init__.py
+++ b/Lib/sysconfig/__init__.py
@@ -697,12 +697,16 @@ def get_platform():
# When Python is running on 32-bit ARM Android on a 64-bit ARM kernel,
# 'os.uname().machine' is 'armv8l'. Such devices run the same userspace
# code as 'armv7l' devices.
+ # During the build process of the Android testbed when targeting 32-bit ARM,
+ # '_PYTHON_HOST_PLATFORM' is 'arm-linux-androideabi', so 'machine' becomes
+ # 'arm'.
machine = {
"x86_64": "x86_64",
"i686": "x86",
"aarch64": "arm64_v8a",
"armv7l": "armeabi_v7a",
"armv8l": "armeabi_v7a",
+ "arm": "armeabi_v7a",
}[machine]
elif osname == "linux":
# At least on Linux/Intel, 'machine' is the processor --
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -376,6 +376,7 @@ def test_get_platform(self):
'aarch64': 'arm64_v8a',
'armv7l': 'armeabi_v7a',
'armv8l': 'armeabi_v7a',
+ 'arm': 'armeabi_v7a',
}.items():
with self.subTest(machine):
self._set_uname(('Linux', 'localhost', '3.18.91+',
@@ -585,6 +586,7 @@ def test_android_ext_suffix(self):
"aarch64": "aarch64-linux-android",
"armv7l": "arm-linux-androideabi",
"armv8l": "arm-linux-androideabi",
+ "arm": "arm-linux-androideabi",
}[machine]
self.assertEndsWith(suffix, f"-{expected_triplet}.so")
PATCHEOF
DOCKEREOF
WORKDIR /root/cpython/Android
RUN <<DOCKEREOF
./android.py configure-build
./android.py make-build
DOCKEREOF
RUN <<DOCKEREOF
./android.py configure-host arm-linux-androideabi
./android.py make-host arm-linux-androideabi
./android.py package arm-linux-androideabi
mv ../cross-build/arm-linux-androideabi/dist/python-*-arm-linux-androideabi.tar.gz \
/root/python-arm-linux-androideabi.tar.gz
DOCKEREOF
WORKDIR /root/cpython
RUN <<DOCKEREOF
git apply -v << 'PATCHEOF'
--- a/Android/android.py
+++ b/Android/android.py
@@ -214,6 +214,8 @@ def unpack_deps(host, prefix_dir):
download(f"{deps_url}/{name_ver}/{filename}")
shutil.unpack_archive(filename)
os.remove(filename)
+ shutil.copyfile("/root/cpython-android-source-deps/zstd/build/1.5.7/i686-linux-android/prefix/lib/libzstd.a",
+ "/root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a")
def download(url, target_dir="."):
PATCHEOF
DOCKEREOF
WORKDIR /root/cpython/Android
RUN <<DOCKEREOF
./android.py configure-host i686-linux-android
./android.py make-host i686-linux-android
./android.py package i686-linux-android
mv ../cross-build/i686-linux-android/dist/python-*-i686-linux-android.tar.gz \
/root/python-i686-linux-android.tar.gz
DOCKEREOF
FROM scratch
COPY --from=build /root/python-arm-linux-androideabi.tar.gz /
COPY --from=build /root/python-i686-linux-android.tar.gz /There is also an error specific to 32-bit x86 that occurs because of the https://github.com/beeware/cpython-android-source-deps repository compiling libzstd.a for 32-bit x86 without applying the -fPIC argument to the compiler:
ld.lld: error: relocation R_386_PC32 cannot be used against symbol 'HUF_writeCTable_wksp'; recompile with -fPIC
>>> defined in /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a(huf_compress.o)
>>> referenced by huf_compress.c
>>> huf_compress.o:(HUF_optimalTableLog) in archive /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a
The way to fix that error would be in the cpython-android-source-deps repository, so I have opened a separate issue and PR for that here:
Then, I was able to use these commands to build, extract and run the python-arm-linux-androideabi.tar.gz on my 32-bit ARM device connected to ADB, with ID 4251484430323498.
docker build --output=. .
tar xf python-arm-linux-androideabi.tar.gz
./android.py test --connected 4251484430323498CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux