Meta-issue: support of the android platform

[xdegaye]

BPO	26865
Nosy	@ethanfurman, @xdegaye, @ericsnowcurrently, @Fak3, @zware, @moreati, @yan12125, @ethanhs
Dependencies	bpo-16255: subrocess.Popen needs /bin/sh but Android only has /system/bin/sh bpo-23968: rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET) bpo-26849: android does not support versioning in SONAME bpo-26851: android compilation and link flags bpo-26853: missing symbols in curses and readline modules on android bpo-26854: missing header on android for the ossaudiodev module bpo-26855: android: add platform.android_ver() bpo-26856: android does not have pwd.getpwall() bpo-26857: gethostbyname_r() is broken on android bpo-26858: setting SO_REUSEPORT fails at API 21 bpo-26862: android: SYS_getdents64 does not need to be defined on android API 21 bpo-26863: android lacks some declarations for the posix module bpo-26884: android: cross-compilation of extension module links to the wrong python library bpo-26918: android: test_pipes fails bpo-26919: on Android python fails to decode/encode command line arguments bpo-26920: android: test_sys fails bpo-26924: android: test_concurrent_futures fails bpo-26925: android: test_multiprocessing_main_handling fails bpo-26926: test_io large file test failure on 32 bits Android platforms bpo-26927: test_mmap does not handle ValueError when no large file support bpo-26928: bootlocale imports locale at startup on Android, causing test_site to fail bpo-26929: android: test_strptime fails bpo-26931: android: test_distutils fails bpo-26932: RTLD* macros are not defined on Android bpo-26933: android: test_posixpath fails bpo-26934: android: test_faulthandler fails bpo-26935: android: test_os fails bpo-26936: android: test_socket fails bpo-26937: the chown() method of the tarfile.TarFile class fails on Android bpo-26938: android: test_concurrent_futures hangs on armv7 bpo-26939: android: test_functools hangs on armv7 bpo-26940: android: test_importlib hangs on armv7 bpo-26941: android: test_threading hangs on armv7 bpo-26942: android: test_ctypes crashes on armv7 and aarch64 bpo-26944: test_posix: Android 'id -G' is entirely wrong or missing the effective gid bpo-27027: add the 'is_android' attribute to test.support bpo-27442: expose the Android API level in sysconfig.get_config_vars() bpo-27472: add the 'unix_shell' attribute to test.support bpo-27606: Android cross-built for armv5te with clang and '-mthumb' crashes with SIGSEGV or SIGILL bpo-27627: clang fails to build ctypes on Android armv7 bpo-27659: Prohibit implicit C function declarations bpo-28180: Implementation of the PEP 538: coerce C locale to C.utf-8 bpo-28538: _socket module cross-compilation error on android-24 bpo-28562: test_asyncio fails on Android upon calling getaddrinfo() bpo-28596: on Android _bootlocale on startup relies on too many library modules bpo-28662: catch also PermissionError in tests when spawning a non existent program bpo-28664: test_bz2 fails with BrokenPipeError when bunzip2 is missing bpo-28668: instanciation of multiprocessing.Queue raises ImportError in test_logging bpo-28683: bind() on a unix socket raises PermissionError on Android for a non-root user bpo-28684: [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user bpo-28740: Add sys.getandroidapilevel() bpo-28746: cannot set_inheritable() for a file descriptor on Android bpo-28759: access to mkfifo, mknod and hard links is controled by SELinux MAC on Android bpo-28762: lockf() is available now on Android API level 24, but the F_LOCK macro is not defined bpo-28764: test_mailbox fails when run as a non-root user on Android API 24 bpo-28996: wcscoll is broken on Android and test_locale fails bpo-28997: test_readline.test_nonascii fails on Android bpo-29040: building Android with android-ndk-r14 bpo-29176: /tmp does not exist on Android and is used by curses.window.putwin() bpo-29177: skip tests of test_logging when bind() raises PermissionError (non-root user on Android) bpo-29180: skip tests that raise PermissionError in test_os (non-root user on Android) bpo-29181: skip tests that raise PermissionError in test_tarfile (non-root user on Android) bpo-29184: skip tests of test_socketserver when bind() raises PermissionError (non-root user on Android) bpo-29185: test_distutils fails on Android API level 24 bpo-32031: Do not use the canonical path in pydoc test_mixed_case_module_names_are_lower_cased bpo-32059: detect_modules() in setup.py must also search the sysroot paths bpo-32126: [asyncio] test failure when the platform lacks a functional sem_open() bpo-32138: android: test_faulthandler fails also on API 24 bpo-32139: android: locale is modified by test_strftime bpo-32199: uuid.getnode() should return the MAC address on Android bpo-32202: [ctypes] all long double tests fail on android-24-x86_64 bpo-32203: [ctypes] test_struct_by_value fails on android-24-arm64 bpo-32205: test.pythoninfo does not print the cross-built sysconfig data bpo-32210: Add platform.android_ver() to test.pythoninfo for Android platforms bpo-32246: test_regrtest alters the execution environment on Android

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-04-26.15:36:51.967>
labels = ['type-feature', '3.7', 'build']
title = 'Meta-issue: support of the android platform'
updated_at = <Date 2017-12-10.09:31:57.501>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2017-12-10.09:31:57.501>
actor = 'xdegaye'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Cross-Build']
creation = <Date 2016-04-26.15:36:51.967>
creator = 'xdegaye'
dependencies = ['16255', '23968', '26849', '26851', '26853', '26854', '26855', '26856', '26857', '26858', '26862', '26863', '26884', '26918', '26919', '26920', '26924', '26925', '26926', '26927', '26928', '26929', '26931', '26932', '26933', '26934', '26935', '26936', '26937', '26938', '26939', '26940', '26941', '26942', '26944', '27027', '27442', '27472', '27606', '27627', '27659', '28180', '28538', '28562', '28596', '28662', '28664', '28668', '28683', '28684', '28740', '28746', '28759', '28762', '28764', '28996', '28997', '29040', '29176', '29177', '29180', '29181', '29184', '29185', '32031', '32059', '32126', '32138', '32139', '32199', '32202', '32203', '32205', '32210', '32246']
files = []
hgrepos = []
issue_num = 26865
keywords = []
message_count = 41.0
messages = ['264310', '264485', '264746', '264791', '265610', '265963', '265988', '266075', '266085', '269236', '269296', '269718', '270045', '270942', '271148', '271389', '271516', '271522', '271547', '271567', '271568', '271571', '271731', '279496', '279733', '279983', '280653', '281209', '281367', '282661', '283113', '283477', '283780', '284743', '284757', '284854', '285493', '285772', '307901', '307902', '307948']
nosy_count = 8.0
nosy_names = ['ethan.furman', 'xdegaye', 'eric.snow', 'Roman.Evstifeev', 'zach.ware', 'Alex.Willmer', 'yan12125', 'ethan smith']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26865'
versions = ['Python 3.7']

[xdegaye]

This issue lists issues that may have to be fixed in the perspective of a future support of the android platform.

build
issue bpo-26849: android does not support versioning in SONAME
issue bpo-26851: android compilation and link flags
issue bpo-26852: add a COMPILEALL_FLAGS Makefile variable

curses, readline
issue bpo-26853: missing symbols in curses and readline modules on android

ossaudiodev
issue bpo-26854: missing header on android for the ossaudiodev module

platform
issue bpo-16353: add function to os module for getting path to default shell
issue bpo-26855: add platform.android_ver() for android

pwd
issue bpo-26856: android does not have pwd.getpwall()

socketmodule
issue bpo-26857: gethostbyname_r() is broken on android

asyncio tests
issue bpo-26858: setting SO_REUSEPORT fails on android

unittest
issue bpo-26859: unittest fails with "Start directory is not importable"

posixmodule
issue bpo-26862: SYS_getdents64 does not need to be defined on android API 21
issue bpo-26863: android lacks some declaration for the posix module

[xdegaye]

build
issue bpo-26884: cross-compilation of extension module links to the wrong python library

This should be added to the meta-issue dependencies, I guess.

[xdegaye]

All the dependencies of this issue are tagged with Components:Cross-Build and a title starting with 'android'.

Tests that fail on an android emulator running an x86 system image at API level 21:

issue bpo-26918: android: test_pipes fails
issue bpo-26919: android: test_cmd_line fails
issue bpo-26920: android: test_sys fails
issue bpo-26924: android: test_concurrent_futures fails
issue bpo-26925: android: test_multiprocessing_main_handling fails
issue bpo-26926: android: test_io fails
issue bpo-26927: android: test_mmap fails
issue bpo-26928: android: test_site fails
issue bpo-26929: android: test_strptime fails
issue bpo-26931: android: test_distutils fails
issue bpo-26932: android: test_posix fails
issue bpo-26933: android: test_posixpath fails
issue bpo-26934: android: test_faulthandler fails
issue bpo-26935: android: test_os fails
issue bpo-26936: android: test_socket fails
issue bpo-26937: android: test_tarfile fails

Tests that fail on an android emulator running an armv7 system image (but not fail on x86) at API level 21:

issue bpo-26938: android: test_concurrent_futures hangs on armv7
issue bpo-26939: android: test_functools hangs on armv7
issue bpo-26940: android: test_importlib hangs on armv7
issue bpo-26941: android: test_threading hangs on armv7
issue bpo-26942: android: test_ctypes crashes on armv7

[xdegaye]

Tests that fail on an android emulator running an x86 system image at API level 21:

issue bpo-26944: android: test_posix fails

[xdegaye]

issue bpo-27027: add is_android in test.support to detect Android platform

[zware]

Xavier: It's been long enough that I don't remember whether regular Users can adjust Dependencies, but since you didn't do it yourself I assume they can't. I've just given you the Developer role, so you can now :)

[xdegaye]

No, regular users can't adjust dependencies, but I can now. Thanks Zachary.

[xdegaye]

issue bpo-22747: Interpreter fails in initialize on systems where HAVE_LANGINFO_H is undefined

[xdegaye]

issue bpo-16255: subrocess.Popen needs /bin/sh but Android only has /system/bin/sh
issue bpo-16353: add function to os module for getting path to default shell

[xdegaye]

issue bpo-23968: rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

[xdegaye]

issue bpo-22724: byte-compile fails for cross-builds

[xdegaye]

issue bpo-27442: expose Android API level in sys.implementation

[xdegaye]

issue bpo-27472: add the 'unix_shell' attribute to test.support

[xdegaye]

Larry asked in msg 270937:

Is there a plan to make Android a supported platform in 3.6?

This is the list of Android related issues that will modify the standard library and core Python when fixed:

bpo-16255: subrocess.Popen needs /bin/sh but Android only has /system/bin/sh

[1] bpo-26851: android compilation and link flags
bpo-22724: byte-compile fails for cross-builds
bpo-23968: rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)
bpo-26919: android: test_cmd_line fails
bpo-26920: android: test_sys fails
bpo-26928: _bootlocale imports locale at startup on Android, causing test_site to fail
bpo-26937: android: test_tarfile fails
bpo-26934: android: test_faulthandler fails
bpo-26929: android: test_strptime fails
[2] bpo-26942: android: test_ctypes crashes on armv7

All the other issues listed here in this meta-issue are changes to the test suite (mostly skipping some tests). A buildbot must also be setup for the Android emulators and maybe also for an Android device. That means installing the Android sdk and ndk and a build system on the buildbot.

[1] bpo-26851 is an enhancement.
[2] The root cause in bpo-26942 is unknown yet.

[xdegaye]

issue bpo-27606: Android cross-built for armv5te with clang and '-mthumb' crashes with SIGSEGV or SIGILL

[xdegaye]

issue bpo-27627: clang fails to build ctypes on Android armv7

[xdegaye]

issue bpo-27640: add the '--disable-test-suite' option to configure

[Fak3]

While not only android issue, there is a problem with dumbdbm module: it internally tries to do os.chmod() on a FAT-formatted sdcard and fails, because FAT does not support chmod.

[yan12125]

Some dependent issues, like bpo-26852, bpo-26859 and bpo-27640, are for reducing the size of an installation. How about moving them to another meta-issue? First they are not critical for normal usages on Android. Second they are not limited to Android - other platforms may benefit as well.

[xdegaye]

Roman, can you please enter a new issue for that problem.

[xdegaye]

Some dependent issues, like bpo-26852, bpo-26859 and bpo-27640, are for reducing the size of an installation. How about moving them to another meta-issue? First they are not critical for normal usages on Android. Second they are not limited to Android - other platforms may benefit as well.

Many issues that deal with failed tests are also not critical for normal usages on Android, but they must be fixed to run a buildbot and to support Android.
Issue bpo-22724 (byte-compile fails for cross-builds) is also not specific to Android.
IMHO this meta-issue collects (1) the issues that must be fixed for the support of Android and (2) the issues that should possibly be fixed. The issues you are naming and bpo-22724 are in the second category.

[yan12125]

Maybe the term "normal usages" is not accurate. I was trying to refer all possible Python usages on Android, and the test suite is a subset of them, so they should be fixed, too. I propose the aforementioned change (creating another meta-issue) because there are already many issues in this meta-issue, and I'm afraid new issues in the second category lead to complexity in tracking CPython's Android support. Currently things are still under control, so you can ignore my request until the number of issues goes crazy.

[xdegaye]

issue bpo-27659: Check for the existence of crypt()

[xdegaye]

issue bpo-28538: _socket module cross-compilation error on android-24

[xdegaye]

issue bpo-28562: test_asyncio fails on Android upon calling getaddrinfo()

[xdegaye]

issue bpo-28596: on Android _bootlocale on startup relies on too many library modules

[xdegaye]

issue bpo-28662: catch also PermissionError in tests when spawning a non existent program
issue bpo-28664: test_bz2 fails with BrokenPipeError when bunzip2 is missing
issue bpo-28668: instanciation of multiprocessing.Queue raises ImportError in test_logging

[xdegaye]

issue bpo-28740: Add sys.getandroidapilevel()

[xdegaye]

New issues raised upon testing on the Android API 24 emulator:
issue bpo-28683: bind() on a unix socket raises PermissionError on Android for a non-root user
issue bpo-28684: [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user
issue bpo-28746: cannot set_inheritable() for a file descriptor on Android
issue bpo-28759: access to mkfifo, mknod and hard links is controled by SELinux MAC on Android
issue bpo-28762: configure links with lockf and F_LOCK is not declared in Android API 24
issue bpo-28764: test_mailbox fails when run as a non-root user on Android API 24

[xdegaye]

issue bpo-20211: setup.py: do not add system header locations when cross compiling
issue bpo-28190: Detect curses headers correctly for cross-compiling

[xdegaye]

Removing from the dependencies some issues that have become out of scope.

[xdegaye]

issue bpo-28996: wcscoll is broken on Android and test_locale fails
issue bpo-28997: test_readline.test_nonascii fails on Android

[xdegaye]

issue bpo-29040: building Android with android-ndk-r14

[xdegaye]

issue bpo-28833: cross compilation of third-party extension modules

[xdegaye]

Removing issues from the dependencies list:
issue bpo-22747: Interpreter fails in initialize on systems where HAVE_LANGINFO_H is undefined
This issue is fixed for Android.
issue bpo-26859: unittest fails with "Start directory is not importable" when trying to run sourceless tests
This issue is out of scope.

[xdegaye]

issue bpo-29176: /tmp does not exist on Android and is used by curses.window.putwin()
issue bpo-29177: skip tests of test_logging when bind() raises PermissionError (non-root user on Android)
issue bpo-29180: skip tests that raise PermissionError in test_os (non-root user on Android)
issue bpo-29181: skip tests that raise PermissionError in test_tarfile (non-root user on Android)
issue bpo-29184: skip tests of test_socketserver when bind() raises PermissionError (non-root user on Android)
issue bpo-29185: test_distutils fails on Android API level 24

[xdegaye]

Current status:
---------------

Available using the pyona build system [1]:

• Cross compilation of Python for a given Android API level and architecture with android-ndk-r13b.
• Cross compilation of third party extension modules (currently as a patch).
• Interactive interpreter with curses and readline support, the Android adb (remote) shell is used to start the interpreter on the qemu emulator or a device connected with USB.
• Remote debugging with gdb and support of the python-gdb module.
• The adb shell is used to start a run of the test suite on the emulator or a device.

To be defined for the support of the Android platform:

• Support starting with which Python release ?
  sys.getandroidapilevel() has been defined in Python 3.7 by bpo-28740, so unless this enhancement is backported to 3.6 the first release to support Android could be 3.7.
• What is the supported Android API level(s) [2] ?
  Level 21 is the first to provide a reliable wide character support.
  Level 24 is the most recent api level currently supported by the NDK and is the first where the adb shell is run as the 'shell' user instead of as 'root' and as a consequence, where the test suite must now cope with Android SELinux non permitted operations (hard link, mkfifo, mknod, bind on a unix socket).
• On which architecture(s) ?
  The x86 platform is useful for testing and debugging as it runs fast on the Android qemu emulator.
  AFAIK the armv7 platform is still one of the most widespread Android platforms [3] [4].
• Building with which NDK version ?
  The next android-ndk-r14 release is the first to provide "Unified headers", see issue bpo-29040.
• The buildbots run the test suite on the Android qemu emulator or on a device or both ?

Test suite results on the Android qemu emulator:
test_builtin is excluded in all the tests - test_asyncio is excluded on armv7-android-api-21.
x86 platform (duration: about 27 minutes, to be compared with 26 minutes when the test suite is run natively on the same laptop).
api 21: success
api 24: success
armv7 platform (duration: about 410 minutes. Without the latest released libffi-3.2.1 that does not build on armv7).
api 21: success
api 24: success

Remaining issues:

1. Blocker issues:
   test_builtin:
   issue bpo-13886: failure when test_readline is run before test_builtin.
   This problem is not specific to Android.
   test_asyncio:
   issue bpo-26858: android: setting SO_REUSEPORT fails
   Failure only on armv7 at api 21.

2. Issues with a patch tested successfully on x86 and armv7 at Android api 21 and 24:
   Build:
   issue bpo-28833: cross compilation of third-party extension modules.
   PermissionError raised at api 24:
   test_eintr test_genericpath test_pathlib test_posix test_shutil test_stat
   issue bpo-28759: access to mkfifo, mknod and hard links is controled by SELinux MAC on Android
   test_os:
   issue bpo-29180: skip tests that raise PermissionError in test_os (non-root user on Android)
   test_tarfile:
   issue bpo-29181: skip tests that raise PermissionError in test_tarfile (non-root user on Android)
   test_socketserver:
   issue bpo-29184: skip tests of test_socketserver when bind() raises PermissionError (non-root user on Android)
   test_distutils:
   issue bpo-29185: test_distutils fails on Android api 24
   test_asyncio:
   issue bpo-28684: [asyncio] bind() on a unix socket raises PermissionError on Android for a non-root user.
   Miscellaneous:
   test_asyncio:
   issue bpo-28562: test_asyncio fails on Android upon calling getaddrinfo().
   test_readline:
   issue bpo-28997: test_readline.test_nonascii fails on Android.
   test_curses:
   issue bpo-29176: /tmp does not exist on Android and is used by curses.window.putwin()

3. Enhancement issues:
   issue bpo-26855: android: add platform.android_ver().
   issue bpo-27659: Prohibit implicit C function declarations.
   issue bpo-29040: building Android with android-ndk-r14.

4. Languishing issues:
   issue bpo-22724: byte-compile fails for cross-builds.
   issue bpo-27606: Android cross-built for armv5te with clang and '-mthumb' crashes with SIGSEGV or SIGILL.
   issue bpo-26852: add the '--enable-sourceless-distribution' option to configure.
   issue bpo-27640: add the '--disable-test-suite' option to configure.

[1] https://bitbucket.org/xdegaye/pyona
[2] Andoid versions:
Android Version Released API Level Name
Android 7.1 October 2016 25 Nougat
Android 7.0 August 2016 24 Nougat
Android 6.0 August 2015 23 Marshmallow
Android 5.1 March 2015 22 Lollipop
Android 5.0 November 2014 21 Lollipop
Android 4.4W June 2014 20 Kitkat Watch
Android 4.4 October 2013 19 Kitkat
Android 4.3 July 2013 18 Jelly Bean
Android 4.2-4.2.2 November 2012 17 Jelly Bean
Android 4.1-4.1.1 June 2012 16 Jelly Bean
Android 4.0.3-4.0.4 December 2011 15 Ice Cream Sandwich
Android 4.0-4.0.2 October 2011 14 Ice Cream Sandwich
Android 3.2 June 2011 13 Honeycomb
Android 3.1.x May 2011 12 Honeycomb
Android 3.0.x February 2011 11 Honeycomb
Android 2.3.3-2.3.4 February 2011 10 Gingerbread
Android 2.3-2.3.2 November 2010 9 Gingerbread
Android 2.2.x June 2010 8 Froyo
Android 2.1.x January 2010 7 Eclair
Android 2.0.1 December 2009 6 Eclair
Android 2.0 November 2009 5 Eclair
Android 1.6 September 2009 4 Donut
Android 1.5 May 2009 3 Cupcake
Android 1.1 February 2009 2 Base
Android 1.0 October 2008 1 Base
[3] https://en.wikipedia.org/wiki/Comparison_of_smartphones
[4] https://en.wikipedia.org/wiki/ARM_Cortex-A

[xdegaye]

issue bpo-28180: sys.getfilesystemencoding() should default to utf-8

[xdegaye]

issue bpo-32031: Do not use the canonical path in pydoc test_mixed_case_module_names_are_lower_cased
issue bpo-32059: detect_modules() in setup.py must also search the sysroot paths
issue bpo-32126: [asyncio] test failure when the platform lacks a functional sem_open()
issue bpo-32138: android: test_faulthandler fails also on API 24
issue bpo-32139: android: locale is modified by test_strftime
issue bpo-32205: test.pythoninfo prints the native sysconfig data when cross-compiling
issue bpo-32199: uuid.getnode() now returns the MAC address on Android
issue bpo-32246: test_regrtest alters the execution environment on Android
issue bpo-32210: Add platform.android_ver() to test.pythoninfo for Android platforms
issue bpo-32202: [ctypes] all long double tests fail on android-24-x86_64
issue bpo-32203: [ctypes] test_struct_by_value fails on android-24-arm64

[xdegaye]

bpo-22724 is fixed for Android.

[xdegaye]

Remove depencies that are not specific to Android.