Fixes Python for Android API 19

[vstinner]

BPO	32654
Nosy	@vstinner, @ned-deily, @pmp-p, @yan12125
PRs	[WIP] bpo-32654: Fixes for Android API 19 #5305

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-29.21:46:33.867>
created_at = <Date 2018-01-24.17:31:20.819>
labels = ['3.8']
title = 'Fixes Python for Android API 19'
updated_at = <Date 2018-05-29.21:46:33.867>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-05-29.21:46:33.867>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.21:46:33.867>
closer = 'vstinner'
components = []
creation = <Date 2018-01-24.17:31:20.819>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 32654
keywords = ['patch']
message_count = 14.0
messages = ['310619', '310682', '310685', '310687', '310689', '310692', '310694', '310695', '310697', '310952', '310959', '310966', '311026', '318100']
nosy_count = 4.0
nosy_names = ['vstinner', 'ned.deily', 'pmpp', 'yan12125']
pr_nums = ['5305']
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue32654'
versions = ['Python 3.8']

[vstinner]

I'm working on a pull request to make https://github.com/pmp-p/droid-pydk/tree/master/sources.32/cpython-bpo-30386.patchset changes upstream.

[vstinner]

I wrote PR 5305 for Paul Peny who uses cheap devices (less than 50$, maybe less 20$? I'm not sure) for development, but these devices use old Android versions.

It seems like some people are exchanging patches, in private or in public, for Android API 19. My intent is to make patches upstream, especially small patches which fix compilation of Python on Android API 19.

I'm not sure that we should support API 19.

Xavier de Gaye wants to focus on the future, support API 24 and newer:
https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst

His document gives many good reasons to not support API < 24.

My intent is not to "fully support" Python on Android API 19. Just to make sure that we can compile Python and that python3 -c pass doesn't crash :-) So only merge the most critical fixes for API 19.

IMHO merging further changes to fix other Python functions should be discussed on a case by base basis.

[yan12125]

Here's another argument for supporting android < 21: according to Google [1], 12.8% devices on the world run android-19. Furthermore, upgrading Android OS for a device is almost impossible for non-hackers unless the manufacturer provides an image for newer Android, and it's common for manufacturers not upgrading a phone soon after its debut.

[ned-deily]

I don't think we should be adding this platform-version-specific code unless we are going to fully support that particular platform and version. That means at a minimum adding a buildbot for it and then having a core developer signed up to maintain the platform version. It's not fair to other code developers and to downstream users to falsely imply that we support something that we are not set up to support, especially an already out-of-date version when we don't yet support any version of Android. We will end up with another support headache as has been the case with other non-mainstream (with regard to Python) platforms.

[pmp-p]

Thanks ! I tested your PR it just built out of the box with the help of a one line change to Xavier de Gaye's android build support tool ( configure-android script + Android folder from bpo-30386 ).

I'll forward to you the result of testsuite on a $10 Arm cortex-a7 board running Android kitkat 4.4.2 ( H3droid ) as soon as it finishes as i run a testbot on it.

My point of view is that "out of date" doesn't mean people don't actually *use* it : vstinner is indeed right , vendors provides almost no updates for hardware which actually can live long and have the power of some low power i*&a* cpu but without the actual drawbacks we heard about recently.

[yan12125]

Agree with Ned Deily here. If CPython is going to be support android-19, it's better to fully support it. I'm optimistic here - I believe there won't be many android-19-specific patches beyond locale-related ones. Once android-24 is done, we can revisit this part and see how many patches are needed.

[vstinner]

Ned Deily: "I don't think we should be adding this platform-version-specific code unless we are going to fully support that particular platform and version."

I know the theory and I know the practice :-)

I started to list platforms that are "supported" by Python, with various support levels:
http://vstinner.readthedocs.io/cpython.html#supported-platforms

For example, Linux has an excellent support, whereas OpenBSD is more on the "best effort" side. "grep OpenBSD" in CPython shows me at least 51 lines of code, whereas OpenBSD has no buildbot and no dedicated developer to handle OpenBSD specific issues. For example, test_crypt is broken since at least 2 years on OpenBSD, as many test_socket tests.

For OpenBSD, IMHO it's ok to have a best effort support, basically apply proposed patches.

For Android, the situation is somehow different. There are more and more people working on supporting officially Python on Android, like Xavier de Gaye and me who are core developers. Xavier is working on that for 2 years, and IMHO the port is done at 95%. The last part is just to setup a buildbot for Android API 24.

The question is the added maintenance cost of Android API 19 #ifdef code, compared to pain of building Python on Android.

Technically, it should be possible to setup a buildbot for Android API 19, but only with a subset of tests. Either explicitly only run a subset of tests on this hypothetic buildbot, or even modify the Python test suite to skip some tests on Android API 19. While I expect a few patches for fix Python compilation on Android API 19, I do expect a lot of @skipIf() in tests, since API 19 has many broken or missing features. The thing is that most people don't care of these missing or broken features, it's not an issue for most applications on Android.

Note: Kivy is using Python on Android for years, but they use CrystaX NDK and not Google NDK. This issue is about supporting Google NDK: have the maximum compatibility.

[vstinner]

Another example of platform with best effort support is AIX. Until very recently, ctypes.util.find_library() didn't work on AIX which causes many tests failure. The AIX buildbot is red as far as I can remember. *Many* tests are failing on AIX for various reasons. While a few people sometimes propose fixes, basically the AIX support didn't evolved much last years in AIX...

AIX is a platform with a buildbot, but no real dedicated developer to fix all AIX specific issues.

[vstinner]

By the way, I started to take notes on Python on Android, since the topic is wide and complex:
http://vstinner.readthedocs.io/python_android.html

[yan12125]

Here are some ideas after testing:

• Testing environment:

• My building scripts at https://github.com/yan12125/python3-android/
• Android NDK r16b
• Google's emulator image 'system-images;android-19;default;x86'

• Ideas/findings:

• Locale emulation for setlocale() no longer works as _locale builds fine.

>>> locale.setlocale(locale.LC_ALL, None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python3/usr/lib/python3.7/locale.py", line 608, in setlocale
    return _setlocale(category, locale)
locale.Error: locale query failed

• Declarations for mmap/sendfile shouldn't be necessary. In Android NDK r16b, sendfile() is defined if __USE_FILE_OFFSET64 is not defined: (from $ANDROID_NDK/sysroot/usr/include/sys/sendfile.h)

#if defined(__USE_FILE_OFFSET64)

#if __ANDROID_API__ >= 21
ssize_t sendfile(int __out_fd, int __in_fd, off_t* __offset, size_t __count) __RENAME(sendfile64) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */

#else
ssize_t sendfile(int __out_fd, int __in_fd, off_t* __offset, size_t __count);
#endif

#if __ANDROID_API__ >= 21
ssize_t sendfile64(int __out_fd, int __in_fd, off64_t* __offset, size_t __count) __INTRODUCED_IN(21);
#endif /* __ANDROID_API__ >= 21 */

__USE_FILE_OFFSET64 is defined as _FILE_OFFSET_BITS is defined to 64 in pyconfig.h. An NDK developer suggest "stop defining _FILE_OFFSET_BITS=64 on 32-bit Android." (android/ndk#536 (comment)) Either disabling large file support on Android or simply don't define _FILE_OFFSET_BITS on Android should work. (Android does not use _LARGEFILE_SOURCE)

• SIGRTMIN/SIGRTMAX: This is a bug on older NDK. r15 has fixed it (Unified Headers: LONG_BIT Undeclared in legacy_signal_inlines.h When APP_API < 21 android/ndk#352). I propose to wait until CPython builds with NDK r15+. I don't like adding patches for a known bug in older toolchains and revert them later.

[pmp-p]

@yan12125
Hi, I ran some tests too and here are my findings:
https://github.com/pmp-p/droid-pydk/tree/master/sources.32/build_logs

• I used some dirty patchset to cover more tests https://github.com/pmp-p/droid-pydk/tree/master/sources.32/cpython-bpo-30386.patchset i would be glad if you could have a look at them, as i am not a C/C++ programmer (i'm "pmpp" on IRC where we met last time).

• Maybe SIG* stuff is fixed only for unified headers API>=21 regardless of NDK that would need checking.

• python build quite fine will all NDK ( sorry to say but especially the non vendor one can support cpython for prehistoric api and version 3.5/3.6 *without patches* under certain condition of use )

my thoughts :

• Only third parties libs that have trouble with NDK versions, like openssl but libssl and libcrypto are provided by OS.

• Third parties are not required to embed libpython. Xavier de Gaye's script use shared linking. you should be able to provide whatever version of those you need that's not a cpython's build script job to deal with that, imho it was just for demontrasting that android build can be easy. I personnally would not build my third parties that way.

• non official cpython build runs very well after android 5.0 there's almost nothing to do left, and micropython is able to run everywhere cpython can't (yet) so nothing is impossible.

• old api are frozen, there won't be suddenly a lot to support, when devices using them will vanish from asian stores it will mark the end of them and their need of support : imho there's no "end of life/support" concept for android os.

• I'd like a possible api19 "best effort", because android kitkat vendor statistics are wrong they don't take into account the *huge* amount of gservice-less devices.

unclear if unrelated to this:

• platform=='android' shoud be set, regardless of any API version : Android platform exists and is already running some pythons, it is a fact.

Thanks Victor for trying to clean my patchset : i did not realize my embed use of libpython could help official cpython too. Anyway I would be glad to help this way or another.

[ned-deily]

Since we won't officially support Android in 3.7, these changes should wait for 3.8 and until there is a PEP for Android support.

[vstinner]

"""
>>> locale.setlocale(locale.LC_ALL, None)
...
locale.Error: locale query failed
"""

IMHO the correct fix here is to not define _locale.setlocale() to get the pure Python locale.setlocale() emulation.

[vstinner]

I'm sorry, but I'm not longer interested to work on supporting Android API 19. I close the bug. If someone wants to continue to work on the Android support, it's probably better to first focus on most recent Android versions.