android: test_concurrent_futures fails

[xdegaye]

BPO	26924
Nosy	@brianquinlan, @xdegaye, @berkerpeksag, @moreati, @applio
Files	sem_unlink.diff sem_unlink_v2.diff sem_unlink_alt.diff sem_unlink_alt_output.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-05-07.17:39:40.643>
created_at = <Date 2016-05-03.13:37:56.125>
labels = ['extension-modules', 'type-bug']
title = 'android: test_concurrent_futures fails'
updated_at = <Date 2016-11-11.20:01:47.839>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2016-11-11.20:01:47.839>
actor = 'davin'
assignee = 'none'
closed = True
closed_date = <Date 2016-05-07.17:39:40.643>
closer = 'berker.peksag'
components = ['Extension Modules']
creation = <Date 2016-05-03.13:37:56.125>
creator = 'xdegaye'
dependencies = []
files = ['42710', '42769', '42773', '42774']
hgrepos = []
issue_num = 26924
keywords = ['patch']
message_count = 17.0
messages = ['264721', '264796', '264802', '264824', '265075', '265079', '265080', '265082', '265083', '265084', '265085', '265090', '265091', '265092', '265093', '265129', '265130']
nosy_count = 6.0
nosy_names = ['bquinlan', 'xdegaye', 'python-dev', 'berker.peksag', 'Alex.Willmer', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26924'
versions = ['Python 3.6']

[xdegaye]

test_concurrent_futures fails on an android emulator running an x86 system image at API level 21.

There are multiple ERRORs, all resulting in the same traceback:

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_concurrent_futures.py", line 68, in setU
p
    self.executor = self.executor_type(max_workers=self.worker_count)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/concurrent/futures/process.py", line 390, in __init__
    EXTRA_QUEUED_CALLS)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 101, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 66, in Lock
    return Lock(ctx=self.get_context())
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 163, in __in
it__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 60, in __ini
t__
    unlink_now)
OSError: [Errno 38] Function not implemented

The ERRORs:
===========

test_duplicate_futures (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_map_chunksize (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_max_workers_negative (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_no_stale_references (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_traceback (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR

[xdegaye]

On android:

root@generic_x86:/data/local/tmp # python
Python 3.6.0a0 (default:811ccdee6f87+, May  4 2016, 10:31:14) 
[GCC 4.9 20140827 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from _multiprocessing import sem_unlink
>>> sem_unlink('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 38] Function not implemented
>>>

[berkerpeksag]

It normally should raise an ImportError if sem_unlink is not available. Could you try the attached patch please?

[xdegaye]

Patch tested, it does not fix the problem.
This is because in pyconfig.h:
/* Define to 1 if you have the `sem_unlink' function. */
#define HAVE_SEM_UNLINK 1

Android declares the function but does not implement it. This is not the only case, for example issue bpo-26857.

[berkerpeksag]

Thanks for testing. Here is a new patch.

[xdegaye]

Here is the result of the test:

root@generic_x86:/data/local/tmp # python -m test -v test_concurrent_futures
== CPython 3.6.0a0 (default:811ccdee6f87+, May 7 2016, 18:51:28) [GCC 4.9 20140827 (prerelease)]
== Linux-3.4.67+-i686-with-libc little-endian
== hash algorithm: fnv 32bit
== /data/local/tmp/test_python_2024
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
Run tests sequentially
0:00:00 [1/1] test_concurrent_futures
test_concurrent_futures skipped -- This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see bpo-3770.
1 test skipped:
test_concurrent_futures
Total duration: 0:00:01

[xdegaye]

Here is the result of the test:

I meant, results with the new patch.

[python-dev]

New changeset 1030aa8357a7 by Berker Peksag in branch 'default':
Issue bpo-26924: Do not define _multiprocessing.sem_unlink under Android
https://hg.python.org/cpython/rev/1030aa8357a7

[berkerpeksag]

Thanks, Xavier.