Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

android: test_concurrent_futures fails #71111

Closed
xdegaye mannequin opened this issue May 3, 2016 · 17 comments
Closed

android: test_concurrent_futures fails #71111

xdegaye mannequin opened this issue May 3, 2016 · 17 comments
Labels
extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented May 3, 2016

BPO 26924
Nosy @brianquinlan, @xdegaye, @berkerpeksag, @moreati, @applio
Files
  • sem_unlink.diff
  • sem_unlink_v2.diff
  • sem_unlink_alt.diff
  • sem_unlink_alt_output.txt
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2016-05-07.17:39:40.643>
    created_at = <Date 2016-05-03.13:37:56.125>
    labels = ['extension-modules', 'type-bug']
    title = 'android: test_concurrent_futures fails'
    updated_at = <Date 2016-11-11.20:01:47.839>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2016-11-11.20:01:47.839>
    actor = 'davin'
    assignee = 'none'
    closed = True
    closed_date = <Date 2016-05-07.17:39:40.643>
    closer = 'berker.peksag'
    components = ['Extension Modules']
    creation = <Date 2016-05-03.13:37:56.125>
    creator = 'xdegaye'
    dependencies = []
    files = ['42710', '42769', '42773', '42774']
    hgrepos = []
    issue_num = 26924
    keywords = ['patch']
    message_count = 17.0
    messages = ['264721', '264796', '264802', '264824', '265075', '265079', '265080', '265082', '265083', '265084', '265085', '265090', '265091', '265092', '265093', '265129', '265130']
    nosy_count = 6.0
    nosy_names = ['bquinlan', 'xdegaye', 'python-dev', 'berker.peksag', 'Alex.Willmer', 'davin']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue26924'
    versions = ['Python 3.6']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 3, 2016

    test_concurrent_futures fails on an android emulator running an x86 system image at API level 21.

    There are multiple ERRORs, all resulting in the same traceback:

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_concurrent_futures.py", line 68, in setU
    p
        self.executor = self.executor_type(max_workers=self.worker_count)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/concurrent/futures/process.py", line 390, in __init__
        EXTRA_QUEUED_CALLS)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 101, in Queue
        return Queue(maxsize, ctx=self.get_context())
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/queues.py", line 42, in __init__
        self._rlock = ctx.Lock()
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/context.py", line 66, in Lock
        return Lock(ctx=self.get_context())
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 163, in __in
    it__
        SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
      File "/sdcard/org.bitbucket.pyona/lib/python3.6/multiprocessing/synchronize.py", line 60, in __ini
    t__
        unlink_now)
    OSError: [Errno 38] Function not implemented

    The ERRORs:
    ===========

    test_duplicate_futures (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
    test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
    test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... ERROR
    test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_map_chunksize (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_max_workers_negative (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_no_stale_references (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_traceback (test.test_concurrent_futures.ProcessPoolExecutorTest) ... ERROR
    test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... ERROR
    test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR
    test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... ERROR

    @xdegaye xdegaye mannequin added stdlib Python modules in the Lib dir build The build process and cross-build type-bug An unexpected behavior, bug, or error labels May 3, 2016
    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    On android:

    root@generic_x86:/data/local/tmp # python
    Python 3.6.0a0 (default:811ccdee6f87+, May  4 2016, 10:31:14) 
    [GCC 4.9 20140827 (prerelease)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from _multiprocessing import sem_unlink
    >>> sem_unlink('foo')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [Errno 38] Function not implemented
    >>>

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented May 4, 2016

    It normally should raise an ImportError if sem_unlink is not available. Could you try the attached patch please?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 4, 2016

    Patch tested, it does not fix the problem.
    This is because in pyconfig.h:
    /* Define to 1 if you have the `sem_unlink' function. */
    #define HAVE_SEM_UNLINK 1

    Android declares the function but does not implement it. This is not the only case, for example issue bpo-26857.

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented May 7, 2016

    Thanks for testing. Here is a new patch.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 7, 2016

    Here is the result of the test:

    root@generic_x86:/data/local/tmp # python -m test -v test_concurrent_futures
    == CPython 3.6.0a0 (default:811ccdee6f87+, May 7 2016, 18:51:28) [GCC 4.9 20140827 (prerelease)]
    == Linux-3.4.67+-i686-with-libc little-endian
    == hash algorithm: fnv 32bit
    == /data/local/tmp/test_python_2024
    Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
    Run tests sequentially
    0:00:00 [1/1] test_concurrent_futures
    test_concurrent_futures skipped -- This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see bpo-3770.
    1 test skipped:
    test_concurrent_futures
    Total duration: 0:00:01

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 7, 2016

    Here is the result of the test:

    I meant, results with the new patch.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 7, 2016

    New changeset 1030aa8357a7 by Berker Peksag in branch 'default':
    Issue bpo-26924: Do not define _multiprocessing.sem_unlink under Android
    https://hg.python.org/cpython/rev/1030aa8357a7

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented May 7, 2016

    Thanks, Xavier.

    @berkerpeksag berkerpeksag added extension-modules C modules in the Modules dir and removed stdlib Python modules in the Lib dir build The build process and cross-build labels May 7, 2016
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 7, 2016

    New changeset eee959fee5f5 by Berker Peksag in branch 'default':
    Issue bpo-26924: Fix Windows buildbots
    https://hg.python.org/cpython/rev/eee959fee5f5

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented May 7, 2016

    Xavier, could you try this alternative patch?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 7, 2016

    Attached the results with sem_unlink_alt.diff.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 7, 2016

    There is a typo in sem_unlink_alt.diff, not in the patch itself but in the neighbouring lines. In Modules/_multiprocessing/semaphore.c, it should be, I think, after one does 's/define sem_unlink(name)/define SEM_UNLINK(name)/':

    @@ -194,8 +194,8 @@
     #  define SEM_FAILED ((sem_t *)-1)
     #endif
     
    -#ifndef HAVE_SEM_UNLINK
    -#  define sem_unlink(name) 0
    +#if !defined(HAVE_SEM_UNLINK) || defined(__ANDROID__)
    +#  define SEM_UNLINK(name) 0
     #endif
     
     #ifndef HAVE_SEM_TIMEDWAIT

    But applying this new patch, gives exactly the same results. Maybe it is not just sem_unlink() that is not implemented. I will run a gdb session to find out.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 7, 2016

    sem_open() is not implemented as well. Here is a gdb session with a breakpoint set at semlock_new() (SEM_FAILED is NULL on android):

    Breakpoint 1, semlock_new (type=0xb6d070c0 <_PyMp_SemLockType>,
    args=(1, 1, 1, '/mp-dkzrq4ed', True), kwds=0x0)
    at /home/xavier/src/packages/android/cpython/Modules/_multiprocessing/semaphore.c:420
    420 {
    (gdb) next
    428 if (!PyArg_ParseTupleAndKeywords(args, kwds, "iiisi", kwlist,
    (gdb)
    432 if (kind != RECURSIVE_MUTEX && kind != SEMAPHORE) {
    (gdb)
    437 if (!unlink) {
    (gdb)
    424 char *name, *name_copy = NULL;
    (gdb)
    445 handle = SEM_CREATE(name, value, maxvalue);
    (gdb)
    447 if (handle == SEM_FAILED || SEM_GET_LAST_ERROR() != 0)
    (gdb) print handle
    $5 = (SEM_HANDLE) 0x0
    (gdb) next
    460 if (handle != SEM_FAILED)
    (gdb)
    462 PyMem_Free(name_copy);
    (gdb)
    463 _PyMp_SetError(NULL, MP_STANDARD_ERROR);
    (gdb)
    464 return NULL;
    (gdb)

    This is run with sem_unlink_alt.diff modified to include the change to 'define SEM_UNLINK(name) 0' although I guess this does not change anything.

    Sorry for misleading you about sem_unlink().

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented May 7, 2016

    Thanks! Perhaps we should not set HAVE_SEM_* to 1 if we are on Android in the configure step.

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 8, 2016

    Undefining the SEM_* macros in multiprocessing.h when __ANDROID__ is set, would allow for an easier update when those functions become implemented at a future API level. The __ANDROID_API__ macro can be used for that purpose. For example, if this happens at API 24, this could be a changed to:

    #if defined(__ANDROID__)
    # include <android/api-level.h>
    # if __ANDROID_API__ < 24
    #  undef sem_open
    #  ...
    # endif
    #endif

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented May 8, 2016

    undef sem_open

    I meant of course:

    # undef HAVE_SEM_OPEN

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    extension-modules C modules in the Modules dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant