Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

instanciation of multiprocessing.Queue raises ImportError in test_logging #72854

Closed
xdegaye mannequin opened this issue Nov 11, 2016 · 9 comments
Closed

instanciation of multiprocessing.Queue raises ImportError in test_logging #72854

xdegaye mannequin opened this issue Nov 11, 2016 · 9 comments
Labels
3.7 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@490c593f-f636-409f-bb35-6abeb38a4595
Copy link
Mannequin

xdegaye mannequin commented Nov 11, 2016

BPO 28668
Nosy @vsajip, @xdegaye, @serhiy-storchaka, @applio
PRs
  • bpo-28668: test.support.requires_multiprocessing_queue is removed #4560
  • Files
  • test__multiprocessing_queue.patch
  • test_multiprocessing_queue_2.patch: using a plain decorator with no argument
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2017-12-01.07:09:37.893>
    created_at = <Date 2016-11-11.15:48:00.795>
    labels = ['3.7', 'type-bug', 'tests']
    title = 'instanciation of multiprocessing.Queue raises ImportError in test_logging'
    updated_at = <Date 2017-12-01.07:09:37.893>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2017-12-01.07:09:37.893>
    actor = 'xdegaye'
    assignee = 'none'
    closed = True
    closed_date = <Date 2017-12-01.07:09:37.893>
    closer = 'xdegaye'
    components = ['Tests']
    creation = <Date 2016-11-11.15:48:00.795>
    creator = 'xdegaye'
    dependencies = []
    files = ['45448', '45458']
    hgrepos = []
    issue_num = 28668
    keywords = ['patch']
    message_count = 9.0
    messages = ['280589', '280593', '280649', '280651', '280654', '280660', '280862', '306958', '307366']
    nosy_count = 5.0
    nosy_names = ['vinay.sajip', 'xdegaye', 'python-dev', 'serhiy.storchaka', 'davin']
    pr_nums = ['4560']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue28668'
    versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

    @490c593f-f636-409f-bb35-6abeb38a4595
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 11, 2016

    Occurs on Android that has a broken shared semaphore implementation (bpo-26924).
    Patch attached.

    One of the backtraces:

    ======================================================================
    ERROR: test_handle_called_with_mp_queue (test.test_logging.QueueListenerTest)
    ----------------------------------------------------------------------

    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/synchronize.py", line 29, in <modu
    le>
        from _multiprocessing import SemLock, sem_unlink
    ImportError: cannot import name 'sem_unlink'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/unittest/mock.py", line 1179, in patched
        return func(*args, **keywargs)
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_logging.py", line 3130, in test_handle_c
    alled_with_mp_queue
        log_queue = multiprocessing.Queue()
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/context.py", line 102, in Queue
        return Queue(maxsize, ctx=self.get_context())
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/queues.py", line 39, in __init__
        from .synchronize import SEM_VALUE_MAX as maxsize
      File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/synchronize.py", line 34, in <modu
    le>
        " function, see issue 3770.")
    ImportError: This platform lacks a functioning sem_open implementation, therefore, the required sync
    hronization primitives needed will not function, see issue 3770.

    @490c593f-f636-409f-bb35-6abeb38a4595 xdegaye mannequin added 3.7 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Nov 11, 2016
    @serhiy-storchaka
    Copy link
    Member

    The decorator doesn't need arguments. I would make it similar to skip_unless_symlink.

    @490c593f-f636-409f-bb35-6abeb38a4595
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 12, 2016

    When the test is to be skipped:

    • The decorator in the previous patch (a decorator with arguments that takes no argument) returns a decorator which is unittest.skip (to be applied to the test method).
    • On the other side, the skip_unless_symlink decorator pattern (no argument) applies the unittest.skip decorator to the test method and returns the decorated test method.
      This new patch uses the second method suggested by Serhiy.

    @serhiy-storchaka
    Copy link
    Member

    requires_multiprocessing_queue is used only in test_logging. Is it worth to add it in test.support?

    @490c593f-f636-409f-bb35-6abeb38a4595
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 12, 2016

    IMHO other tests in the future may instantiate multiprocessing.Queue() and come up with another new decorator when the test fails on the (yet to come) Android buildbot, ignoring that one already exists if we move the decorator to the test_logging module.

    @serhiy-storchaka
    Copy link
    Member

    Okay. test_multiprocessing_queue_2.patch LGTM.

    @1762cc99-3127-4a62-9baf-30c3d0f51ef7
    Copy link
    Mannequin

    python-dev mannequin commented Nov 15, 2016

    New changeset a377e6987821 by Xavier de Gaye in branch '3.5':
    bpo-28668: Skip tests where instanciation of multiprocessing.Queue
    https://hg.python.org/cpython/rev/a377e6987821

    New changeset e5404ba1b19e by Xavier de Gaye in branch '3.6':
    bpo-28668: Merge 3.5
    https://hg.python.org/cpython/rev/e5404ba1b19e

    New changeset 1f0b0ecf7dc1 by Xavier de Gaye in branch 'default':
    bpo-28668: Merge 3.6
    https://hg.python.org/cpython/rev/1f0b0ecf7dc1

    @490c593f-f636-409f-bb35-6abeb38a4595 xdegaye mannequin closed this as completed Nov 15, 2016
    @490c593f-f636-409f-bb35-6abeb38a4595
    Copy link
    Mannequin Author

    xdegaye mannequin commented Nov 25, 2017

    The multiprocessing module is not functional when the multiprocessing.synchronize module cannot be imported and all the multiprocessing tests are skipped in that case. Use the same idiom (i.e. test.support.import_module('multiprocessing.synchronize')) to skip the tests when the platform sem_open implementation is broken and remove test.support.requires_multiprocessing_queue.

    @490c593f-f636-409f-bb35-6abeb38a4595 xdegaye mannequin reopened this Nov 25, 2017
    @490c593f-f636-409f-bb35-6abeb38a4595
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 1, 2017

    New changeset bf2b65e by xdegaye in branch 'master':
    bpo-28668: test.support.requires_multiprocessing_queue is removed (GH-4560)
    bf2b65e

    @490c593f-f636-409f-bb35-6abeb38a4595 xdegaye mannequin closed this as completed Dec 1, 2017
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 only security fixes tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant