instanciation of multiprocessing.Queue raises ImportError in test_logging

[xdegaye]

BPO	28668
Nosy	@vsajip, @xdegaye, @serhiy-storchaka, @applio
PRs	bpo-28668: test.support.requires_multiprocessing_queue is removed #4560
Files	test__multiprocessing_queue.patch test_multiprocessing_queue_2.patch: using a plain decorator with no argument

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-01.07:09:37.893>
created_at = <Date 2016-11-11.15:48:00.795>
labels = ['3.7', 'type-bug', 'tests']
title = 'instanciation of multiprocessing.Queue raises ImportError in test_logging'
updated_at = <Date 2017-12-01.07:09:37.893>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2017-12-01.07:09:37.893>
actor = 'xdegaye'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-01.07:09:37.893>
closer = 'xdegaye'
components = ['Tests']
creation = <Date 2016-11-11.15:48:00.795>
creator = 'xdegaye'
dependencies = []
files = ['45448', '45458']
hgrepos = []
issue_num = 28668
keywords = ['patch']
message_count = 9.0
messages = ['280589', '280593', '280649', '280651', '280654', '280660', '280862', '306958', '307366']
nosy_count = 5.0
nosy_names = ['vinay.sajip', 'xdegaye', 'python-dev', 'serhiy.storchaka', 'davin']
pr_nums = ['4560']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28668'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[xdegaye]

Occurs on Android that has a broken shared semaphore implementation (bpo-26924).
Patch attached.

One of the backtraces:

======================================================================
ERROR: test_handle_called_with_mp_queue (test.test_logging.QueueListenerTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/synchronize.py", line 29, in <modu
le>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'sem_unlink'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/unittest/mock.py", line 1179, in patched
    return func(*args, **keywargs)
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/test/test_logging.py", line 3130, in test_handle_c
alled_with_mp_queue
    log_queue = multiprocessing.Queue()
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/context.py", line 102, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/queues.py", line 39, in __init__
    from .synchronize import SEM_VALUE_MAX as maxsize
  File "/sdcard/org.bitbucket.pyona/lib/python3.7/multiprocessing/synchronize.py", line 34, in <modu
le>
    " function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required sync
hronization primitives needed will not function, see issue 3770.

[serhiy-storchaka]

The decorator doesn't need arguments. I would make it similar to skip_unless_symlink.

[xdegaye]

When the test is to be skipped:

• The decorator in the previous patch (a decorator with arguments that takes no argument) returns a decorator which is unittest.skip (to be applied to the test method).
• On the other side, the skip_unless_symlink decorator pattern (no argument) applies the unittest.skip decorator to the test method and returns the decorated test method.
  This new patch uses the second method suggested by Serhiy.

[serhiy-storchaka]

requires_multiprocessing_queue is used only in test_logging. Is it worth to add it in test.support?

[xdegaye]

IMHO other tests in the future may instantiate multiprocessing.Queue() and come up with another new decorator when the test fails on the (yet to come) Android buildbot, ignoring that one already exists if we move the decorator to the test_logging module.

[serhiy-storchaka]

Okay. test_multiprocessing_queue_2.patch LGTM.

[python-dev]

New changeset a377e6987821 by Xavier de Gaye in branch '3.5':
bpo-28668: Skip tests where instanciation of multiprocessing.Queue
https://hg.python.org/cpython/rev/a377e6987821

New changeset e5404ba1b19e by Xavier de Gaye in branch '3.6':
bpo-28668: Merge 3.5
https://hg.python.org/cpython/rev/e5404ba1b19e

New changeset 1f0b0ecf7dc1 by Xavier de Gaye in branch 'default':
bpo-28668: Merge 3.6
https://hg.python.org/cpython/rev/1f0b0ecf7dc1

[xdegaye]

The multiprocessing module is not functional when the multiprocessing.synchronize module cannot be imported and all the multiprocessing tests are skipped in that case. Use the same idiom (i.e. test.support.import_module('multiprocessing.synchronize')) to skip the tests when the platform sem_open implementation is broken and remove test.support.requires_multiprocessing_queue.

[xdegaye]

New changeset bf2b65e by xdegaye in branch 'master':
bpo-28668: test.support.requires_multiprocessing_queue is removed (GH-4560)
bf2b65e