test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()

[xdegaye]

BPO	36342
Nosy	@brettcannon, @vstinner, @xdegaye, @miss-islington
PRs	bpo-36342: Fix test_multiprocessing in test_venv #12513

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-24.12:06:09.204>
created_at = <Date 2019-03-18.10:02:39.890>
labels = ['3.8', 'type-bug', 'tests']
title = 'test_venv failure when executed by test_multiprocessing and the platform lacks a functional sem_open()'
updated_at = <Date 2019-09-24.12:06:09.203>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2019-09-24.12:06:09.203>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-24.12:06:09.204>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2019-03-18.10:02:39.890>
creator = 'xdegaye'
dependencies = []
files = []
hgrepos = []
issue_num = 36342
keywords = ['patch']
message_count = 10.0
messages = ['338188', '338189', '338192', '338240', '338398', '338464', '338495', '338671', '344004', '353078']
nosy_count = 4.0
nosy_names = ['brett.cannon', 'vstinner', 'xdegaye', 'miss-islington']
pr_nums = ['12513']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36342'
versions = ['Python 3.8']

[xdegaye]

test_multiprocessing fails on platforms lacking a functioning sem_open (Android for example) with:

======================================================================
ERROR: test_multiprocessing (test.test_venv.BasicTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 317, in test_multiprocessing
    out, err = check_output([envpy, '-c',
  File "/data/local/tmp/python/lib/python3.8/test/test_venv.py", line 37, in check_output
    raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command '['/data/local/tmp/python/tmp/tmpli0_hkdl/bin/python', '-c', 'from multiprocessing import Pool; print(Pool(1).apply_async("Python".lower).get(3))']' returned non-zero exit status 1.

[vstinner]

Are you sure that multiprocessing is the issue? On bpo-35978, the test fails because of venv, not because of multiprocessing. Can it be the same issue?

[xdegaye]

You are right, this is a duplicate of bpo-35978.
Closing as duplicate.

[xdegaye]

Too hasty again, remembering now why I linked this failure with sem_open :-(

Re-opening this issue as the test does fail because Android lacks a functioning sem_open implementation. And tagging this issue as a dependency of bpo-35978 as we should wait for the solution on bpo-35978 before fixing this one, if it is still needed.

The link between the failure and sem_open shows up when running the python statements of test_multiprocessing() in the interpreter on Android:

generic_x86_64:/data/local/tmp/python $ python
Python 3.8.0a2+ (heads/bpo-36341-dirty:41f0b78cbf, Mar 18 2019, 10:43:05)
[Clang 3.8.275480 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> print(Pool(1).apply_async("Python".lower).get(3))
Traceback (most recent call last):
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
    from _multiprocessing import SemLock, sem_unlink
ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/local/tmp/python/lib/python3.8/lib-dynload/_multiprocessing.cpython-38dm.so)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 118, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 166, in __init__
    self._setup_queues()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/pool.py", line 300, in _setup_queues
    self._inqueue = self._ctx.SimpleQueue()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 112, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/context.py", line 66, in Lock
    from .synchronize import Lock
  File "/data/local/tmp/python/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module>
    raise ImportError("This platform lacks a functioning sem_open" +
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
>>>

[brettcannon]

So this is a test_multiprocessing issue? If it is then the title is misleading.

[xdegaye]

It is a test_venv issue related to bpo-32126 and also related to the other issues listed in bpo-32126.

Changing the title as it is ambiguous indeed.

check_output() in test_venv.py does not print the full stack trace as it is done by test_asyncio in bpo-32126, is this another issue ?

[brettcannon]

I guess my confusion comes from the fact that test_venv in isolation is a totally fine test suite, it just fails when test_multiprocessing runs it due to something multiprocessing-related.

I've tried tweaking the title to reflect the fact that it's a test_venv failure triggered by test_multiprocessing.

[xdegaye]

Removing the bpo-35978 dependency as changeset 8bba81f that fixes bpo-35978 only skips test_multiprocessing when test_venv is run from a venv.

[miss-islington]

New changeset 5437ccc by Miss Islington (bot) (xdegaye) in branch 'master':
bpo-36342: Fix test_multiprocessing in test_venv (GH-12513)
5437ccc

[vstinner]

It seems like this issue has been fixed in the master branch. Reopen the issue if you consider that the fix should be backported.