test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm

[mcepl]

BPO	38377
Nosy	@vstinner, @ned-deily, @mcepl, @asvetlov, @1st1, @miss-islington
PRs	bpo-38377 _multiprocessing.SemLock requires working /dev/shm on Linux. #16593 bpo-38377: multiprocessing.SemLock requires working /dev/shm on Linux. #19073 bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() #20944 [3.9] bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) #20962 [3.8] bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962) #20966 bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS #20984 [3.9] bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984) #20987 [3.8] bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984) #20988

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-21.21:17:29.495>
created_at = <Date 2019-10-05.13:47:18.114>
labels = ['3.8', '3.9', '3.10', 'expert-asyncio']
title = 'test_asyncio.test_events.GetEventLoopTestsMixin.test_get_event_loop_new_process mixin breaks in the Unix environment without working /dev/shm'
updated_at = <Date 2020-06-21.21:17:29.495>
user = 'https://github.com/mcepl'

bugs.python.org fields:

activity = <Date 2020-06-21.21:17:29.495>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-21.21:17:29.495>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2019-10-05.13:47:18.114>
creator = 'mcepl'
dependencies = []
files = []
hgrepos = []
issue_num = 38377
keywords = ['patch']
message_count = 12.0
messages = ['354010', '371809', '371819', '371826', '371828', '371833', '371836', '371886', '371891', '371892', '371893', '372021']
nosy_count = 6.0
nosy_names = ['vstinner', 'ned.deily', 'mcepl', 'asvetlov', 'yselivanov', 'miss-islington']
pr_nums = ['16593', '19073', '20944', '20962', '20966', '20984', '20987', '20988']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38377'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[mcepl]

======================================================================
ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestCGetEventLoop)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

======================================================================
ERROR: test_get_event_loop_new_process (test.test_asyncio.test_events.TestPyGetEventLoop)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2647, in test_get_event_loop_new_process
    self.loop.run_until_complete(main()),
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/asyncio/base_events.py", line 608, in run_until_complete
    return future.result()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/test/test_asyncio/test_events.py", line 2640, in main
    pool = concurrent.futures.ProcessPoolExecutor()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 555, in __init__
    self._call_queue = _SafeQueue(
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/concurrent/futures/process.py", line 165, in __init__
    super().__init__(max_size, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/queues.py", line 42, in __init__
    self._rlock = ctx.Lock()
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/home/abuild/rpmbuild/BUILD/Python-3.8.0rc1/Lib/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(
OSError: [Errno 38] Function not implemented

---

The problem is that in the OpenBuildService (the build system of SUSE distributions) /dev/shm is limited to:

abuild@milic:/rpmbuild/BUILD/Python-3.8.0rc1> ls -ld /dev/shm
drwxr-xr-x 1 root root 0 Oct 5 13:23 /dev/shm
abuild@milic:/rpmbuild/BUILD/Python-3.8.0rc1>

Provided PR will skip test if /dev/shm doesn’t exist or it doesn’t sufficient permissions.

[vstinner]

New changeset ddbeb2f by Victor Stinner in branch 'master':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944)
ddbeb2f

[vstinner]

New changeset b1e7361 by Victor Stinner in branch '3.9':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962)
b1e7361

[vstinner]

New changeset e805618 by Victor Stinner in branch '3.8':
bpo-38377: Add support.skip_if_broken_multiprocessing_synchronize() (GH-20944) (GH-20962) (GH-20966)
e805618

[vstinner]

Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

x86-64 macOS 3.9:
https://buildbot.python.org/all/#builders/696/builds/119

FAIL: test_resource_tracker_reused (test.test_multiprocessing_forkserver.TestResourceTracker)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.9.billenstein-macos/build/Lib/test/_test_multiprocessing.py", line 5219, in test_resource_tracker_reused
    self.assertTrue(is_resource_tracker_reused)
AssertionError: False is not true

[ned-deily]

Hum, test_resource_tracker_reused() failed twice on macOS, but it's unclear to me if it's a regression caused by my change or not.

It does appear to be caused by your change. I did a quick check with the 3.8 branch. With that PR, test_multiprocessing_forkserver fails solidly. Backing up to the previous commit, test_multiprocessing_forkserver passes.

[vstinner]

test_multiprocessing_forkserver.test_resource_tracker_reused() failed on 3.8, 3.9 and master buildbots. It looks like a real reproducible bug. I'm not sure how my change triggered this bug.

[vstinner]

From what I understood, the problem is that creating a SemLock has an impact on the resource tracker.

The failing unit tests checks something about the resource tracker.

The test runs a function in a child process. The function is defined in _test_multiprocessing.py. _test_multiprocessing.py calls skip_if_broken_multiprocessing_synchronize() which creates a SemLock(). At the end, the child process modifies the resource tracker because it creates a SemLock().

There are different options to fix the issue:

• Modify skip_if_broken_multiprocessing_synchronize() only only attempt to create a SemLock on Linux.
• Modify skip_if_broken_multiprocessing_synchronize() to only check if the current user is allowed to write into /dev/shm/ directory: that would allow touching the resource tracker.
• Modify the test to avoid improting _test_multiprocessing in the child process.

I don't know why the test only fails on macOS.

Note: I ignored macOS failure when I merged my PR because the macOS job of GitHub Action is failing randomly for at least 3 months and nobody attempted to fix it. So I learnt to ignore this job.

[vstinner]

New changeset 3358da4 by Victor Stinner in branch 'master':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
3358da4

[miss-islington]

New changeset ec9bc2d by Miss Islington (bot) in branch '3.8':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
ec9bc2d

[miss-islington]

New changeset 1529322 by Miss Islington (bot) in branch '3.9':
bpo-38377: Fix skip_if_broken_multiprocessing_synchronize() on macOS (GH-20984)
1529322

[vstinner]

Ok, macOS buildbots are back to green, I close the issue.