multiprocessing: test_shared_memory_across_processes() cannot be run twice in parallel

[vstinner]

BPO	40135
Nosy	@vstinner, @miss-islington, @shihai1991
PRs	bpo-40135: don't define shared memory block's name in test_shared_memory_across_… #19892 [3.8] bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892) #19903

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-04.15:07:29.527>
created_at = <Date 2020-04-01.13:18:18.464>
labels = ['3.8', 'tests', '3.9']
title = 'multiprocessing: test_shared_memory_across_processes() cannot be run twice in parallel'
updated_at = <Date 2020-05-04.15:25:29.513>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-05-04.15:25:29.513>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-04.15:07:29.527>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-04-01.13:18:18.464>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40135
keywords = ['patch']
message_count = 6.0
messages = ['365462', '365463', '368007', '368056', '368057', '368059']
nosy_count = 3.0
nosy_names = ['vstinner', 'miss-islington', 'shihai1991']
pr_nums = ['19892', '19903']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40135'
versions = ['Python 3.8', 'Python 3.9']

[vstinner]

Sometimes, I need to run multiprocessing tests multiple times in parallel to attempt to reproduce a race condition. Except that test_shared_memory_across_processes() fails in this case:

$ ./python -m test test_multiprocessing_spawn --fail-env-changed -v -j4 -F -m test_shared_memory_across_processes 

== CPython 3.9.0a5+ (heads/master:17b4733f2f, Apr 1 2020, 15:02:04) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)]
== Linux-5.5.9-200.fc31.x86_64-x86_64-with-glibc2.30 little-endian
== cwd: /home/vstinner/python/master/build/test_python_1133450
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.64 Run tests in parallel using 4 child processes
0:00:01 load avg: 1.64 [ 1/1] test_multiprocessing_spawn failed
test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory) ... ERROR

======================================================================
ERROR: test_shared_memory_across_processes (test.test_multiprocessing_spawn.WithProcessesTestSharedMemory)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/vstinner/python/master/Lib/test/_test_multiprocessing.py", line 3862, in test_shared_memory_across_processes
    sms = shared_memory.SharedMemory('test02_tsmap', True, size=512)
  File "/home/vstinner/python/master/Lib/multiprocessing/shared_memory.py", line 100, in __init__
    self._fd = _posixshmem.shm_open(
FileExistsError: [Errno 17] File exists: '/test02_tsmap'

---

Ran 1 test in 0.247s

FAILED (errors=1)
test test_multiprocessing_spawn failed
Kill <TestWorkerProcess #2 running test=test_multiprocessing_spawn pid=1133458 time=1.2 sec> process group
Kill <TestWorkerProcess #3 running test=test_multiprocessing_spawn pid=1133457 time=1.2 sec> process group
Kill <TestWorkerProcess #4 running test=test_multiprocessing_spawn pid=1133459 time=1.2 sec> process group

== Tests result: FAILURE ==

1 test failed:
test_multiprocessing_spawn

Total duration: 1.2 sec
Tests result: FAILURE

[vstinner]

Python 3.7 is not affected: it doesn't have test_shared_memory_across_processes().

[shihai1991]

The easy way to fix this test case is that don't define the shared memory block's name. test_shared_memory_basics has tested against the shared memory block's name, so canceling the shared memory block'name has no side-effect.

[vstinner]

New changeset caa3ef2 by Hai Shi in branch 'master':
bpo-40135: Fix multiprocessing test_shared_memory_across_processes() (GH-19892)
caa3ef2

[vstinner]

Thanks for the fix Hai Shi!