multiprocessing Manager mutable type member access failure

[vilnistermanis]

BPO	28422
Nosy	@applio, @iritkatriel
Files	manager_pickling.py: Script demonstrating problem

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-11-26.20:55:53.662>
created_at = <Date 2016-10-12.13:03:47.912>
labels = ['3.8', 'type-bug', 'library']
title = 'multiprocessing Manager mutable type member access failure'
updated_at = <Date 2021-11-26.20:55:53.661>
user = 'https://bugs.python.org/vilnistermanis'

bugs.python.org fields:

activity = <Date 2021-11-26.20:55:53.661>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-11-26.20:55:53.662>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2016-10-12.13:03:47.912>
creator = 'vilnis.termanis'
dependencies = []
files = ['45066']
hgrepos = []
issue_num = 28422
keywords = []
message_count = 6.0
messages = ['278529', '407085', '407090', '407091', '407092', '407093']
nosy_count = 3.0
nosy_names = ['vilnis.termanis', 'davin', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28422'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.8']

[vilnistermanis]

Accessing some Manager types (e.g. Lock) through a list, dict or Namespace proxy is not possible if both the mutable container (e.g. list) and contained type instance (e.g. Lock) have been created in the same process.

Symptoms:

In accessing process:
multiprocessing.managers.RemoteError on access, e.g.:
Unserializable message: ('#RETURN', <thread.lock object at 0x7fcf40d31370>)

.. and in underlying manager:
_pickle.PicklingError: Can't pickle <class '_thread.lock'>: attribute lookup lock on _thread failed

The provided test script performs:
0) Initialise SyncManager (via multiprocessing.Manager())

1. Create list proxy through manager
2. Insert another proxied type into the list (e.g. Lock)
3. Try to access type instance from (2) via container created in (1)

Note: When step (2) is run in a child process, everything work as expected. When all steps execute in the same process, one gets the aforementioned exception.

See also:
https://mail.python.org/pipermail/python-list/2009-September/552988.html

[iritkatriel]

I don't get any errors from the attached script. Is there still a problem here?

[vilnistermanis]

If you un-comment the print_exc() call, you'll see that it still fails - for queue.Queue:

Manager failure (for Queue)
Traceback (most recent call last):
  File "fish.py", line 74, in main
    add_type_in_own_process(mgr, type_name)
  File "fish.py", line 61, in add_type_in_own_process
    use_manager(obj, type_name, mgr=mgr)
  File "fish.py", line 35, in use_manager
    obj.append(getattr(mgr, type_name)())
  File "<string>", line 2, in append
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 243, in serve_client
    request = recv()
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.8/multiprocessing/managers.py", line 959, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'

---

[iritkatriel]

Version 3.8 is in security fix only mode. Is this a problem on newer versions?

[vilnistermanis]

Apologies, my mistake - it does indeed work with 3.9 & 3.10. Feel free to close.

[iritkatriel]

Thank you.