Cannot modify dictionaries inside dictionaries using Managers from multiprocessing #51015
Comments
|
It's not possible to modify a dict inside a dict using a manager from Ex: from multiprocessing import Process,Manager
def f(d):
d['1'] = '1'
d['2']['1'] = 'Try To Write'
if __name__ == '__main__':
manager = Manager()
d = manager.dict() p = Process(target=f, args=(d,))
p.start()
p.join()The output Under Windows 7 (32 Bits) / Python 2.6.2 (32 Bits) is: {'2': {}} The output is the same if you change "d['2'] = manager.dict()" to |
|
I get the same results on: I think this is the same issue I was seeing yesterday. You can exercise ##### CODE #####
from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print ns_proxy.my_event_proxy##### TRACEBACK #####
Traceback (most recent call last):
File "test_nsproxy.py", line 39, in <module>
print ns_proxy.my_event_proxy
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 989, in
__getattr__
return callmethod('__getattribute__', (key,))
File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in
_callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:Unserializable message: ('#RETURN', <threading._Event object at 0x1494790>) Storing a proxy into a proxied object and then accessing the proxy |
|
When a manager receives a message, it unpickles the arguments; this |
|
Nothing jumps out to me off the top of my head - I can take a closer look When we included multiprocessing, some tests were deemed too unstable at |
|
The tests for the SyncManager are being automagically generated at |
|
Yeah, the auto-generation is too clever and needs to be pulled out |
|
Even with the patch, I can not resolve this problem. I can reproduce the problem with the patched version with the following code. My system is: Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) import multiprocessing as mp
def f(d):
d['f'] = {}
d['f']['msg'] = 'I am here'
manager = mp.Manager()
d = manager.dict()
p = mp.Process(target=f, args=(d,))
p.start()
p.join()print d d = {}
f(d)print d Output: {'f': {}} |
|
Kaushik, in your example, d is a dict proxy, so assignment to d['f'] correctly ferries the assignment (a new normal dict) to the d['f'] in the original process. The new dict, however, is not a dict proxy, it's just a dict, so assignment of d['f']['msg'] goes nowhere. All hope is not lost, however, because the Manager can be forked to new processes. The slightly modified example below shows how this works: from multiprocessing import Process, Manager
def f(m, d):
d['f'] = m.dict()
d['f']['msg'] = 'I am here'
m = Manager()
d = m.dict()
p = Process(target=f, args=(m,d))
p.start()
p.join()
print d
{'f': <DictProxy object, typeid 'dict' at 0x7f1517902810>}
print d['f']
{'msg': 'I am here'}With the attached patch, the above works as shown, without, it gives the same output as your original example. |
|
Hello, The shared variable in results and what I'm trying to do is simultaneously parsing multiple files. The quality of the code is not very good because I'm a newbie python programmer. Best regards, |
|
I'm getting these results on both: The symptoms are exactly as Terrence described. Nesting proxied containers is supposed to be a supported use case! From the documentation: http://docs.python.org/2/library/multiprocessing.html#proxy-objects >>> a = manager.list()
>>> b = manager.list()
>>> a.append(b) # referent of a now contains referent of b
>>> print a, b
[[]] []
>>> b.append('hello')
>>> print a, b
[['hello']] ['hello']
The documented code works as expected, but:
>>> a[0].append('world') # Appends to b?
>>> print a, b
[['hello']] ['hello']I've attached my reproduction as a script. |
|
I'm still running into these issues with Python 2.7.10. I'm trying to find a way to share dynamically allocated sub-dictionaries through multiprocessing as well as dynamically allocated RLock and Value instances. I can use the manager to create them but when I put them in a managed dict the various issues related in this ticket happen. |
|
Two core issues are compounding one another here:
As Terrence described, it is RebuildProxy where the decision is made to not return a proxy object but a new local instance (copy) of the managed object from the Server. Unfortunately there are use cases where Terrence's proposed modification won't work such as a managed list that contains a reference to itself or more generally a managed list/dict that contains a reference to another managed list/dict when an attempt is made to delete the outer managed list/dict before the inner. The reference counting implementation in multiprocessing.managers.Server obtains a lock before decrementing reference counts and any deleting of objects whose count has dropped to zero. In fact, when an object's ref count drops to zero, it deletes the object synchronously and won't release the lock until it's done. If that object contains a reference to another proxy object (managed by the same Manager and Server), it will follow a code path that leads it to wait forever for that same lock to be released before it can decref that managed object. I agree with Jesse's earlier assessment that the current behavior (returning a copy of the managed object and not a proxy) is unintended and has unintended consequences. There are hints in Richard's (sbt's) code that also suggest this is the case. Merely better documenting the current behavior does nothing to address the lack of or at least limited utility suggested in the comments here or the extra complications described in bpo-20854. As such, I believe this is behavior that should be addressed in 2.7 as well as 3.x. My proposed patch makes the following changes:
Concerned about performance, I've attempted applying the #2 change without the others and put it through stress tests on a 4.0GHz Core i7-4790K in a iMac-Retina5K-late2014 OS X system and discovered no degradation in execution speed or memory overhead. If anything with #2 applied it was slightly faster but the differences are too small to be regarded as anything more significant than noise. These proposed changes enable the following code to execute and terminate cleanly: To produce the following output: Justin: I've tested the RLock values I've gotten back from my managed lists too -- just didn't have that in the example above. Patches to be attached shortly (after cleaning up a bit). |
|
Attaching patch for default (3.6) branch which implements what was previously described and discussed, updates the documentation to explain this updated behavior, and includes new tests. @yselivanov: Can you think of any edge cases that should be handled but we're missing? |
|
Updating previously supplied patch for 3.6 to the right format. |
|
Attaching updated patch to reflect Yury's suggested changes from review. |
|
New changeset 39e7307f9aee by Davin Potts in branch 'default': |
|
Fixed in upcoming 3.6. |
|
Hi all, I'm trying to use multiprocessing with a 3d list. From the documentation I expected it to work. As I found this report a bid later, I opened a bug report here: https://bugs.python.org/issue32538. Am I doing sth. wrong or is it still not working in 3.6.3? |
|
Hi team, Looks like this issue remains per code below: import multiprocessing, sys, time, traceback;
if __name__ == '__main__': mpd = multiprocessing.Manager().dict();
mpd['prcss'] = {'q' : 'queue_1', 'ctlg' : 'ctlg_1' }; # update 3 - works!
mpd_prcss = mpd['prcss'];
mpd_prcss['name'] = 'concfun_1';
mpd['prcss'] = mpd_prcss;
print('Result of successful update 3:', mpd['prcss']);### --- output ### |
|
Hey folks, This is still an issue with 3.7.2 =============================================== # Python 3.7.2 (default, Jan 10 2019, 23:51:51) from multiprocessing import Manager
manager = Manager()
d = manager.dict({})d["test"] = {"a": 123} print(d) |
|
Just to be clear, the patch detailed by @applio only works if the nested dictionaries/lists are created through a manager. Therefore, you must store your dictionaries/lists inside a manager before nesting them inside another managed dictionary/list. There is, however, a way to handle this automatically without explicitly storing the nested objects inside a manager detailed here: https://stackoverflow.com/a/73418403/16310741 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: