Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing.managers.Server: problem with returning proxy of registered object #65053

allista mannequin opened this issue Mar 5, 2014 · 4 comments

multiprocessing.managers.Server: problem with returning proxy of registered object #65053

allista mannequin opened this issue Mar 5, 2014 · 4 comments
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error


Copy link

allista mannequin commented Mar 5, 2014

BPO 20854
Nosy @applio

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-04.23:20:09.530>
created_at = <Date 2014-03-05.16:03:30.259>
labels = ['type-bug', 'library']
title = 'multiprocessing.managers.Server: problem with returning proxy of registered object'
updated_at = <Date 2022-01-04.23:20:09.529>
user = '' fields:

activity = <Date 2022-01-04.23:20:09.529>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-04.23:20:09.530>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2014-03-05.16:03:30.259>
creator = 'allista'
dependencies = []
files = []
hgrepos = []
issue_num = 20854
keywords = []
message_count = 4.0
messages = ['212763', '212934', '212959', '301651']
nosy_count = 3.0
nosy_names = ['sbt', 'allista', 'davin']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = ''
versions = ['Python 2.7', 'Python 3.4']

Copy link
Mannequin Author

allista mannequin commented Mar 5, 2014

  1. I have a tree-like recursive class MyClass. It's method 'get_child(i)' returns an instanse of that same class.
  2. I register this class with BaseManager as follows:
class MyManager(BaseManager): pass
MyManager.register('MyClass', MyClass, method_to_typeid={'get_child':'MyClass'})
  1. When I call 'get_child' method of AutoProxy[MyClass] object, the exception is raised in the '__init__' method of MyClass: it is called with a single argument which is the instance of MyClass returned by 'get_child'. This happens in the following code of multiprocessing.managers.Server.create method:

373 def create(self, c, typeid, *args, **kwds):
382 if callable is None:
383 assert len(args) == 1 and not kwds
384 obj = args[0]
385 else:
386 obj = callable(*args, **kwds) <-This line raises the exception

This means that if ANY method registered with a Manager should return a proxy for a registered typeid, for which a callable is provided, it will fail, unless the callable is capable to handle such unexpected arguments.

@allista allista mannequin added the stdlib Python modules in the Lib dir label Mar 5, 2014
Copy link

sbt mannequin commented Mar 8, 2014

I am not sure method_to_typeid and create_method were really intended to be public -- they are only used by Pool proxies.

You can maybe work around the problem by registering a second typeid without specifying callable. That can be used in method_to_typeid:

import multiprocessing.managers

class MyClass(object):
    def __init__(self):
        self._children = {}
    def get_child(self, i):
        return self._children.setdefault(i, type(self)())
    def __repr__(self):
        return '<MyClass %r>' % self._children

class MyManager(multiprocessing.managers.BaseManager):

MyManager.register('MyClass', MyClass,
                   method_to_typeid = {'get_child': '_MyClass'})
                   method_to_typeid = {'get_child': '_MyClass'},

if __name__ == '__main__':
    m = MyManager()
        a = m.MyClass()
        b = a.get_child(1)
        c = b.get_child(2)
        d = c.get_child(3)
        print a  # <MyClass {1: <MyClass {2: <MyClass {3: <MyClass {}>}>}>}>

Copy link
Mannequin Author

allista mannequin commented Mar 9, 2014

Thanks for the suggestion.
method_to_typeid and create_method are documented features, so I don't see why not. It does the trick in a cleaner way than my workaround: a metaclass for MyClass that just checks the arguments before creating a new instance. It just seems to me somewhat counterintuitive.

Another issue that arises in my case is: when I try to pass a proxy of MyClass to a subprocess it looses its' _manager during pickling and thus the ability to create proxies for children returned by get_child. This is solved by reimplementing the (not-working: __reduce__ method of BaseManager in MyManager and creating corresponding custom proxy for MyClass with __reduce__ method also reimplemented.

So the working solution for the situation is:

class ReturnProxy(type):
def __call__(cls, *args, **kwargs):
if not kwargs and args and isinstance(args[0], cls):
return args[0]
return super(ReturnProxy, cls).__call__(*args, **kwargs)

class MyClass(object):
    __metaclass__ = ReturnProxy
    ###class body###

Your solution with the second typeid registration.

class AutoProxyMeta(type):
'''Metaclass that replicates multiprocessing.managers.MakeProxyType
functionality, but allows proxy classes that use it to be pickable'''
def __new__(cls, name, bases, attrs):
dic = {}
for meth in attrs.get('_exposed_', ()):
exec '''def %s(self, *args, **kwds):
return self._callmethod(%r, args, kwds)''' % (meth, meth) in dic
return super(AutoProxyMeta, cls).__new__(cls, name, bases, dic)

class MyClassProxy(BaseProxy):
__metaclass__ = AutoProxyMeta
_exposed_ = ('get_child',)
_method_to_typeid_ = dict(get_child='MyClass')
#or: _method_to_typeid_ = dict(get_child='_MyClass')

def \_\_reduce__(self):
    \_unpickle, (cls, token, serializer, kwds) = BaseProxy.\_\_reduce__(self)
    kwds['manager'] = self.\_manager
    return \_unpickle, (cls, token, serializer, kwds)
class MyClassManager(UManager):
    def __reduce__(self):
        return (RebuildMyClassManager,
                (self._address, None, self._serializer))
WorkCounterManager.register('MyClass', MyClass, MyClassProxy)
#optionally: WorkCounterManager.register('_MyClass', None, MyClassProxy, create_method=False)

def RebuildMyClassManager(address, authkey, serializer):
    mgr = MyClassManager(address, authkey, serializer)
    return mgr

Copy link

applio commented Sep 7, 2017

It appears that the multiple workarounds proposed by the OP (@allista) address the original request and that there is no bug or unintended behavior arising from multiprocessing itself. Combined with the lack of activity in this discussion, I'm inclined to believe that the workarounds have satisfied the OP and this issue should be closed.

@applio applio added the type-bug An unexpected behavior, bug, or error label Sep 7, 2017
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
None yet

No branches or pull requests

2 participants