Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

type(obj).__module__ is not a string when obj is a celery.local.Proxy() instance #1087

Closed
mgedmin opened this Issue Nov 23, 2012 · 6 comments

Comments

Projects
None yet
3 participants
Contributor

mgedmin commented Nov 23, 2012

I've a webapp that has the Dozer memleak middleware enabled. I once saw this error:

Exception in thread Thread-8:
Traceback (most recent call last):
  Module threading, line 552, in __bootstrap_inner
    self.run()
  Module threading, line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  Module dozer.leak, line 119, in start
    self.tick()
  Module dozer.leak, line 134, in tick
    typename = objtype.__module__ + "." + objtype.__name__
TypeError: unsupported operand type(s) for +: 'property' and 'str'

What dozer.leak does is essentially

    for obj in gc.get_objects():
        obtype = type(obj)
...
        typename = objtype.__module__ + "." + objtype.__name__

A recursive grep over my project and all its dependencies found only one place where __module__ is defined as a property: celery.local.Proxy.

This can be easily fixed by writing a custom descriptor.

Running into the exact same problem. What's the solution?

Contributor

mgedmin commented Feb 15, 2013

Dozer 0.3.2 has a workaround for this.

Contributor

mgedmin commented Feb 15, 2013

Actually, my experiments show that this cannot be "easily fixed by writing a custom descriptor".

some_class.__module__ invokes type.__dict__['__module__'].__get__(some_class), which is some C code in CPython's typeobject.c that returns some_class.__dict__['__module__'] without checking if it's a descriptor.

The only thing I can come up with is therefore

class module_property(property):
    def __str__(self):
        return self.fget.__module__

class Proxy(object):
    def __init__(self, obj):
        self._obj = obj

    @module_property
    def __module__(self):
        return self._obj.__module__

This seems to work:

>>> import decimal
>>> obj = decimal.Decimal
>>> print(obj.__module__)
decimal
>>> proxy = Proxy(obj)
>>> print(proxy.__module__)
decimal
>>> print(type(proxy).__module__)
__main__
>>> '%s.%s' % (type(proxy).__module__, type(proxy).__name__)
'__main__.Proxy'

Not everything works:

>>> type(proxy).__module__ + '.' + type(proxy).__name__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'module_property' and 'str'
Contributor

mgedmin commented Feb 15, 2013

Oh, look, I solved it. Behold the unholy horror:

class module_property(str):
    def __new__(cls, fn):
        instance = str.__new__(cls, fn.__module__)
        instance._fn = fn
        return instance
    def __get__(self, obj, cls=None):
        if obj is not None:
            return self._fn(obj)
        else:
            return self
    def __set__(self, obj, value):
        raise AttributeError('cannot set attribute')


class Proxy(object):

    def __init__(self, obj):
        self._obj = obj

    @module_property
    def __module__(self):
        return self._obj.__module__


import decimal
obj = decimal.Decimal
print(obj.__module__)
proxy = Proxy(obj)
print(proxy.__module__)
print(type(proxy).__module__)
print(type(proxy).__module__ + '.' + type(proxy).__name__)
try:
    proxy.__module__ = "haha can't change it"
except AttributeError:
    print('ok')

Run it and you'll see

decimal
decimal
__main__
__main__.Proxy
ok
Owner

ask commented Apr 11, 2013

I'm not sure why this happens, does it happen when the tool analyzes the Proxy class itself?

In that case you're right, Proxy.__module__ should indeed be a string,
same with __doc__ and __name__. And your solution looks like it should work.
It's hairy indeed, but this is not an easy problem, so we need some tricks to have it work correctly in all cases.

Owner

ask commented Apr 11, 2013

Thanks for the solution! fixed in a3bba87

@ask ask closed this Apr 11, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment