Calling help executes @classmethod @property decorated methods

[randolf-scholz]

BPO	45356
Nosy	@rhettinger, @terryjreedy, @ambv, @berkerpeksag, @serhiy-storchaka, @wimglenn, @corona10, @pablogsal, @akulakov, @sirosen, @randolf-scholz, @AlexWaygood
PRs	bpo-45356: fix incorrect access of class property in pydoc and inspect #29239
Files	classmethod_property.py classmethod_property.py ClassPropertyIdea.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = None
created_at = <Date 2021-10-03.19:45:55.963>
labels = ['type-bug', '3.11']
title = 'Calling `help` executes @classmethod @property decorated methods'
updated_at = <Date 2022-02-17.20:03:40.349>
user = 'https://github.com/randolf-scholz'

bugs.python.org fields:

activity = <Date 2022-02-17.20:03:40.349>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2021-10-03.19:45:55.963>
creator = 'randolf.scholz'
dependencies = []
files = ['50325', '50326', '50344']
hgrepos = []
issue_num = 45356
keywords = ['patch']
message_count = 23.0
messages = ['403109', '403110', '403111', '403504', '403505', '403578', '403582', '403611', '403613', '403653', '403699', '403713', '405100', '405115', '405121', '405142', '405143', '406600', '406605', '406606', '407493', '407499', '413451']
nosy_count = 13.0
nosy_names = ['rhettinger', 'terry.reedy', 'grahamd', 'lukasz.langa', 'berker.peksag', 'serhiy.storchaka', 'wim.glenn', 'corona10', 'pablogsal', 'andrei.avk', 'sirosen0', 'randolf.scholz', 'AlexWaygood']
pr_nums = ['29239']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45356'
versions = ['Python 3.11']

Linked PRs

• gh-89519: Remove classmethod descriptor chaining, deprecated since 3.11 #110163

[randolf-scholz]

I noticed some strange behaviour when calling help on a class inheriting from a class or having itself @classmethod @Property decorated methods.

from time import sleep
from abc import ABC, ABCMeta, abstractmethod

class MyMetaClass(ABCMeta):
    @classmethod
    @property
    def expensive_metaclass_property(cls):
        """This may take a while to compute!"""
        print("computing metaclass property"); sleep(3)
        return "Phew, that was a lot of work!"

    
class MyBaseClass(ABC, metaclass=MyMetaClass):
    @classmethod
    @property
    def expensive_class_property(cls):
        """This may take a while to compute!"""
        print("computing class property .."); sleep(3)
        return "Phew, that was a lot of work!"
    
    @property
    def expensive_instance_property(self):
        """This may take a while to compute!"""
        print("computing instance property ..."); sleep(3)
        return "Phew, that was a lot of work!"

class MyClass(MyBaseClass):
    """Some subclass of MyBaseClass"""
    
help(MyClass)

Calling help(MyClass) will cause expensive_class_property to be executed 4 times (!)

The other two properties, expensive_instance_property and expensive_metaclass_property are not executed.

Secondly, only expensive_instance_property is listed as a read-only property; expensive_class_property is listed as a classmethod and expensive_metaclass_property is unlisted.

The problem is also present in '3.10.0rc2 (default, Sep 28 2021, 17:57:14) [GCC 10.2.1 20210110]'

Stack Overflow thread: https://stackoverflow.com/questions/69426309

[randolf-scholz]

I updated the script with dome more info. The class-property gets actually executed 5 times when calling help(MyClass)

Computing class property of <class '__main__.MyClass'> ...DONE!
Computing class property of <class '__main__.MyClass'> ...DONE!
Computing class property of <class '__main__.MyClass'> ...DONE!
Computing class property of <class '__main__.MyBaseClass'> ...DONE!
Computing class property of <class '__main__.MyClass'> ...DONE!

[AlexWaygood]

See also: https://bugs.python.org/issue44904

[terryjreedy]

Randolf, what specific behaviors do you consider to be bugs that should be fixed. What would a test of the the changed behavior look like?

This should perhaps be closed as a duplicate of bpo-44904. Randolf, please check and say what you thing.

[terryjreedy]

On current 3.9, 3.10, 3.11, on Windows running in IDLE, I see
computing class property ..
computing class property ..
computing class property ..
computing class property ..
computing class property ..
Help ...

[randolf-scholz]

@terry I think the problem boils down to the fact that @classmethod @property decorated methods end up not being real properties.

Calling MyBaseClass.__dict__ reveals:

mappingproxy({'__module__': '__main__',
              'expensive_class_property': <classmethod at 0x7f893e95dd60>,
              'expensive_instance_property': <property at 0x7f893e8a5860>,
              '__dict__': <attribute '__dict__' of 'MyBaseClass' objects>,
              '__weakref__': <attribute '__weakref__' of 'MyBaseClass' objects>,
              '__doc__': None,
              '__abstractmethods__': frozenset(),
              '_abc_impl': <_abc._abc_data at 0x7f893fb98740>})

Two main issues:

1. Any analytics or documentation tool that has special treatment for properties may not identify 'expensive_class_property' as a property if they simply check via isinstance(func, property). Of course, this could be fixed by the tool-makers by doing a conditional check:

isinstance(func, property) or (`isinstance(func, classmethod)` and `isinstance(func.__func__, property)`

2. expensive_class_property does not implement getter, setter, deleter. This seems to be the real dealbreaker, for example, if we do

MyBaseClass.expensive_class_property = 2
MyBaseClass().expensive_instance_property = 2

Then the first line erroneously executes, such that MyBaseClass.dict['expensive_class_property'] is now int instead of classmethod, while the second line correctly raises AttributeError: can't set attribute since the setter method is not implemented.

[randolf-scholz]

If fact, in the current state it seem that it is impossible to implement real class-properties, for a simple reason:

descriptor.__set__ is only called when setting the attribute of an instance, but not of a class!!

import math

class TrigConst: 
    const = math.pi
    def __get__(self, obj, objtype=None):
        print("__get__ called")
        return self.const
    
    def __set__(self, obj, value):
        print("__set__ called")
        self.const = value
        

class Trig:
    const = TrigConst()              # Descriptor instance

Trig().const             # calls TrigConst.__get__
Trig().const = math.tau  # calls TrigConst.__set__
Trig.const               # calls TrigConst.__get__
Trig.const = math.pi     # overwrites TrigConst attribute with float.

[rhettinger]

I'm merging bpo-44904 into this one because they are related.

[rhettinger]

It may have been a mistake to allow this kind of decorator chaining.

• As Randolf and Alex have noticed, it invalidates assumptions made elsewhere in the standard library and presumably in third-party code as well.

• And as Randolf noted in his last post, the current descriptor logic doesn't make it possible to implement classmethod properties with setter logic.

• In bpo-44973, we found that staticmethod and property also don't compose well, nor does abstract method.

We either need to undo the Python 3.9 feature from bpo-19072, or we need to put serious thought into making all these descriptors composable in a way that would match people's expectations.

[randolf-scholz]

Dear Raymond,

I think decorator chaining is a very useful concept. At the very least, if all decorators are functional (following the functools.wraps recipe), things work out great -- we even get associativity of function composition if things are done properly!

The question is: can we get similar behaviour when allowing decoration with stateful objects, i.e. classes? This seems a lot more difficult. At the very least, in the case of @classmethod I think one can formulate a straightforward desiderata:

### Expectation

• classmethod(property) should still be a property!
• More generally: classmethod(decorated) should always be a subclass of decorated!

Using your pure-python versions of property / classmethod from https://docs.python.org/3.9/howto/descriptor.html, I was able to write down a variant of classmethod that works mostly as expected in conjunction with property. The main idea is rewrite the classmethod to dynamically be a subclass of whatever it wrapped; roughly:

def ClassMethod(func):
  class Wrapped(type(func)):
      def __get__(self, obj, cls=None):
          if cls is None:
              cls = type(obj)
          if hasattr(type(self.__func__), '__get__'):
              return self.__func__.__get__(cls)
          return MethodType(self.__func__, cls)
  return Wrapped(func)

I attached a full MWE. Unfortunately, this doesn't fix the help bug, though it is kind of weird because the decorated class-property now correctly shows up under the "Readonly properties" section. Maybe help internally checks isinstance(cls.func, property) at some point instead of isinstance(cls.__dict__["func"], property)?

### Some further Proposals / Ideas

1. Decorators could always have an attribute that points to whatever object they wrapped. For the sake of argument, let's take __func__.
   ⟹ raise Error when typing @decorator if not hasattr(decorated, "__func__")
   ⟹ Regular functions/methods should probably by default have __func__ as a pointer to themselves?
   ⟹ This could hae several subsidiary benefits, for example, currently, how would you implement a pair of decorators @print_args_and_kwargs and @time_execution such that both of them only apply to the base function, no matter the order in which they are decorating it? The proposed __func__ convention would make this very easy, almost trivial.
2. type.__setattr__ could support checking if attr already exists and has __set__ implemented.
   ⟹ This would allow true class-properties with getter, setter and deleter. I provide a MWE here: https://mail.google.com/mail/u/0/#label/Python+Ideas/FMfcgzGlkPRbJVRkHHtkRPhMCxNsFHpl
3. I think an argument can be made that it would be really, really cool if @ could become a general purpose function composition operator?
   ⟹ This is already kind of what it is doing with decorators
   ⟹ This is already exacltly what it is doing in numpy -- matrix multiplication *is* the composition of linear functions.
   ⟹ In fact this is a frequently requested feature on python-ideas!
   ⟹ But here is probably the wrong place to discuss this.

[AlexWaygood]

Some thoughts from me, as an unqualified but interested party:

Like Randolph, I very much like having class properties in the language, and have used them in several projects since their introduction in 3.9. I find they're especially useful with Enums. However, while the bug in doctest, for example, is relatively trivial to fix (see my PR in bpo-44904), it seems to me plausible that bugs might crop up in other standard library modules as well. As such, leaving class properties in the language might mean that several more bugfixes relating to this feature would have to be made in the future. So, I can see the argument for removing them.

It seems to me that Randolph's idea of changing classmethod from a class into a function would break a lot of existing code. As an alternative, one small adjustment that could be made would be to special-case isinstance() when it comes to class properties. In pure Python, you could achieve this like this:

oldproperty = property

class propertymeta(type):
    def __instancecheck__(cls, obj):
        return super().__instancecheck__(obj) or (
            isinstance(obj, classmethod)
            and super().__instancecheck__(obj.__func__)
            )


class property(oldproperty, metaclass=propertymeta): pass

This would at least mean that isinstance(classmethod(property(lambda cls: 42)), property) and isinstance(classmethod(property(lambda cls: 42)), classmethod) would both evaluate to True. This would be a bit of a lie (an instance of classmethod isn't an instance of property), but would at least warn the caller that the object they were dealing with had some propertylike qualities to it.

Note that this change wouldn't fix the bugs in abc and doctest, nor does it fix the issue with class-property setter logic that Randolph identified.

With regards to the point that @staticmethod cannot be stacked on top of @property: it strikes me that this feature isn't really needed. You can achieve the same effect just by stacking @classmethod on top of @property and not using the cls parameter.

[randolf-scholz]

@alex Regarding my proposal, the crucial part is the desiderata, not the hacked up implementation I presented.

And I really believe that at least that part I got hopefully right. After all, what does @classmethod functionally do? It makes the first argument of the function receive the class type instead of the object instance and makes it possible to call it directly from the class without instantiating it. I would still expect MyClass.__dict__["themethod"] to behave, as an object, much the same, regardless if it was decorated with @classmethod or not.

Regarding code breakage - yes that is a problem, but code is already broken and imo Python needs to make a big decision going forward:

1. Embrace decorator chaining and try hard to make it work universally (which afaik was never intended originally when decorators were first introduced). As a mathematician I would love this, also adding @ as a general purpose function composition operator would add quite some useful functional programming aspects to python.
2. Revert changes from 3.9 and generally discourage decorator chaining.

At this point however I want to emphasize that I am neither a CS major nor a python expert (I only started working with the language 3 years ago), so take everything I say as what it is - the opinion of some random stranger from the internet (:

[akulakov]

I've put up a PR; I'm not sure it's the best way to fix it. I will look more into it and will try to post some details about the PR later today.

[akulakov]

I missed that this is assigned to Raymond, hope we didn't duplicate any effort (it only took me a short while to do the PR). Apologies..

[akulakov]

I've looked more into it, the issue is that even before an object can be tested with isinstance(), both inspect.classify_class_attrs and in pydoc, classdoc local function spill() use a getattr() call, which triggers the property.

So I think my PR is going in the right direction, adding guards in the inspect function and in two spill() functions. If isinstance() can be fixed to detect class properties correctly, these guards can be simplified but they still need to be there, I think.

[wimglenn]

added Graham Dumpleton to nosy list in case he has some useful insight here, I think the PR from bpo-19072 may have been adapted from grahamd's patch originally?

[grahamd]

Too much to grok right now.

There is already a convention for what a decorator wraps. It is __wrapped__.

cpython/Lib/functools.py

Line 70 in 3405792

wrapper.__wrapped__ = wrapped

Don't use __func__ as that has other defined meaning in Python related to bound methods and possibly other things as well and overloading on that will break other stuff.

In part I suspect a lot of the problems here are because things like classmethod and functools style decorators are not proper transparent object proxies, which is the point of what the wrapt package was trying to solve so that accessing stuff on the wrapper behaves as much as possible as if it was done on what was wrapped, including things like isinstance checks.

[rhettinger]

Also see: https://bugs.python.org/issue42073

The classmethod pass through broke some existing code and the "fix" for it looks dubious:

            if hasattr(type(self.f), '__get__'):
                return self.f.__get__(cls, cls)

[rhettinger]

I propose deprecating classmethod chaining. It has become clear that it doesn't really do what people wanted and can't easily be made to work.

By even suggesting that some stateful decorators are composable, we've ventured onto thin ice. Wrapping property in a classmethod doesn't produce something that behaves like a real property. Mixing staticmethod and property doesn't work at all. Putting abstractmethod in the mix doesn't work well either. The ecosystem of code inspection tools, like help() in this issue, is wholly unprepared for recognizing and working around these combinations. The latest "fix" for classmethod chaining looks weird and worriesome as well: self.f.__get__(cls, cls).

Classmethod chaining is relatively new, so we will affect very little code by deprecating it. Any of the possible use cases can be served in other ways like the wrapt package or by explicit code in __getattribute__.

[AlexWaygood]

It makes me sad that the stdlib will no longer provide a way to compose classmethods with other descriptors. However, I agree that deprecating classmethod chaining is probably the correct course of action, given the complications this feature has caused, and the backwards-compatibility issues it raises.

This is probably a conversation for another BPO issue or the python-ideas mailing list, but I hope some consideration can be given in the future as to whether a new classmethod-like feature could possibly be added to functools that would enable this kind of decorator chaining without the same code-breakage concerns that this feature has had.

[sirosen]

Probably >90% of the use-cases for chaining classmethod are a read-only class property.
It's important enough that some tools (e.g. sphinx) even have special-cased support for classmethod(property(...)).

Perhaps the general case of classmethod(descriptor(...)) should be treated separately from the common case?

I propose deprecating classmethod chaining. It has become clear that it doesn't really do what people wanted and can't easily be made to work.

If classmethod(property(f)) is going to be removed, can an implementation of classproperty be considered as a replacement?

Or perhaps support via the other ordering, property(classmethod(f))?

[ProgramRipper]

I notice that this feature will be removed by #110163 in Python 3.13. So are there any workaround? If so, that may be great if it could be added to the Descriptor HowTo Guide, because it has been commonly used.

[Prometheus3375]

Here is the sample of the implementation. You can also use a metaclass to add a property.

[ReHoss]

In Python 3.10, classmethod() added a wrapped attribute. Presumably, any use case for implicit chaining can now be accomplished in an explicit and controlled manner.

@rhettinger - Could you share an example of how you would do this as a replacement for chaining @classmethod and @property?

@zduvall Here is an implementation for a read-only class property that is backwards/forwards compatible:

import sys
from typing import Generic, Optional, TypeVar

if sys.version_info < (3, 9):
    from typing import Callable, Type
else:
    from builtins import type as Type
    from collections.abc import Callable

T = TypeVar("T")
RT = TypeVar("RT")


class classproperty(Generic[T, RT]):
    """
    Class property attribute (read-only).

    Same usage as @property, but taking the class as the first argument.

        class C:
            @classproperty
            def x(cls):
                return 0

        print(C.x)    # 0
        print(C().x)  # 0
    """

    def __init__(self, func: Callable[[Type[T]], RT]) -> None:
        # For using `help(...)` on instances in Python >= 3.9.
        self.__doc__ = func.__doc__
        self.__module__ = func.__module__
        self.__name__ = func.__name__
        self.__qualname__ = func.__qualname__
        # Consistent use of __wrapped__ for wrapping functions.
        self.__wrapped__: Callable[[Type[T]], RT] = func

    def __set_name__(self, owner: Type[T], name: str) -> None:
        # Update based on class context.
        self.__module__ = owner.__module__
        self.__name__ = name
        self.__qualname__ = owner.__qualname__ + "." + name

    def __get__(self, instance: Optional[T], owner: Optional[Type[T]] = None) -> RT:
        if owner is None:
            owner = type(instance)
        return self.__wrapped__(owner)

Additionally, as shown in this issue thread, running help(...) on things like this will not work as expected. In the example shown in help(classproperty), running help(C) will show x = 0 instead of the documentation for the class property, if any. This is the same behavior as the current @classmethod @property solution, nothing fancy there. As such, you may want to simplify the __init__ and remove the __set_name__ as being completely unnecessary.

Note also that as shown by @Paebbels, the descriptor protocol ONLY supports class getters, it does not support class setters or class deleters. As such, only read-only class properties are possible.

This isn't to say you can't have mutable class properties however, only that you can't have them follow the descriptor protocol for the nicer syntax. You can instead have the class getter return objects with .get, .set, and .delete methods as a workaround:

import sys
from types import MethodType
from typing import Any, Callable, Generic, Optional, TypeVar, Union

if sys.version_info < (3, 9):
    from typing import Type
else:
    from builtins import type as Type

if sys.version_info < (3, 11):
    Self = TypeVar("Self", bound="classproperty")
else:
    from typing import Self

T = TypeVar("T")


class BoundClassDescriptor(Generic[T]):

    def __init__(
        self,
        owner: Type[T],
        descriptor: property,
        module: Optional[str],
        name: Optional[str],
        qualname: Optional[str],
    ) -> None:
        self.__doc__ = descriptor.__doc__
        self.__module__ = module
        self.__name__ = name
        self.__qualname__ = qualname
        self.__wrapped__: property = descriptor

        self.get: Callable[[], Any]
        self.set: Callable[[Any], object]
        self.delete: Callable[[], object]

        if descriptor.fget is not None:
            self.get = MethodType(descriptor.fget, owner)

        if descriptor.fset is not None:
            self.set = MethodType(descriptor.fset, owner)

        if descriptor.fdel is not None:
            self.delete = MethodType(descriptor.fdel, owner)


class classproperty(Generic[T]):

    def __init__(
        self,
        fget: Optional[Callable[[Type[T]], Any]] = None,
        fset: Optional[Callable[[Type[T], Any], object]] = None,
        fdel: Optional[Callable[[Type[T]], object]] = None,
        doc: Optional[str] = None,
    ) -> None:
        self.__module__ = None
        self.__name__ = None
        self.__qualname__ = None
        self.__wrapped__: property = property(fget, fset, fdel, doc)

    def __set_name__(self, owner: Type[T], name: str) -> None:
        self.__module__ = owner.__module__
        self.__name__ = name
        self.__qualname__ = owner.__qualname__ + "." + name

    def __get__(
        self, instance: Optional[T], owner: Optional[Type[T]] = None
    ) -> BoundClassDescriptor[T]:
        if owner is None:
            owner = type(instance)
        return BoundClassDescriptor(
            owner,
            self.__wrapped__,
            self.__module__,
            self.__name__,
            self.__qualname__,
        )

    def getter(self: Self, func: Callable[[Type[T]], Any]) -> Self:
        self.__wrapped__ = self.__wrapped__.getter(func)
        return self

    def setter(self: Self, func: Callable[[Type[T], Any], object]) -> Self:
        self.__wrapped__ = self.__wrapped__.setter(func)
        return self

    def deleter(self: Self, func: Callable[[Type[T]], object]) -> Self:
        self.__wrapped__ = self.__wrapped__.deleter(func)
        return self

Example:

class A:
    _foo = 0

    @classproperty
    def foo(cls):
        """Description of foo class property."""
        return cls._foo

    @foo.setter
    def foo(cls, value):
        cls._foo = value


class B(A):
    pass

Usage:

>>> # Getters/setters/deleters work as you'd expect using `A._foo` directly.
>>> A.foo.get()
0
>>> B.foo.get()
0
>>> 
>>> A.foo.set(1)
>>> A.foo.get()
1
>>> B.foo.get()
1
>>> 
>>> B.foo.set(2)
>>> A.foo.get()
1
>>> B.foo.get()
2
>>> 
>>> help(A.foo)  # Instance documentation works in Python >= 3.9.
Help on BoundClassDescriptor in A:

A.foo = <BoundClassDescriptor object>
    Description of foo class property.

>>> help(A.foo.get)
Help on method foo in module __main__:

foo() method of builtins.type instance
    Description of foo class property.

>>> help(A.foo.set)  # You could have custom setter documentation.
Help on method foo in module __main__:

foo(value) method of builtins.type instance

>>> help(A.foo.delete)  # Only has a delete method if one was set.
Traceback (most recent call last):
  ...
AttributeError: 'BoundClassDescriptor' object has no attribute 'delete'

Of course the documentation won't work on static analyzers, but you'll at least be able to dynamically use help(...) and get what you'd expect.

Hi @SimpleArt ,

The pyright type checker for the instruction return self.__wrapped__(owner) returns:

Argument of type "type[T@classproperty] | type[object]* | None; cannot be assigned to parameter of type ";type[T@classproperty]";
  Type ";type[T@classproperty] | type[object]* | None"; is incompatible with type ";type[T@classproperty]"; (reportArgumentType)

for this slightly modified version of the __get__ method (modified such that type(None) does not happen)

    def __get__(self, instance: Optional[T], owner: Type[T] | None = None) -> RT:
        if owner is None:
            if instance is not None:
                owner = type(instance)
            else:
                owner = None
        return self.__wrapped__(owner)  # pyright triggers warning

also this assignment

self.__module__ = func.__module__

gives

Cannot find reference 'module' in '(T) -> RT'

Moreover, could you provide a simple example of using abstract class properties for Python>=3.13 ?